GenSynth Documentation

Using GenSynth Explain

GenSynth Explain helps you better understand bias and errors in your models, as well as the overall behaviour of your models, by learning about their inner workings and revealing how they are making decisions based on what they have learned.

Enabling GenSynth Explain

When you are creating your job, in the Explain section of the New Job tab, you can choose whether or not you want to enable Root Cause Analysis for Classification Models or Root Cause Analysis for Object Detection Models from the drop-down, which will let GenSynth learn more about the inner workings of the model and identify how the model makes decisions about the data. Enabling this option will give you access to the GenSynth Explain root cause analysis Dynamic Confusion Matrix (in the Details view, alongside the Network and Layer Visualizations) for each completed cycle.

image4.png

Note

Currently, only Root Cause Analysis for Classification Models and Root Cause Analysis for Object Detection Models are available.

You must configure how many test samples are selected for analysis, either by a percentage or by a number of batches. The test samples are selected at random, by the batch, for both options. To use a percentage, make sure Provide Percentage of Samples for Analysis is enabled and choose a percentage in the slider. Otherwise, disable it and enter a specific number of batches.

You can save your Explain settings as a template. 

GenSynth Explain Parameters

GenSynth Explain allows GenSynth to learn more about the inner workings of the model and identify how the model makes decisions about the data (classification models) and visually identifies and illustrates the various scenarios where the model generates different results than those identified by a human labeller (object detection models).

Where “key” or “index” is mentioned below, these correspond to the means of accessing tensors from the get_next() results that you defined when you created your Dataset objects (Refer to the GenSynth Data Preparation Guide for more information). For example , if your get_next() returns a list or tuple of tf.Tensor, you will provide the index; if your get_next() returns a dictionary of keys to tf.Tensor, you will provide the key name. For some parameters you may alternatively provide a key of the dictionary returned by the update() function in the performance metric interface.

If the parameter value is both the name of a tensor or operation in the network and a key in the dictionary returned by the update() of the peformance metrics interface, the custom metrics dictionary value will be used for explainability.

All parameters are required unless otherwise indicated. The parameters available to you will be different depending on whether you selected Root Cause Analysis for Classification Models or R oot Cause Analysis for Object Detection Models.

Template

GenSynth allows you to create and save templates. You can save your settings for some or all Explain parameters as long as the required parameters have values.

Image Data Key

A key/index of the image tensor in the dataset (not a key in the return values dictionary of update() for a custom metric). Indicates image content for GenSynth Explain.

Note: For classification Root Cause Analysis, if this is a key in the return values dictionary of update(), you must ensure that it matches the key of the original image in the dataset.

Using the custom metric function allows for post-processing, e.g. colour adjustment, but, generally, the shapes should match, and positions should be preserved, since boxes or masks will be overlaid on the images.

If this refers to a tensor or op in the graph and a key in the update() return value dictionary, the value for the latter will be used for display.

Ground Truth Classes Data Key

A key/index of the ground truth box class labels in the dataset, or its key in the return values dictionary of update() for a custom metric. Indicates the class of each ground truth object box.

Class Logits Tensor

The tensor name of the logits array in the network graph, with shape (batch_size, num_classes). Typically immediately precedes a Softmax, SoftmaxCrossEntropyWithLogits, or SparseSoftmaxCrossEntropyWithLogits. Applying a softmax to the values returns the actual predicted class probabilities or confidences.

Normalize images 0-255

This toggle determines whether or not you want to normalize the image color channels into the (inclusive) range [0, 255] for display.

Default: True

Optional

Bounding Box Format

The bounding box format for object detection root cause analysis, as referenced by the Detection Boxes and Ground Truth Boxes Data Key for both detection boxes and ground truth boxes.

Acceptable values: (top, left, height, width) and (top, left, bottom, right).

Default: (top, left, bottom, right)

Bounding Box Scale

The scale of the box coordinates/dimensions, as referenced by the Detection Boxes and Ground Truth Boxes Data Key.

Acceptable values: Normalized or Absolute

Default: Normalized

Detection Classes

The name of the tensor with the detection (prediction) classes output by the network after all filtering (including non-max suppression, if applicable), or its key in the return values dictionary of update() for a custom metric.

Detection Boxes

The name of the tensor with the detection (prediction) box coordinates/dimensions output by the network, after all filtering (including non-max suppression, if applicable), or its key in the return values dictionary of a custom metric's update() function.

Detection Confidence

The name of the tensor with the detection (prediction) confidences/scores output by the network, after all filtering (including non-max suppression, if applicable),or its key in the return values dictionary of a custom metric's update() function.

Total Number of Classes

The number of object-detection classes the network supports. This is optional if Detection Classes setting is a one-hot encoding (the number of classes is available from the shape).

Number of Detections

The name of the tensor with the actual numbers of detection (prediction) boxes output by the network after all filtering (including non-max suppression, if applicable),or its key in thereturn values dictionary of a custom metric'supdate() function.

Optional if batch size is 1 or the num_box dimension is variable.

Default: None

Provide Mask Tensor

The name of the tensor whose output is a Boolean mask indicating for each detection (prediction) whether that detection should actually be used (filling from the start of the last dimension of the array), after all filtering (including non-max suppression, if applicable), or its key in the return values dictionary of a custom metric's update() function.

Should be True if the batch size is greater than 1 or the num_boxes dimension is fixed.

Default: False

Ground Truth Classes Data Key

The key/index of the tensor in the dataset indicating the true classes of each box.

Ground Truth Boxes Data Key

A key/index of the ground truth box coordinates/dimensions in the dataset or its key in the return values dictionary of a custom metric's update() function.

The coordinates must be relative to the image dimensions (so normalized into (0, 1)), assuming no boxes extend past the boundary).

Ground Truth “Is Crowd” Data Key

A key/index of the tensor in the dataset or its key in thereturn values dictionary of a custom metric's update() function, whose corresponding values indicate for each ground truth box, whether the box corresponds to a crowd and so can be matched to multiple detections.

Default: None. In this case, no ground truth box can be matched to multiple detections.

Optional

Provide Masks Data Key

This toggle indicates whether the valid boxes are determined by a number of boxes or by a mask. Depending on the setting you will see either Ground Truth Num Boxes Data Key or Ground Truth Masks Data Key.

Ground Truth Num Boxes Data Key

A key/index of the tensor in the dataset or its key in th ereturn values dictionary of a custom metric's update() function, whose corresponding values indicate for each image in the batch how many boxes are actually present (with present boxes filling from the start of the last dimension of the array).

If the shapes of the groundtruth_classes_data_key, groundtruth_boxes_data_key and groundtruth_confidences_data_key tensors are fixed, then you will probably need this or the groundtruth_masks_data_key.

Default: None. In this case, each image in a given batch is assumed to have the same number of boxes.

Ground Truth Masks Data Key

A key/index of the tensor in the dataset or a key in the return values dictionary of a custom metric's update() function, whose corresponding values are Boolean masks indicating for each index whether a box is actually present.

If the shapes of the groundtruth_classes_data_key, groundtruth_boxes_data_key and groundtruth_confidences_data_key tensors are fixed, then you will probably need this or num_groundtruth_boxes_data_key.

Default: None. In this case, each image in a given batch is assumed to have the same number of boxes.

Provide Percentage of Samples for Analysis

You must configure how many test samples are selected, either by a percentage or by absolute number of batches. The samples are selected at random for both options.

Subsample Ratio

This slider lets you select the percentage of samples you want to use for testing. This slider is only available if you enable Provide Percentage of Samples for Analysis.

Number of Batches

Enter the number of batches of samples you want to use for testing. This option is only available if you disable Provide Percentage of Samples for Analysis.