Evaluation charts | IBM Cloud Pak for Data as a Service

Evaluation charts

Evaluation charts are similar to histograms or collection graphs. Evaluation charts show how accurate models are in predicting particular outcomes. They work by sorting records based on the predicted value and confidence of the prediction, splitting the records into groups of equal size (quantiles), and then plotting the value of the criterion for each quantile, from highest to lowest. Multiple models are shown as separate lines in the plot.

Outcomes are handled by defining a specific value or range of values as a "hit". Hits usually indicate success of some sort (such as a sale to a customer) or an event of interest (such as a specific medical diagnosis).

Flag: Output fields are straightforward; hits correspond to true values.
Nominal: For nominal output fields, the first value in the set defines a hit.
Continuous: For continuous output fields, hits equal values greater than the midpoint of the field's range.

Evaluation charts can also be cumulative so that each point equals the value for the corresponding quantile plus all higher quantiles. Cumulative charts usually convey the overall performance of models better, whereas noncumulative charts often excel at indicating particular problem areas for models.

Creating a simple Evaluation chart

In the Chart Type section, click the Evaluation icon.
The canvas updates to display an Evaluation chart template.
Set the Target field, Predict field and Confidence field variables. The target field can be any instantiated flag or nominal field with two or more values. The predict field defines the variable that is used as the predicted value. The confidence field defines the variable that is used to establish the confidence of the prediction.
Note: The Predict field variable type must match the variable type that is selected for the Target field.
Specify a custom condition used to indicate the User defined hit. This option is useful for defining the outcome of interest rather than deducing it from the type of target field and the order of values.
You must specify a CLEM expression for a hit condition. For example, @TARGET = "YES" is a valid condition that indicates a value of Yes for the target field is counted as a hit in the evaluation. The specified condition is used for all target fields.
Click the Save visualization in the project control. Select Create a new asset or Append to existing asset. Provide a Visualization asset name, an optional description, and a chart name.
Click Apply to save the visualization to the project. The new visualization asset is now available on the Assets tab.

Options

Target field

Lists instantiated flag or nominal field variables with two or more values.

User defined hit

Specify a hit value. Hits indicate events of interest (for example, a specific medical diagnosis).

Predict field

Lists variables that can be used as the predicted value.

Confidence field

Lists variables that can establish the confidence of the prediction.

Cumulative plot

Create a cumulative chart when enabled. Values in cumulative charts are plotted for each quantile plus all higher quantiles.

Display mode

The settings control which charts display in preview mode and in the output.

Single mode: When selected, the Model Classification Tuning chart is in the only chart that displays in preview mode and in the output.
Classical mode: When selected, the Model Classification Tuning, Cutoff, Matrix Bar, ROC, Gains, ROI, and Profit charts display in preview mode and in the output.
Full mode: When selected, the Model Classification Tuning, Cutoff, Matrix Bar, ROC, Gains, ROI, Profit, GINI, Lift, and Response charts display in preview mode and in the output.

Evaluation charts

Cutoff: The cutoff chart shows the predicted versus actual values for selected variables for a specified cutoff value.
Matrix Bar: Matrix Bar charts are a good way to determine whether linear correlations exist between multiple variables.
ROC: ROC (Receiver Operating Characteristic) evaluates the performance of classification schemes where subjects are classified for one variable with two categories.
Gains: Gains are defined as the proportion of total hits that occurs in each quantile. Gains are computed as (number of hits in quantile / total number of hits) × 100%.
ROI: ROI (return on investment) is similar to profit in that it involves defining revenues and costs. ROI compares profits to costs for the quantile. ROI is computed as (profits for quantile / costs for quantile) × 100%.
Profit: Profit equals the revenue for each record minus the cost for the record. Profits for a quantile are the sum of profits for all records in the quantile. Revenues are assumed to apply only to hits, but costs apply to all records. Profits and costs can be fixed or can be defined by fields in the data. Profits are computed as (sum of revenue for records in quantile − sum of costs for records in quantile).
Kolmogorov-Smirnov: Compares the observed cumulative distribution function for a variable with a specified theoretical distribution, which can be normal, uniform, exponential, or Poisson.
GINI: GINI measures statistical dispersion and is intended to represent the income or wealth distribution. It is the most commonly used measurement of inequality.
Lift: Lift compares the percentage of records in each quantile that are hits with the overall percentage of hits in the training data. It is computed as (hits in quantile / records in quantile) / (total hits / total records).
Response: Response is the percentage of records in the quantile that are hits. Response is computed as (hits in quantile / records in quantile) × 100%.

Evaluation chart settings

The following settings apply only to profit and ROI charts.

Costs: Specify the fixed cost associated with each record.
Revenue: Specify the fixed revenue associated with each record that represents a hit.
Weight: If the records in your data represent more than one unit, you can use frequency weights to adjust the results. Specify the fixed weight associated with each record.