Feature drift in Watson OpenScale drift v2 metrics

Last updated: Jul 27, 2023
Feature drift in Watson OpenScale drift v2 metrics

Watson OpenScale calculates feature drift by measuring the change in value distribution for important features.

How it works

Watson OpenScale calculates drift for categorical and numeric features by measuring the probability distribution of continuous and discrete values. To identify discrete values for numeric features, Watson OpenScale uses a binary logarithm to compare the number of distinct values of each feature to the total number of values of each feature.

Watson OpenScale uses the following binary logarithm formula to identify discrete numeric features:

Binary logarithm formula is displayed

If the distinct_values_count is less than the binary logarithm of the total_count, the feature is identified as discrete.

Do the math

Watson OpenScale uses the following formulas to calculate feature drift:

Jensen Shannon distance

Jensen Shannon Distance is the normalized form of Kullback-Liebler (KL) Divergence that measures how much one probability distribution differs from the second probabillity distribution. Jensen Shannon Distance is a symmetrical score and always has a finite value.

Watson OpenScale uses the following formula to calculate the Jensen Shannon distance for two probability distributions, baseline (B) and production (P):

Jensen Shannon distance formula is displayed

KL Divergence is displayed is the KL Divergence.

Total variation distance

Total variation distance measures the maximum difference between the probabilities that two probability distributions, baseline (B) and production (P), assign to the same transaction as shown in the following formula:

Probability distribution formula is displayed

If the two distributions are equal, the total variation distance between them becomes 0.

Watson OpenScale uses the following formula to calculate total variation distance:

Total variation distance formula is displayed

  • đť‘Ą is a series of equidistaant samples that span the domain of circumflex f is displayed that range from the combined miniumum of the baseline and production data to the combined maximum of the baseline and production data.

  • d(x) symbol is displayed is the difference between two consecutive đť‘Ą samples.

  • explanation of formula is the value of the density function for production data at a đť‘Ą sample.

  • explanation of formula is the value of the density function for baseline data for at a đť‘Ą sample.

  • The explanation of formula denominator represents the total area under the density function plots for production and baseline data. These summations are an approximation of the integrations over the domain space and both these terms should be 1 and total should be 2.

Overlap coefficient

Watson OpenScale calculates the overlap coefficient by measuring the total area of the intersection between two probability distributions. To measure dissimilarity between distributions, the intersection or the overlap area is subtracted from 1 to calculate the amount of drift. Watson OpenScale uses the following formula to calculate the overlap coefficient:

Overlap coefficient formula is displayed

  • đť‘Ą is a series of equidistant samples that span the domain of circumflex f is displayed that range from the combined miniumum of the baseline and production data to the combined maximum of the baseline and production data.

  • d(x) symbol is displayed is the difference between two consecutive đť‘Ą samples.

  • explanation of formula is the value of the density function for production data at a đť‘Ą sample.

  • explanation of formula is the value of the density function for baseline data for at a đť‘Ą sample.

Learn more

Reviewing drift v2 results

Parent topic: Drift v2 metrics