Prediction drift evaluation metric
The prediction drift metric measures the change in distribution of the LLM predicted classes.
Metric details
Prediction drift is a drift v2 evaluation metric that evaluates data distribution changes.
Scope
The prediction drift metric evaluates generative AI assets only.
Types of AI assets: Prompt templates
- Generative AI tasks:
- Text classification
- Supported languages: English
Scores and values
The prediction drift score indicates the change in distribution of the LLM predicted classes.
- Range of values: 0.0-1.0
- Best possible score: 0.0
- Ratios:
- At 0: No change is detected.
- Over 0: Increasing change is detected.
Do the math
The following Jensen Shannon distance formula is used to calculate prediction drift:
Jensen Shannon Distance is the normalized form of Kullback-Leibler (KL) Divergence that measures how much one probability distribution differs from the second probabillity distribution. Jensen Shannon Distance is a symmetrical score and always has a finite value.
is the KL Divergence.
Parent topic: Evaluation metrics