Average odds difference metric

Scope

Copy link to section

The average odds difference metric evaluates generative AI assets and machine learning models.

• Types of AI assets:
  • Prompt templates
  • Machine learning models
• Generative AI tasks: Text classification
• Machine learning problem type: Binary classification

Scores and values

Copy link to section

The average odds difference metric score indicates the difference in false positive and false negative rates for monitored and reference groups.

• Range of values: 0.0-1.0
• Best possible score: 0.0
• Ratios:
  • At 0: Both groups have equal odds
  • Under 0: Biased outcomes for monitored group
  • Over 0: Biased outcomes for reference group

Do the math

Copy link to section

The following formula is used for calculating false positive rate (FPR):

The following formula is used for calculating true positive rate (TPR):

The following formula is used for calculating average odds difference:

Parent topic: Evaluation metrics