Fairness for a group

The Fairness for a group metric gives the model’s propensity to deliver favorable outcomes to one group over another. A group can be any group, such as age, sex, or race.

Fairness for a group at a glance

  • Description: The models propensity to deliver favorable outcomes to one group over another.
  • Default thresholds: Lower limit = 80%
  • Default recommendation: Debiased scoring endpoint that you can use in your business application for receiving debiased responses from your deployed model.
  • Problem type: All
  • Data type: Structured
  • Chart values: Last value in the timeframe
  • Metrics details available: Yes

Protected attributes

Watson OpenScale automatically identifies whether any known protected attributes are present in a model. When Watson OpenScale detects these attributes, it can automatically configure bias monitors for each attribute present to ensure that bias against these potentially sensitive attributes is tracked in production.

Sex

Watson OpenScale configures the bias monitor for the Sex attribute so that Female and Non-Binary are the monitored values, and Male is the reference value.

Ethnicity

Watson OpenScale configures the bias monitor for the ethnicity attribute so that White-caucasian is the reference value while other ethnicities are monitored values.

Marital status

Watson OpenScale configures the bias monitor for the marital status attribute so that married is the reference value and single is the monitored value.

Age

Watson OpenScale configures the bias monitor for the age attribute so that the range of ages produces actionable debiasing, for example 18-25 and 26-75.

Postal code

Watson OpenScale configures the bias monitor for the postal code attribute so that individual postal codes are scored.

Interpreting the display

Fairness alerts display in tiles on the Insights dashboard, where you can sort and filter.

By clicking a model deployment tile, you can review all metrics values over time on the model Evaluations dashboard.

By clicking a monitor tile you can view graphical representations of specific metrics.

Fairness Score for a group

The fairness score pane displays violations for a group that is below the threshold that you set.

Monitored Groups

The monitored groups pane displays the average score and score for a particular group.

Schedule

The Schedule pane shows the Last evaluation and Next evaluation times. Quality metrics are evaluated every hour. You can force evaluation by clicking Check fairness now. You can also add feedback by clicking Make a scoring request.

Do the math

                          (% of favorable outcome in monitored group)
Disparate Impact Ratio =  ____________________________________________
                          (% of favorable outcome in reference group)