Fairness for a group
The Fairness for a group metric gives the model’s propensity to deliver favorable outcomes to one group over another. A group can be any group, such as age, sex, or race.
Fairness for a group at a glance
- Description: The models propensity to deliver favorable outcomes to one group over another.
- Default thresholds: Lower limit = 80%
- Default recommendation: Debiased scoring endpoint that you can use in your business application for receiving debiased responses from your deployed model.
- Problem type: All
- Data type: Structured
- Chart values: Last value in the timeframe
- Metrics details available: Yes
Watson OpenScale automatically identifies whether any known protected attributes are present in a model. When Watson OpenScale detects these attributes, it can automatically configure bias monitors for each attribute present to ensure that bias against these potentially sensitive attributes is tracked in production.
Watson OpenScale configures the bias monitor for the Sex attribute so that
Non-Binary are the monitored values, and
Male is the reference value.
Watson OpenScale configures the bias monitor for the ethnicity attribute so that
White-caucasian is the reference value while other ethnicities are monitored values.
Watson OpenScale configures the bias monitor for the marital status attribute so that
married is the reference value and
single is the monitored value.
Watson OpenScale configures the bias monitor for the age attribute so that the range of ages produces actionable debiasing, for example
Watson OpenScale configures the bias monitor for the postal code attribute so that individual postal codes are scored.
Interpreting the display
Fairness alerts display in tiles on the Insights dashboard, where you can sort and filter.
By clicking a model deployment tile, you can review all metrics values over time on the model Evaluations dashboard.
By clicking a monitor tile you can view graphical representations of specific metrics.
Fairness Score for a group
The fairness score pane displays violations for a group that is below the threshold that you set.
The monitored groups pane displays the average score and score for a particular group.
The Schedule pane shows the Last evaluation and Next evaluation times. Quality metrics are evaluated every hour. You can force evaluation by clicking Check fairness now. You can also add feedback by clicking Make a scoring request.
Do the math
(% of favorable outcome in monitored group) Disparate Impact Ratio = ____________________________________________ (% of favorable outcome in reference group)