Watson OpenScale fairness metrics

• How it works: When you view the details of a model deployment, the Fairness section of the model summary that is displayed, provides the fairness scores for different groups that are described as metrics. The fairness scores are calculated with the disparate impact formula.

• Uses the confusion matrix to measure performance: No

• Do the math:

                    (num_positives(privileged=False) / num_instances(privileged=False))
Disparate impact =   ______________________________________________________________________
                    (num_positives(privileged=True) / num_instances(privileged=True))              

The num_positives value represents the number of individuals in the group who received a positive outcome, and the num_instances value represents the total number of individuals in the group. The privileged=False label specifies unprivileged groups and the privileged=True label specifies privileged groups. In Watson OpenScale, the positive outcomes are designated as the favorable outcomes, and the negative outcomes are designated as the unfavorable outcomes. The privileged group is designated as the reference group, and the unprivileged group is designated as the monitored group.

The calculation produces a percentage that specifies how often the rate that the unprivileged group receives the positive outcome is the same rate that the privileged group receives the positive outcome. For example, if a credit risk model assigns the “no risk” prediction to 80% of unprivileged applicants and to 100% of privileged applicants, that model has a disparate impact of 80%.

• Supported fairness details

  • Watson OpenScale supports the following details for fairness metrics:
    • The favorable percentages for each of the groups
    • Fairness averages for all the fairness groups
    • Distribution of the data for each of the monitored groups
    • Distribution of payload data

• Description: Fairness metric that describes the fairness for the model predictions. It is the difference between the ratio of favorable outcomes in monitored and reference groups

  • Under 0: Higher benefits for the monitored group.
  • At 0: Both groups have equal benefit.
  • Over 0 Implies higher benefit for the reference group.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

                                    num_positives(privileged=False)     num_positives(privileged=True) 
Statistical parity difference =  ________________________________ -  ________________________________
                                    num_instances(privileged=False)     num_instances(privileged=True)

• Description: Returns the difference in false negative rates for the monitored and reference groups

  • At 0: Both groups have equal benefit.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating false negative rate (FNR):

                                        false negatives         
            False negative rate  =  __________________________
                                        all positives

The following formula is used for calculating false negative rate difference:

            False negative rate difference  =  FNR of monitored group - FNR of reference group

The false positive rate difference gives the percentage of negative transactions that were incorrectly scored as positive by your model in Watson OpenScale.

• Description: Returns the ratio of false positive rate for the monitored group and reference groups.

  • At 0: Both groups have equal odds.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating false positive rate (FPR):

                                        false positives       
            False positive rate   =   ________________________
                                        total negatives

The following formula is used for calculating false positive rate difference:

            False positive rate difference  =   FPR of monitored group - FPR of reference group

• Description: Returns the difference in false discovery rate for the monitored and reference groups.

  • At 0: Both groups have equal odds.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating the false discovery rate (FDR):

                                                false positives        
            False discovery rate  =   _________________________________________
                                        true positives + false positives

The following formula is used for calculating the false discovery rate difference:

            False discovery rate difference  = FDR of monitored group - FDR of reference group

• Description: Returns the difference in false omission rate for the monitored and reference groups

  • At 0: Both groups have equal odds.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating the false omission rate (FOR):

                                                false negatives        
            False omission rate   =   ________________________________________
                                        true negatives + false negatives

The following formula is used for the false omission rate difference:

            False omission rate difference  =   FOR of monitored group - FOR of reference group                                         

• Description: Returns the difference in error rate for the monitored and reference groups.

  • At 0: Both groups have equal odds.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating the the error rate (ER):

                                false positives + false negatives        
            Error rate  =   ___________________________________________
                                all positives + all negatives

The following formula is used for calculating the error rate difference:

            Error rate difference  = ER of monitored group - ER of reference group

• Description: Returns the difference in error rate for the monitored and reference groups.

  • At 0: Both groups have equal odds.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating false positive rate (FPR):

                                        false positives       
            False positive rate   =   _________________________
                                        total negatives

The following formula is used for calculating true positive rate (TPR):

                                        True positives      
            True positive rate   =   ______________________
                                        All positives

The following formula is used for calculating average odds difference:

                                        (FPR monitored group - FPR reference group) + (TPR monitored group - TPR reference group)       
            Average odds difference  =   ___________________________________________________________________________________________

                                                                                    2

• Description: Returns the average of the absolute difference in false positive rate and true positive rate for the monitored and reference groups.

  • At 0: Both groups have equal odds.

• Uses the confusion matrix to measure performance: Yes

• Do the math:

The following formula is used for calculating false positive rate (FPR):

                                            false positives       
            False positive rate   =   ____________________________
                                            all negatives

The following formula is used for calculating true positive rate (TPR):

                                        True positives      
            True positive rate   =   ________________________
                                        All positives

The following formula is used for calculating average absolute odds difference:

                                                |FPR monitored group - FPR reference group| + |TPR monitored group - TPR reference group|      
            Average absolute odds difference  =   ______________________________________________________________________________________________

                                                                                            2