Shortcomings of the 80% Rule - Insensitive to Sample Size

One problem with the 80% Rule is that it is insensitive to sample size (i.e., the number of things being studied). Assume that we have a group containing 3 protected members and 3 non-protected members. Further assume that all three non-protected members are selected for some positive employment outcome (i.,e., promotion, training opportunity, etc.), and two of the three protected members are selected. 

The selection ratio would be 67%, thereby inferring disparate impact under the 80% Rule. However, because of the small sample size (6 total individuals), the result could very likely have occurred by chance. The z-score in this example would be 1.08, which is well below the normally accepted standard of statistical significance. Using the 80% Rule, we would be committing a Type I error - inferring disparate impact when none exists.


Sample size cal also lead to Type II errors. Assume that we have a group of 5,000 protected members and 5,000 non-protected members. Further assume that all 5,000 non-protected members are selected, and 4,000 of the 5,000 protected members are selected.

The selection ratio here would be 80%, thereby inferring no disparate impact under the 80% Rule. However, with such a large sample size, the result is unlikely to have occurred by chance. The z-score in this example is 33.33, which is well above the normally accepted standard of statistical significance. Using the 80% Rule, we would be committing a Type II error - inferring no disparate impact when it does exist.