Fisher's Exact Test: Alternative to 80% Rule

Considerable time has been spent discussing the shortcomings of the 80% Rule. We know:

1. The 80% Rule is insensitive to sample size, which can lead to Type I and Type II errors
2. The 80% Rule is insensitive to the size of the disparity
3. The 80% Rule is highly sensitive to the framing of the question
4. The 80% Rule is subjective and lacks a statistical basis

What alternatives are available? For "employment decisions" questions (i.e., hiring, promotion, termination, etc.) one alternative is the Fisher's Exact Test.

The Fisher's Exact Test is a test of statistical significance used to study categorical data when employees are classified two different ways, such as (1) "protected" or "non-protected" and (2) "selected" or "not selected". Assume that we have a population of 100 and that this population is one-half "protected" and one-half "non-protected". Further assume that of the 50 "protected" individuals, 10 are "selected" and of the 50 "non-protected" individuals, 30 are "selected":



The underlying assumption is that under a "protected status-neutral" selection process, the selection rates for "protected" and "non-protected" individuals would be the same. In this example, the overall selection rate is 40%. We see, however, that the protected selection rate is 20% and the non-protected selection rate is 60%. Can we infer from this that the selection process was not "protected status-neutral"?

To answer this question, we can use the the Fisher's Exact Test to compare the actual and expected selections of "protected" individuals. Under a "protected status-neutral" selection process, we would expect that the selection rate among "protected" individuals would be 40%. In other words, the likelihood of being selected, regardless of protected status, is 40%. We statistically compare the expected 40% selection rate with the 20% actual selection rate via the Fisher's Exact Test, and find that there is a shortfall of 10 "protected" individuals selected (20 expected selections minus 10 actual selections). This shortfall is statistically significant at 3.77 units of standard deviation.

The Fisher's Exact Test is based on exact probabilities from the hypergeometric distribution, rather than relying on large sample approximations as in the Chi Square test.  This makes the Fisher's Exact Test particularly useful for small samples.