You are being asked to approve a facial AI feature. The demo works, the vendor dashboard shows strong overall accuracy, and the launch team is ready. Before you sign off, ask the question that the average cannot answer: which users are hidden inside that number?
Joy Buolamwini’s work turns that question into a practical product discipline. You need to define the task precisely, measure performance at meaningful intersections, connect each error to its consequence, and decide what evidence is required before launch. That is how bias becomes a release decision rather than a vague ethical concern.
An average accuracy score can hide the failure that matters
Buolamwini encountered the problem directly in 2015 while building a graduate project at the MIT Media Lab. Facial analysis software struggled to detect her dark-skinned face, yet detected it more reliably when she wore a white mask. What looked like an isolated technical failure became a testable product question: does performance change depending on whose face enters the system?
The 2018 Gender Shades study, developed by Buolamwini with Timnit Gebru, evaluated commercial gender-classification systems across combinations of gender and skin tone. In the evaluated systems, error rates for darker-skinned women reached 34.7%, while the highest reported error rate for lighter-skinned men was 0.8%.
Those numbers are not a universal benchmark for every facial AI product. They describe particular systems and an evaluation conducted at a particular time. They also concern gender classification, not every task casually placed under the label facial recognition. Use the methodology as a leadership lesson, not the figures as a substitute for testing your own system.
That distinction matters because facial AI includes several different jobs. Face detection asks whether a face is present. Facial analysis assigns an attribute or category. Verification asks whether a person matches an enrolled identity. Identification searches for a match among multiple identities. Each job has different errors, consequences, and acceptable safeguards.
A single aggregate score can remain high while one intersection of users experiences a much worse product. Large, well-represented groups dominate the average; smaller or poorly represented groups can disappear inside it. Buolamwini also found that commonly used evaluation benchmarks were unevenly composed of lighter-skinned people, making important disparities easier to miss.
Your review should therefore contain both the overall result and the disaggregated results. Ask for the worst-performing relevant intersection, its sample size, its error type, and its distance from the release threshold. If a team cannot produce that view, it does not yet know whether the system works equitably.
Treat the coded gaze as a product-system problem
Buolamwini calls the forces embedded in computational systems the coded gaze. The idea does not require a developer to consciously encode prejudice. Unequal outcomes can emerge from ordinary product choices about data, labels, objectives, evaluation, and deployment.
- Problem framing: The team decides what the model should infer and whether that inference is necessary. If an attribute is not required to deliver the user outcome, removing it may be safer than trying to improve it.
- Training data: The examples available to the model shape which faces and conditions it learns well. Representation must cover the people and environments the product will actually encounter.
- Labels and objectives: The categories selected by the team determine what counts as correct. A technically convenient label is not automatically a legitimate product decision.
- Evaluation: The benchmark determines which failures become visible before launch. An aggregate result can certify a system while concealing a concentrated failure.
- Deployment: The workflow determines who bears the error. The same model output has very different consequences when it sorts photos, unlocks an account, or contributes to a consequential identity decision.
This framing changes ownership. Bias is not a defect that can be handed to the machine-learning team after model training. Product defines the use case and acceptance criteria. Design owns recovery paths. Engineering controls implementation. Data teams construct evaluations. Legal and privacy specialists assess sensitive-data handling and jurisdiction-specific obligations. Operations and support see the failures that escape into production. An accountable launch joins those decisions rather than treating fairness as a final review.
The same responsibility applies when you buy instead of build. A vendor’s headline accuracy is not enough. Request task-specific, group-level results under conditions resembling your deployment; definitions for every reported metric; sample sizes for relevant cells; known limitations; model-update practices; and an incident-escalation path. If demographic testing cannot be disclosed or independently verified, record that uncertainty as a product risk rather than converting it into an assumption of safety.
Build an intersectional release gate before launch
The most useful operational response to Gender Shades is an intersectional release gate. Do not wait for a complaint to decide which groups, metrics, and consequences matter. Write those conditions before you see the results, when a disappointing number is less likely to be rationalized away.
- Name the exact task. State whether the system detects, classifies, verifies, or identifies. Document the output, the downstream decision, and whether another component can transform a small model error into a serious user consequence.
- Map who is exposed. Identify the populations expected to use or be evaluated by the system. Include meaningful intersections rather than checking one demographic dimension at a time. A result for gender and a separate result for skin tone would not have revealed the disparity found at their intersection.
- Test the deployment conditions. Use conditions that resemble the real product: expected devices, image quality, lighting, camera position, and user behavior. Demographic coverage does not compensate for an evaluation environment that is unrelated to production.
- Report the distribution, not only the mean. Show the overall metric, every relevant group result, the worst-performing cell, sample sizes, and uncertainty. Separate error types because false matches, false rejections, misclassifications, and failures to detect create different harms.
- Set the decision rule in advance. Define what passes, what triggers remediation, and what blocks release. The threshold should reflect the consequence of the use case; no universal percentage can make a low-stakes photo feature and a high-stakes identity workflow equivalent.
- Design the fallback at the same time. Specify what users can do when the model fails, who can override the output, how an error can be contested, and how the event enters monitoring. A fallback that exists only in a policy document is not a usable recovery path.
- Reopen the gate after change. Repeat the sliced evaluation when the model, vendor version, input pipeline, target population, or operating environment changes. A launch result is evidence about one configuration, not a permanent property of the product.
There is an important constraint: demographic evaluation can require sensitive data. Do not create an unmanaged privacy risk in the name of fairness. Decide with privacy and legal specialists which attributes may be collected, how consent and access will work, how long the data will be retained, and whether an independent evaluator can reduce exposure. If a lawful and responsible evaluation is impossible, that is evidence against deploying the system in a consequential workflow, not permission to declare the risk unknown and proceed.
Match each safeguard to the cost of a wrong answer
Fairness metrics do not make the product decision for you. The same percentage can be tolerable in one workflow and unacceptable in another. Start with the user consequence, then choose the metric and safeguard that expose it.
| Facial AI task | Failure to inspect by group | Product safeguard |
|---|---|---|
| Face detection | The system repeatedly fails to find a face | Provide another input method and stop forcing the user through identical retries |
| Identity verification | A legitimate user is rejected, or another person is incorrectly accepted | Offer a secure non-biometric recovery route and use additional verification where the consequence warrants it |
| Identification | The wrong candidate is returned, or the correct person is absent | Do not make a consequential decision from the match alone; require corroboration and a contestable review |
| Attribute classification | A person is assigned the wrong category | Remove the inference when it is not essential, and prevent uncertain output from silently driving a high-stakes action |
Human review is not automatically a safeguard. A reviewer needs to see the model’s limitations, relevant context, and uncertainty. They also need authority to reject the automated recommendation. If the interface presents an output as authoritative, a nominal human in the loop may simply approve the machine’s mistake.
Production monitoring should preserve the same discipline as pre-launch testing. Log failure types, recovery attempts, user complaints, overrides, and downstream outcomes in a form that can be examined by relevant group where collection is lawful. Watch for a gap that grows after a model update or a change in the user population. Aggregate uptime can be healthy while one set of users is trapped in the recovery flow.
For high-consequence uses, the safest control may be a narrower role for the model. Let it assist with triage or surface candidates without granting it final authority. If a false match could affect a person’s access, opportunity, safety, or rights, preserve a non-biometric path and a meaningful way to challenge the result.
Make failures visible enough to change the decision
Numbers are necessary, but they do not always communicate what failure feels like. Buolamwini founded the Algorithmic Justice League in 2016 and used the spoken-word project AI, Ain’t I a Woman? to demonstrate facial-analysis failures involving prominent Black women, including Oprah Winfrey, Michelle Obama, and Serena Williams. The technical disparity became concrete and difficult to dismiss as a minor variance in a dashboard.
You can bring the same clarity to a launch review without exposing users. Pair the scorecard with a privacy-safe failure narrative: what the person attempted, what the system returned, which recovery path appeared, how many additional steps followed, and what would happen if the user could not recover. Use consented or appropriately protected examples, and never turn a person’s biometric failure into a presentation prop.
A strong executive review should fit the decision into five views: the exact task, the intersectional performance table, the worst credible user consequence, the proposed fallback, and the named owner who can pause deployment. That pack makes it harder for an attractive overall score to outrank evidence about who will be failed.
Key takeaways
- Do not treat facial detection, analysis, verification, and identification as interchangeable; evaluate the task you will actually deploy.
- Keep the overall metric, but require results for relevant demographic intersections, including sample sizes and distinct error types.
- Set thresholds, escalation rules, and fallback paths before the team sees the results.
- Connect every model error to its downstream product consequence; severity comes from the workflow, not the percentage alone.
- Require equivalent evidence from vendors, and treat unavailable evidence as uncertainty rather than proof of acceptable performance.
- Repeat the evaluation after material changes to the model, input pipeline, user population, or deployment environment.
Take the next facial AI decision on your calendar and replace its single accuracy tile with four artifacts: a precise task definition, an intersectional scorecard, a consequence map, and a named recovery path. If the available evidence cannot support those artifacts, the product is not ready for approval. It is still unevaluated.
References








