Fairness Algorithms Forget the Human Behind the Data
The promise and the removal
Removing demographic information from an algorithm sounds like the cleanest possible fix for bias. If a model cannot see race, gender, or ethnicity, the reasoning goes, it cannot discriminate on those grounds. This intuition has driven a generation of fairness engineering, and it has the feel of moral hygiene: delete the offending variable, and the harm disappears with it. A recent paper by Aditya Parikh and five co-authors, submitted to arXiv on 25 September 2026, tests that intuition directly and finds it wanting. [1] The title states the finding without ornament: invariance is not all you need for algorithmic fairness, and removing demographic information can create new bias rather than remove it. [1] The process is not mysterious once you see it. When a model loses the explicit signal, it does not become blind. It becomes a detective, hunting for proxies — zip codes, shopping patterns, language use, the company a person keeps in a dataset — that correlate with the hidden attribute. The paper’s own framing is narrower and more precise: demographic information can be recovered from latent representations even when a model is neither trained to predict demographic attributes nor trained using them. [1]. The bias does not vanish. It goes underground, where it is harder to name, harder to audit, and harder to contest. A system that once could be accused of using race now uses a variable that merely happens to track race with unsettling precision. The discrimination becomes deniable precisely because the label is gone.
This is the first way the machine makes a human role superfluous. The human who would have examined the variable, argued about its legitimacy, and been answerable for its use is replaced by a pipeline that never surfaces the question. The decision arrives without the vocabulary needed to challenge it.
What the fairness literature already knew
The broader field has been circling this problem for years. A 2020 overview of algorithmic fairness laid out the landscape: bias can arise without any intention, and the remedies fall into three families — pre-process, in-process and post-process mechanisms. [2]. Each family assumes something is known about the protected attribute. Each family, in other words, depends on the very information that privacy advocates and anti-discrimination lawyers often want kept out of the system.
By 2022, researchers were naming the trap explicitly. A paper on demographic-reliant fairness characterized the risks of collecting the sensitive data that fairness techniques require. [3]. It catalogued harms to individuals — privacy exposure, miscategorization, use of data beyond what subjects expected — and harms to communities: expanded surveillance in fairness’s name, misrepresentation of what group membership means, and the surrender of a community’s power to define for itself what counts as unfair treatment. Confronting these questions before and during data collection is what makes fairness methods more likely to reduce harm rather than entrench it.
Read together, the two strands form a vise. Collect the data, and you build the surveillance apparatus that fairness was meant to check. Remove the data, and the model invents its own proxies, producing bias that no one can see. The Parikh paper sits at the second jaw of that vise, demonstrating empirically what the 2022 work warned about conceptually.
The measurement that misses the person
A third line of work sharpens the stakes further. A 2026 paper on bridging formal and perceived fairness argues that fairness is not merely a technical property but a subjective, context-sensitive human judgment shaped by cognitive heuristics, mental models, normative expectations, and sociotechnical conditions. [4]. Users’ perceptions of fairness may diverge substantially from the fairness criteria an algorithm formally satisfies. A system can pass every technical audit and still be experienced as unjust by the people it decides about.
That gap has consequences the engineering literature tends to understate. A system that meets its formal requirements but is perceived as unjust will not be trusted, accepted, or considered legitimate. A framework that integrates computational fairness audits with user-centered assessments, built through literature synthesis, interdisciplinary workshops, and stakeholder interviews, is proposed. The goal is to guide the design of systems that support what they call informed, well-calibrated fairness judgments by those affected. [4].
The phrase is doing a lot of work. A fairness judgment that is well-calibrated is one that tracks something real — not just the metric, but the experience. And the only entity capable of making that judgment is a person, situated in a context, with a stake in the outcome. The framework is, at bottom, an argument for keeping the human in the loop not as a rubber stamp but as the source of the standard itself.
The contradiction at the center
Here is where the three papers collide, and where the angle of this article becomes sharp. The technical apparatus of fairness was built to substitute for human judgment — to make bias measurable, comparable, and correctable at scale, in ways that a room full of people arguing case by case never could. That substitution is the whole point. You cannot audit a million loan decisions by hand. You need the metric.
But the metric requires the category, and the category is exactly what the removal instinct wants to delete. So the field has constructed a machine that can only measure fairness by first encoding the very distinctions it is trying to transcend. And when it tries to transcend them — when it removes the category to avoid the encoding — it loses the ability to measure anything at all, while the bias migrates into proxies it cannot name.
The human who once held the judgment — who looked at an applicant and weighed context against pattern — is now superfluous in two directions at once. On one side, the metric has replaced the case-by-case reasoning that made room for nuance. On the other, the removal of the category has replaced the human’s ability to contest the basis of the decision with a system that no longer knows what it is doing or why. The person is gone from the process, and the process has become both more confident and less accountable.
What invariance actually delivers
The Parikh paper’s contribution is to make this concrete. The authors distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. [1] Invariance — the property of a model whose outputs do not change when a protected attribute changes — is a formal guarantee. It says: flip the race variable, and the prediction stays the same. That sounds like fairness. But the guarantee holds only for the variable you flipped. Flip a proxy instead, and the prediction moves. The model was never invariant to the thing that matters; it was invariant to the label.
This is a distinction with teeth. A system can be certified invariant and still produce disparate outcomes, because the invariance was tested on a variable the system no longer uses. The certification becomes a kind of theater — a performance of fairness that satisfies the audit while leaving the underlying pattern untouched. And the people affected by the pattern have no way to point at the mechanism, because the mechanism has been designed to be unpointable. The paper’s own conclusion is blunter than the theater metaphor: representation invariance is neither desirable nor sufficient for fairness. [1].
The paper’s authors are not arguing against fairness engineering. They are arguing against a specific, widespread, and intuitively appealing shortcut: the belief that deleting the sensitive variable deletes the sensitivity. What the work shows is that the deletion relocates the problem rather than solving it, and that the relocation is where accountability dies. [1] Their own recommendation is not to aim for no demographic encoding, but to limit encoding to the smallest level sufficient for accurate, robust, and fair prediction. [1]
The judgment that has no replacement
Step back and the shape of the loss becomes clear. What the machine has made superfluous is not the data entry, not the calculation, not the audit — those were always going to be automated. What it has made superfluous is the human act of deciding what counts. Which distinctions are legitimate grounds for a decision, which are proxies in disguise, which harms are acceptable and which are not — these are not technical questions. They are normative ones, and they require someone to stand behind an answer.
The fairness metric cannot stand behind anything. It reports a number. The invariance guarantee cannot stand behind anything either; it reports a property. Neither can say why the property matters, or to whom, or at what cost. That act of saying — of taking responsibility for a distinction — is the thing that has been quietly removed from the pipeline, replaced by a certification that no one has to defend because no one has to understand it.
The 2026 framework on perceived fairness is, in this light, an attempt to put the human back. Its authors want computational audits and user-centered assessments side by side, because they recognize that a system can satisfy the formal criteria and still fail the people it touches. [4] But the framework is a work in progress, and the gap it addresses is widening faster than the tools to close it.
The detail that shows the distance
, The Parikh paper was submitted to arXiv on 25 September 2026. [1] It is careful, technical, and narrow in the way good research is narrow. It does not claim to solve fairness. It claims to show that one popular solution does not work as advertised.
That modesty is the point. The promise of algorithmic fairness was that we could have scale and justice at once — that the machine could do what the human could not, and do it fairly. What the research now shows is that the machine can do the scale, but the justice part requires a judgment the machine cannot supply and that the removal instinct actively erases. The distance between the promise and the practice is measured in exactly this: a system that has been certified fair, that no one can explain, that no one can contest, and that has quietly taken the place of the person who would have had to answer for it. The paper’s authors put the same point in their own words: encoded demographics are, in many practical applications, necessary to obtain the most equitable and high-performing models. [1].