How to counteract bias in AI-driven hiring
How wrong can things go when a company starts to use AI? Workday, providers of an HR and accounting platform, may be on their way to finding out, after a judge in California ruled at the end of May that a lawsuit against their AI-enabled hiring technology can proceed.
The suit alleges that Workday’s AI discriminated against applicants based on their age, but the claim is part of a larger issue with AI that has been roiling over the course of years: this technology seems plagued by entrenched biases, as became apparent in Amazon’s notorious and eventually failed attempt to develop an AI for their own hiring processes between 2014 and 2018. In the case of Amazon the issue was that their AI consistently exhibited a bias against hiring women that proved fatally resistant to efforts to make the system oblivious to any notion of gender. This makes sense when we consider that the AI was simply finding ways to recapitulate what it saw in its training data – a world in which opportunities were always skewed towards men, in ways that were both overt and also hidden in chains of correlations which in the end trace the very complexities that make concepts at the root of human bias, like “gender”, so much more than discrete and independent features.
But Amazon’s attempt at applicant ranking technology was born and died at a moment in the history of AI where the prevailing models were classifiers which mapped from features in data to categorical outputs like “good candidate” or “bad candidate”. These models are known to be susceptible to hidden correlations between the features they learn to use as a basis for their predictions. What might seem surprising today with the situation at Workday is that this problem of bias remains thoroughly entrenched in the application of AI towards decision-making processes like hiring, at a point in time when “generative AI” is allegedly making the technology increasingly responsive to some of the human-level nuance that is required to make such processes more fair and thoughtful. That said, maybe the problem is precisely that the technology is now, more than ever, simply imitating certain human-level behaviours.
Why can’t AI lose its biases and get hiring right?
So, what keeps going wrong for AI tasked with reviewing job candidates? And why isn’t it enough to just say to a generative AI, which is supposed to react in a dynamic way to natural language instructions, “hey don’t be biased”?
There are at least two shortcomings of AI when it comes to making value judgements, both of which are well-known but also seemingly insurmountable. The first problem is that generative AI, despite what it’s called, is not actually creative. In fact technically speaking the term “generative” refers to the mathematical framework underpinning this technology: a generative model is one which models a joint probability distribution over an observed and target variable, as compared to a discriminative model, which assigns a probability to a target based on the condition of an observation. It’s like the difference between saying “what are the chances that we will observe that it’s raining in London today and also that Stephen is carrying his umbrella” versus saying “given the observation that Stephen is carrying his umbrella today, what are the chances that it’s raining in London”. Discriminative models are good for classifying data, and are the type of model that was considered state-of-the-art around the time of Amazon’s foray into applicant rating technology.
Generative models on the other hand are particularly good for extrapolating data, because they in principle can make predictions across all possible worlds. Most language models – so, models which assign a probability to the next symbol in a sequence, like a sequence of words – are generative models, including the large language models that are the basis for generative AI. But these are still data-driven models, and so, while they can assign probabilities to unobserved sequences, and in this way output new sentences that might superficially seem to reflect “new” ideas, they are in fact primed to keep coming up with different ways of making output that is similar to the things they’ve already seen. In fact they are specifically geared to find ways to return to the mean, to move from the unexpected towards what is already known. An extreme example of this tendency is exposed in some types of “jailbreak” attacks on LLMs, where bombarding them with lots of weird context – for example extensively repeated sequences of the same characters – results in them suddenly revealing artefacts directly from their training data, which can include personal data like names and addresses.
Thus if the patterns observed in training data are laced with underlying biases, these biases will keep coming up in subsequent model output, even if they seem like they’re expressed in new ways. To put it in the context of hiring decisions, if an AI that has seen lots of examples of certain categories of people being selected in the hiring processes represented in its training data, for instance a preference for younger people, these categorical preferences will continue to surface in the seemingly generative output, regardless of whether the AI gets prompted with a sentence like “by the way pay no mind to an applicant’s age”. Ironically, it’s the generative models which turn out to discriminate.
The second problem with using AI to make decisions is that there is no reasoning behind the evident decision-making process. Sometimes the term “black box” is used to refer to the way that the inner workings of a model result in an output, which is accurate in that the tangle of parameters defining a model are inscrutable, but it also gives the false impression that the thinking that an AI does is unknowable, which is wrong; the fact is that the AI doesn’t “think” at all. When we describe ourselves as “thinking about something”, the word about does some important work, because our thoughts are grounded in representations of things that are or could be in the world. This “aboutness”, or intentionality to use a more technical term, is what’s on the other side of the expressions we use to communicate with one another linguistically, and the assumption that other humans have thoughts and feelings underlying their statements is a very important part of being human.
A recent trend in generative AI has involved the advent of “reasoning” models, which are said to do more “thinking” in order to arrive at their outputs. And a related idea that is emerging in this space is the notion that sets of AI agents can be “orchestrated” into chains of processes, with different agents responsible for different steps in a problem-solving plan. The theory behind both reasoning models and orchestrations of agents is that passing model outputs around in some way simulates the process humans undergo when they think through problems and possible solutions, a process which itself in reality involves our ability to simulate possible worlds in our minds. In practice, extending modelling context by injecting intermediate runs just gives the models more chances to return to something that resembles whatever counts as “typical” in the data that they encountered during training. The “chains of thought” that these models are supposed to exhibit are in fact neither thoughts nor chained; they are simply maps of pathways back towards the outcomes that are most likely because they reflect what has already been observed.
An effective group of humans – what might be referred to in certain circumstances as a “team” or a “committee” – is more than the sum of its parts, with different members collaborating to discover ideas and solutions that no individual could have produced. Members of a team use each other and the world itself to test hypotheses they form dynamically in the course of understanding a problem and coming up with a plan to solve the problem. A concatenation of AI contexts and an orchestration of AI models, on the other hand, is more akin to the game children play where a message is passed in whispers from ear to ear, altering a bit with each transmission, until, after a number of turns, the original message has mutated into something predictably nonsensical. The difference is that with AI, well-meaning instructions get transmuted across parameters and auto-regressive strings of tokens into something disappointingly reflective of the biases inherent in the underlying training data.
We’re all creative non-geniuses
Here is a conjecture regarding generative AI: data-driven models are well equipped to imitate the data they are trained on, which, in the case of LLMs and any models that are conditioned to reconstruct massive, unannotated corpora of digital text, is many instances of humans communicating with one another. The kind of tasks that communication imitators are good at performing involve situations where it is necessary to explore different ways of expressing something that is already known, so things like summarising a long document, coming up with ten different ways of expressing the content of a sentence, responding to questions about information that was already present during model training, or making lists that pertain to what would be considered general and public knowledge. Generative AI models are less well equipped to deal on their own with tasks that require lots of analysis, where new meaning needs to be made. This is a generalisation of the lack of creativity these models exhibit, as evident in the specific example of the way they simply restate established biases when looking at job applicants.
But if we step back to take a look at how humans make meaning, it turns out that it happens in at least a couple different ways. On the one hand we have the paradigmatic lone genius, someone who has attained, through a combination of experience and native intelligence, a visionary ability to discover incredible new ways of thinking about things. This type of meaning-making involves Creativity with a capital-C, the type of mastery we associate with great works of art and scientific paradigm shifts. On the other hand, though, we have situations where humans make new meaning in the process of communicating with one another in order to get aligned on some mundane problem, like deciding where to go for dinner, or arranging a trip together. Communication is necessary in these situations because the different stakeholders don’t have advanced knowledge of each other’s expectations and requirements (arguably all linguistic communication is predicated on this need to align initially obscure perspectives). Once the stakeholders begin to communicate, they discover previously unknowable conjunctions and disjunctions in ways of viewing the task at hand, and meaningful possibilities begin to emerge in the space between the stakeholders, exposed by the words they use to explore the world as it could be. And so meaning is made.
Collaborating to pull together a solution to a group planning problem very often doesn’t require any particular brilliance or specialism – you don’t need to be a genius to plan a birthday party or schedule a meeting – but it does involve a good deal of tedium, because a lot of repetitive work using linguistic communication to gradually align a lot of different points of view is required, and this is boring. At TenFive AI, one of our fundamental hypotheses is that generative AI can play a useful role in solving these types of group planning problems, by unloading the tedium of exploring different ways of saying fairly straightforward things in order to find meaningful solutions to problems that don’t really involve deep reasoning, but that are nonetheless worth solving.
A solution to a problem worth solving
Selecting job candidates in a way that eliminates bias is without a doubt a problem that is worth solving. The question for us at this stage is, is this a problem that necessarily requires deep lone-genius type creativity, or is it more like a problem that requires the use of communication to pull a solution out into a space where it can be transparently interrogated? We’re pretty sure it’s the latter, and, this being the case, we’re confident that there is in fact an AI-enabled approach to communicating about the candidate review process that eliminates rather than reinforces biases that are entrenched in existing data. In what follows, we’ll explore what a TenFive solution to candidate selection would look like, and how this solution resolves the biases that can creep into automatic candidate ratings.
A theoretical touchpoint for us is the idea that natural language has come about primarily for the purpose of communicating in order to achieve alignment in a chaotic environment. This is a claim with evolutionary and anthropological implications, and there is good, deep discourse on this topic which we won’t engage with here, but hopefully the idea that language is for talking about getting things done isn’t too controversial as a starting point. From there, our next move is to engage with data-driven AI in a way that takes into account that they are trained to imitate humans involved in communicating about joint goals. In practical terms, we set up an information processing structure composed of what we call Agent Committees, so groups of LLMs organised to negotiate and collaborate on a specific topic.
The idea of agentic AI has been on trend for some time now: this refers to networks of specialised AI agents who are able to recruit one another in the course of accomplishing some greater goal. The idea is that any agent in one of the orchestrations of agents mentioned above has some specific objective that it’s good at autonomously pursuing. In the context of a hiring pipeline, there might be an AI dedicated to candidate background checks, and another agent dedicated to contacting prospective candidates (we’re not being original here – these are exactly the examples Google uses in the presentation of their “A2A” protocol for orchestrating networks of agents). If we have confidence in each agent’s ability to do the thing it claims it can do, we can chain the agents together into a linear process for accomplishing a task with a few different parts. So with the right architecture, an orchestration of agents can be nifty, but it’s not going to do anything particularly creative or original, and there is no sense in which the agents are a gestalt, which is to say, they are never more than the sum of their parts. There certainly doesn’t seem to be scope for a chain of agents with different specialisms overcoming biases in training data in order to review job applications in a thoughtful way motivated by the unique requirements of a particular employer.
A very big difference between orchestrations of agents and a TenFive Agent Committee is that, where an orchestration is a hierarchical process of delegating tasks and receiving output, a committee is a dynamic and collaborative structure where there is give and take, back and forth. With this in mind, for an orchestration, natural language is a barrier, imprecise and so susceptible to misinterpretation and transmission error; an Agent Committee on the other hand thrives on the infinite communicative possibilities afforded by the compositionality of natural language, using words to explore ways the world could be. This is just what humans do with language, and language is a big part of the way that humans can discover a new, creative solution that may transgress the established expectations of each of the individuals involved in a collaborative problem-solving exercise.
This is exactly what we’ve designed our Agent Committees to do at TenFive: transcend the shortcomings that are endemic in a single data-driven model. Here’s how it can work for a candidate selection process:
- We take as a starting point a situation where the initial stage of a recruitment process has resulted in a large number of qualified applicants. So we have a big stack of CVs representing a group of potential good candidates for the job.
- At the company making the hire, there will be a range of stakeholders with an interest in finding the right candidate. This may include people like a hiring manager, an HR representative, a person from upper management, or a future teammate of the successful applicant. These stakeholders are notified that their input on ideal candidates is requested.
- Each stakeholder is guided through a straightforward information gathering process in which they independently offer their expectations and requirements for a successful applicant. This happens asynchronously, at each stakeholder’s convenience, without the need for discussion between stakeholders.
- A TenFive Agent Committee is formed, with each stakeholder represented by an AI advocate, and additional support from our TenFive monitoring and refereeing agents who keep track of the conversation and keep things moving. The Committee discusses applications comparatively and in detail, paying close attention to all the information provided by each applicant. Tensions and dynamics between different stakeholders are revealed through dialogue, and solutions are discovered by using language to gradually find a way towards a consensus. The crucial “aboutness” of the conversation between agents is maintained by the informational structures that we’ve developed at TenFive.
- Over the course of the conversation, a ranking of applicants emerges, based on the inputs from all the agents on the Committee and so taking into account different perspectives of all stakeholders. This process would be exhausting for a group of humans – but the AIs are immune to data fatigue and so will apply the same level of attention to every application, no matter how many there are and regardless of the order in which they are presented.
- In addition to a ranking of top candidates, the TenFive Agent Committee provides a clear explanation of how decisions were reached. This is possible because the Committee’s deliberations, due to their basis in natural language dialogue, are completely transparent and auditable. So for any given application, a balance of opinions is apparent across specific qualities of a candidate in the context of the overall role description as well as the specific expectations of each stakeholder.
A TenFive Agent Committee is specifically impervious to underlying biases in data, because each participant in the committee is configured to continually bring forward the explicit intentions of a stakeholder or a dedicated role such as monitoring or refereeing. The executions of stakeholder preferences are evident as interpretable linguistic output. This means that when apparent reasons and meanings emerge from a Committee, they arise in the language, in the communicative space between the agents, not within the impenetrable information processing of the parameters internal to any particular model.
The outcome of an Agent Committee is therefore first of all a decision which can be taken, ignored, or updated by adjusting preferences and running the Committee again. Secondly, the outcome includes a clear indication of how a decision has been made, which means that the decision can be explored for any unintended bias with confidence. In this way, by leveraging language processing technology as a tool for facilitating communication and discovery, we can make processes like hiring easier, more fair, and more transparent.