Part 4 – Enter AI
We ended last time with a question: if doing a background check on someone can now be this fast, this automated, this far-reaching — how long before the checking itself stops being done by a person at all?
Here’s the uncomfortable answer.
It already has.
Not everywhere, and not entirely. But in enough real, documented places that it’s no longer a “what if.”
The Fear Was Aimed at the Wrong Thing
A lot of people’s worry about AI lives somewhere dramatic — machines becoming self-aware, humanity losing control, the toaster developing opinions.
It’s an understandable place for the fear to land.
It’s also, for the purposes of this chapter, not where the actual story is.
The real story isn’t a future robot uprising — it’s much quieter than that — and it’s already happened: ordinary hiring and firing decisions, made by software, in workplaces that already exist.
Hired by an Algorithm
In 2018, Amazon quietly scrapped an AI recruitment tool after discovering what it had actually learned. Trained on a decade of resumes submitted mostly by men, the system taught itself to downgrade anything associated with women — including, in one widely reported detail, penalising the literal word “women’s” wherever it appeared.
Around the same time, a tool called HireVue was being used by major employers to analyse video interviews — facial expressions, tone of voice, speech patterns — and generate an automatic score.
In 2019, a privacy watchdog filed a federal complaint against it, arguing its results were, in their words, biased, unprovable and not replicable. The company later dropped the facial analysis component, though concerns about other biometric scoring have continued.
Then it stopped being theoretical.
In 2023, the U.S. Equal Employment Opportunity Commission settled its first case built specifically around AI hiring discrimination: iTutorGroup had configured its recruitment software to automatically reject female applicants aged 55 or older, and male applicants aged 60 or older.
Not an accident of biased data — a deliberate setting.
The company paid out to the applicants affected.
And in 2025, a case against Workday — whose applicant-screening software is used by employers across the United States — reached another significant stage when a nationwide class action was certified, with claims involving race, age and disability discrimination.
The details differ.
The underlying problem doesn’t.
Once software is making the decision, the bias in the system can operate at a scale no individual human could match.
Fired by an Algorithm
Hiring wasn’t the only side of this.
In 2019, documents obtained by journalists revealed that Amazon had been using an automated system to track warehouse workers’ productivity — and, when performance dropped below a set threshold, to generate warnings and termination paperwork without a supervisor making the initial decision.
Roughly 300 people were reportedly terminated at a single facility in about a year through this system.
Amazon subsequently disputed the idea that people were fired with no human involvement at all, saying managers retained the ability to intervene.
Either way, the case became a widely cited example of exactly the future Part 1 gestured towards: a world in which software can play a decisive role in who stays and who goes.
Where This Leaves Judgment
None of this requires anyone to deliberately build something unfair.
These systems learn from history — and history, in hiring, is not a neutral teacher.
If the people historically hired, promoted, or rated highly skew a certain way, a system trained on that pattern will quietly learn to prefer it too.
Efficient, consistent, and wrong in exactly the same shape every time.
That’s a different kind of risk to the ones this series has covered so far.
Part 2 was about a company that knew someone was dangerous and looked away.
Part 3 was about a company that simply couldn’t check fast enough.
This is a third thing entirely: a system that checks instantly, confidently, and at scale — while potentially being wrong about an entire category of people, all at once, without meaning to be.
And there’s something else worth sitting with.
The EEOC’s position in the iTutorGroup case was straightforward: employers cannot escape responsibility for discriminatory decisions simply because the decision was made by software.
A hundred and some years after Ballard, “the algorithm decided” turns out to be about as strong a defence as “I didn’t know” ever was.
What South Africa’s Own Law Already Says
This is where POPIA becomes directly relevant, not just as a general privacy law, but specifically on this point.
Section 71 sets out a real restriction: broadly, a person may not be subjected to a decision that has legal consequences for them, or affects them substantially, if that decision is based solely on automated processing.
The section specifically refers to the use of a person’s personal information to create a profile of their performance at work.
In plain terms: South African law already says a machine isn’t supposed to be the only thing deciding whether you keep your job, or get one.
It isn’t an absolute rule.
There are exceptions — including certain decisions necessary for entering into or performing a contract, provided appropriate measures are in place, as well as circumstances where another law or code of conduct provides for the decision.
And there are still difficult questions around what “solely” automated actually means.
If a human technically signs off on an algorithm’s decision without meaningfully reviewing it, is that genuinely human decision-making?
Then there’s the question of explanation.
POPIA does not simply give every person a straightforward right to have an algorithm explain itself.
So the honest answer is that the law saw this coming before it became a live issue in the workplace.
Whether the legal framework is strong enough to deal with what comes next is a different question — and probably one this series will need to revisit, not close off.
So, Could AI Already Be Making Hiring and Firing Decisions?
Not “could.”
In some places, it already is — quietly, gradually, without most people noticing it happen.
The more useful question isn’t whether that’s possible.
It’s whether the human judgment these systems are quietly removing is worth keeping.
We think it is.
Where This Leaves Us
We started this series with two phone calls, a couple of people who knew you, and a decision made largely on what they said.
We end it with software that can review, score, hire, and terminate a person without ever having spoken to them.
The tools changed completely, across four chapters.
The underlying question never did.
Can I reasonably trust this person — and can I show, if I ever need to, that I took reasonable care in deciding that?
That’s exactly the space Credence sits in.
Not a black box making consequential decisions with no human judgement behind them.
Real records.
Real verification.
And a result fast enough to actually be useful, put in front of a person who still gets to make the decision.
The judgment stays with you.
We just make sure it’s informed.
SHARE THIS ARTICLE
By 1996, over half of large companies were already running background checks digitally — a strange jump for an industry that had barely existed a decade before. Once checking could happen in minutes instead of weeks, doing it slowly stopped looking like a limitation, and started looking like a choice. Part 3 traces how that shift happened, and what it quietly changed about what “reasonable care” means.
In 1908, a court held an employer responsible for a death at work — not for what an employee did, but for what the employer already knew and chose to ignore. That idea, that you’re responsible for what you should have known, turns out to be older than the background check industry itself. Here’s where it began.
There was a time when getting hired took two phone calls and a good referee. Today, background checks have changed the way we build trust in hiring. We look at what changed — and why trust had to find a new shape.