About this story: This is a fictional composite. It illustrates a common risk and is not a report about a particular employer.
The feature was already switched on
A small Irish recruitment agency upgraded its applicant-tracking system. The new package could read CVs, compare them with a job description and rank candidates. Recruiters still made the final call, so the team assumed the feature was harmless.
Three months later, a recruiter noticed that experienced candidates returning after career breaks regularly appeared near the bottom. Nobody could explain the ranking. The vendor’s sales page promised better matches, but the agency had not reviewed the instructions, tested the results or recorded when staff should ignore the score.
Why recruitment is different
AI that helps decide who gets access to employment can fall into the EU AI Act’s high-risk categories. The exact classification depends on the system’s intended purpose and how it is used, but this is not an area where “the vendor handles compliance” is a safe assumption.
The agency was using the system—making it a deployer in AI Act language. It still needed to understand the tool, follow relevant instructions, assign human oversight and keep appropriate records. GDPR also mattered because CVs contain personal information and automated ranking can affect people significantly.
The hidden business risk
If the ranking was poor, the agency could reject strong candidates, send weaker shortlists to clients and reproduce patterns hidden in historic data. A candidate complaint could force the agency to explain a process it had never documented. “The software gave them a low score” would not be a convincing answer.
There was also a commercial risk. The agency sold judgement and trust. An unexplained tool making part of that judgement could damage both.
What the team changed
The agency paused automatic ranking while it investigated. It asked the vendor for the intended purpose, limitations, testing information, oversight instructions and data-retention terms. It then tested the tool with a set of known CVs and compared its rankings with experienced recruiters.
The new process was easy to explain:
- No candidate was rejected only because of an AI score.
- A trained recruiter reviewed the underlying CV.
- Staff recorded when AI influenced a shortlist.
- Candidates received clear information about how their data was used.
- Unexpected patterns were reported and reviewed.
The firm also documented who could switch the feature on, change its settings or approve it for a new type of role.
What if they had carried on?
The likely cost was bigger than software fees: unfair outcomes, candidate complaints, damaged client confidence and a rushed compliance project later. Depending on the facts, regulatory duties and enforcement could also apply. Early review gave the agency room to test the feature without a live incident forcing the timetable.
Plain-English takeaway
A person clicking “approve” is not automatically meaningful human oversight.
The reviewer must understand what the score means, be able to challenge it and have enough information and authority to make a different decision.
Your first three actions
- Check whether recruitment, HR or workforce software ranks, filters or recommends people.
- Ask the vendor what the system is designed to do and what its known limits are.
- Write down who reviews the result and when they must override or stop using it.
Official sources: EU AI Act, Articles 6, 26 and Annex III ↗ · Irish DPC guidance on automated decisions ↗