- Published on
HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
- Authors
- Name
- DeepHorus
HackerRank has open-sourced its automated resume scoring system, known as the Hiring Agent, sparking widespread discussion on platforms like LinkedIn and Reddit. The tool, available on GitHub, promises to streamline hiring by using an LLM to parse resumes and assign scores out of 100. However, tests reveal a critical flaw: the tool produces wildly inconsistent scores for the same resume, turning hiring into a luck-based filter rather than a merit-based assessment.
According to a tech professional who tested the system, their resume scored 90/100 in an initial run, but after merely cleaning up debug print statements, the score dropped to 74/100. Further testing over 100 runs showed scores ranging from 66 to 99, meaning that if a company's cutoff is 85, the candidate would fail 65% of the time—entirely due to randomness. The tool works by parsing a PDF resume, calling an LLM six times to extract structured data, and then scoring the candidate based on categories including open source contributions (35 points), personal projects (30), work experience (25), technical skills (10), and up to 20 bonus points.
The variability is not evenly distributed across categories. Technical skills, being a simple checklist, scored 8/10 nearly every time. However, project scores fluctuated dramatically, with the LLM sometimes praising "real-world deployment" and other times citing "lack of architectural complexity." Even at a low temperature setting of 0.1, the non-determinism persisted. A GitHub issue from October showed similar inconsistency at temperature 0, underscoring that this is a fundamental design flaw, not a tunable bug.
Switching from a local model (gemma3:4b) to Gemini tightened the score distribution to 48–64, but with a cutoff of 60, candidates still fail 28% of runs arbitrarily. Work experience scoring is particularly concerning: every resume, from an intern to a principal engineer, received a perfect 25/25. The prompt for this category is merely two lines with no rubric, making consistency meaningless. In contrast, projects have a detailed rubric but yield the noisiest results—useless for consistent evaluation.
The implications for hiring are significant. If companies adopt such tools as a primary filter, qualified candidates may be rejected due to randomness, while others get lucky. The system is effective at parsing resumes into structured data—an area where LLMs excel—but fails at the subjective judgment required for scoring. As the tester noted, using an LLM to check a checklist is fine, but relying on it for nuanced evaluation introduces a luck factor.
Industry reactions have been mixed. Some applaud the transparency of open-sourcing the ATS, while others warn against over-reliance on LLM-based scoring. The developer community has raised concerns on forums about the potential for unfair hiring practices. The tool's creator may need to address these issues by incorporating more deterministic scoring methods or providing clearer rubrics for each category.
Looking ahead, HackerRank could improve the system by reducing reliance on LLM judgment for subjective categories and adding guardrails to ensure consistency. Until then, job seekers and employers should approach automated resume scoring with caution. The revelation that a candidate's fate can hinge on a random number generator highlights the need for more robust and equitable hiring technologies.
