How (not) to Incorporate Humans into LLM-evaluation?

Almost everyone building with LLMs has arrived at the same conclusion: a single-prompt LLM-as-a-Judge does not lead to good evaluation measures due to circularity. The common fix is to incorporate a human back in the evaluation. But what is often overlooked is that how you incorporate the human determines whether you get a real quality signal or just expensive theater. Most evaluation tools today add humans in one of two ways, each with its own issues.

1. The Human Verifies and Corrects the Judge

The LLM-judge scores an output, a human reviews the score, and the human corrects the ones they disagree with. Often those corrections are used in a feedback cycle, for example as few-shot examples, so the judge “learns” from the human.

The problem is that most humans are prone to anchoring biases. Showing the human the machine’s answer before they form their own opinion will not lead to independent judgment. Anchoring, the tendency to over-rely on a starting value, goes back to Tversky and Kahneman (1974). More to the point, Agudo and colleagues (2024) tested exactly this configuration and found that the standard sequence, where the system shows its assessment first and the human merely confirms or modifies it, usually leads to blind agreement with a judge, even when it is wrong. Moreover, when overburdened or under time pressure, humans tend to over-rely on the AI proposal and you get rubber-stamping.

Worse, when those anchored corrections are fed back into the judge, the bias contaminates the very signal that was supposed to correct it. This is the Rubber-Stamp Effect (Judge Trope #12, Dietz et al. 2025): a lack of critical oversight when humans blindly trust LLM labels.

2. Collect an Independent Golden Test Set

The second pattern is safer, but very slow and tedious for humans. Here the human grades a set of examples independently, before seeing any machine score. Those grades become a benchmark, and you tune the judge prompt until its agreement with the human labels is high enough, or use it to measure drift.

Grading first removes the anchoring effects, which is a real improvement. But it replaces one problem with another: the human is now completely unsupported. Humans are not good at providing a single numerical score that captures how well lengthy responses compare to a complex rubric. This is Black-box Labeling (Judge Trope #13, Dietz et al. 2025): when the criterion is complex, the label becomes hard to produce and hard to interpret. This is a high-effort, high-variance judgment task with no structure to lean on.

Two human annotators will arrive at different judgments as they interpret the annotation rubric differently, or overlook relevant pieces of information. Shankar and colleagues (2024) documented this precisely and named it criteria drift: people refine their criteria as they grade, and some criteria turn out to depend on the specific outputs they happen to see, rather than being definable in advance. So the benchmark for calibrating against is full of human biases, which are not just random noise. Moreover, it is a waste of human expertise, and it makes the task daunting enough that people disengage from it. Among the few exceptions are highly paid assessors such as those hired by NIST for Text REtrieval Conference (TREC) campaigns.

Why This Fails

While one approach inadvertently restricts humans too much, the other underspecifies the task and does not enable the human. Neither plays to the strength of the human experts. So the human safeguard is ineffective and comes at a cost you cannot justify.

The way out is not a cleverer judge prompt. It is changing how we ask the human for their expertise.

3. Humans Identify Relevant Information, Use AI for Heavy Lifting

We are working on a better way to enable and support human experts, while avoiding anchoring and rubber-stamping. One approach is to use Nugget-based LLM-Judges (RUBRIC, Auto-ARGUE) where human experts determine which pieces of information are important for each request, then use the LLM to scan large volumes of AI output and conduct the linguistic matching of nuggets to text. This method has been around for over two decades, and with NLP technology and LLM, it is easy to scale.

But we do not rest here. We have some ideas on how to support a human in creating the nuggets, while letting them take the initiative. At the same time we help them avoid reading similar responses again and again by pre-aligning already identified nuggets to the responses (Farzi et al. 2026).

The goal is to enable humans to genuinely provide accountable oversight of automated evals: how to let people author and audit evaluation artifacts without being blindsided, and without sliding into the rubber-stamping trap that the correction loop walks straight into. If you are running into this on your own systems, building evals you are not sure you can trust for shipping decisions, I would like to compare notes. The fix is not a better prompt. It is a better division of labor between the human and the judge.

References:

Series on Vulnerabilities and Failure Modes of LLM-as-a-Judge, and Series on using LLMs for Evaluation: https://www.cs.unh.edu/~dietz/publications/