Consequence from the field guide: eval-first, in CI, against a golden set, from the first commit. Prefer programmatic gates. Treat the judge as an instrument to calibrate, never as an oracle to trust. With EU AI Act enforcement arriving August 2026, this becomes an audit trail, not just hygiene.

How it's done

You already made this leap for services: you stopped promising "correct" and started promising "correct with probability p, within budget." Apply it wholesale. You cannot assert output == expected against a non-deterministic system; you assert P(good) ≥ threshold over a distribution of inputs. Evals are the SLO and monitoring layer.

Mechanics: assemble a golden dataset of representative inputs (and expected outputs or rubrics). Score each with programmatic metrics where ground truth exists (exact match, schema validity, a passing test) and LLM-as-judge where "correct" is a property of prose (relevance, faithfulness, tone). Run the suite as a CI gate on every change, and run a sampled online eval (1–5% of production traffic) as your monitoring layer. Manage an explicit error budget — you will never hit 100%.

Tools available

• DeepEval — pytest-style, CI-native; covers RAG, agents, multi-turn; G-Eval for custom criteria, DAGMetric for multi-step scoring. Good default PR gate.
• RAGAS — RAG-specific metrics that cleanly separate retrieval quality from generation quality; good for ongoing RAG monitoring.
• Promptfoo — matrix testing across prompts/models; also doubles as a red-team harness.
• LangSmith / Braintrust — managed eval, dataset management, experiment tracking; fastest path if you want built-in infrastructure.
• MLflow — LLM-as-judge plus end-to-end GenAI lifecycle.
• Arize Phoenix — 50+ research-backed metrics (faithfulness, relevance, toxicity, hallucination).

Note the benchmark landscape: MMLU/HellaSwag have largely saturated; public benchmarks tell you little about your task. Build domain evals.

Best practices

Eval-first — write the golden set before you tune the prompt. Prefer a programmatic check to a judge every time ground truth exists (cheaper, exact, more reliable). Calibrate the judge against human labels on a 5–10% sample and re-check when the judge's task changes. Design against the four judge biases: position (mitigate with both-orderings in pairwise scoring), verbosity (longer isn't better), self-preference (a model tends to favor its own family's style — don't judge blindly with the same model you generate with), and authority (confident tone ≠ correct). Pairwise comparison is more reliable than pointwise but quadratic in calls — use both. Sample production; don't judge every request (judge cost scales linearly with volume). Version your eval set — it is a living specification, not a fixed fixture.

Failure points teams ignore — and what each costs

• No evals at all. Most production agents ship without them. Consequence: silent regressions — a prompt tweak or model upgrade quietly degrades quality and nobody knows until users complain.
• Judge as oracle. Using an uncalibrated LLM judge as ground truth. Consequence: you are measuring the judge's noise and biases, then optimizing toward them — confident, precise, wrong metrics.
• Unmitigated position/verbosity bias in A/B. Consequence: you pick the wrong prompt or model because the judge preferred whichever came first or was wordier.
• Happy-path-only evals. Consequence: the failure modes that actually matter (adversarial inputs, edge cases, refusals) are unmeasured, so your green dashboard is meaningless.
• Eval-set overfitting. Tuning to the golden set until it no longer represents production. Consequence: evals pass, production fails — the classic train/serve skew, reborn.
• Judging where truth exists. Using an LLM judge for something a == would settle. Consequence: wasted cost and lower reliability than the deterministic check you skipped.
• Pointwise scores read as trends. Pointwise judge scores drift between runs. Consequence: you chase phantom regressions or miss real ones in the noise.

How to evaluate and mitigate

Evaluate the evals themselves: measure judge-vs-human agreement (aim for ~85–92%, the published range; good, not good enough to be sole safeguard). Measure inter-run stability of your scores. Measure coverage of known failure modes. And run the acid test: does the suite catch a deliberately-injected regression?

Mitigate: calibration sampling against human labels; both-orderings and ensemble judges for high-stakes; grow the golden set continuously from real production failures (this is the flywheel); keep the CI gate programmatic and put the judge on sampled online traffic.

Practical vs. still research

Practical / mature: LLM-as-judge, programmatic CI gates, RAG metric separation, dataset-managed experiments. This is production-grade today and is the highest-leverage rigor you can add.

Practical but unsolved: judge-bias mitigation — the heuristics (both-orderings, ensembles, calibration) work but the biases are not eliminated.

Still research / open: rigorous judge calibration theory, quantifying self-preference, the "eval set as evolving spec" as a formal discipline, and auto-generated adversarial eval sets. Reference-free faithfulness scoring at production scale is partly research. Bottom line: treat the judge as a calibrated instrument with known error bars, not a source of truth.