Where does your team actually sit?

We have at least one automated eval for an AI feature in production.

Every shipped AI feature has its own dedicated eval suite (not shared, not generic).

Eval suites cover BOTH correctness (does it produce the right output?) AND safety/refuse (does it not produce wrong outputs?).

Eval suites are version-controlled alongside the code they test, with a documented baseline metric per suite.

We have failure-mode-specific eval files (faithfulness, refusal-correctness, length-bound, citation), not just general accuracy.

We can run AI evals locally before merging.

CI runs evals on every PR that touches AI-feature code.

A documented threshold exists per AI feature — the score below which the team agrees the feature isn't ready.

Release blocks if eval scores drop below threshold — enforced by CI, not advisory.

We have a documented override policy with audit trail for failed eval gates (who overrode, when, why).

We log every AI-feature input and output in production.

We sample production traffic for ongoing eval review (not just CI).

We compare production eval scores against dev/CI eval scores to detect drift.

Drift alerts page on-call when production eval scores degrade below threshold.

We run online evals on production traffic continuously, not just sampled batches.

We have a written list of known AI failure modes for our product (hallucination, refusal-mismatch, drift, etc.).

Each AI-related production incident gets a written postmortem.

Each incident is tagged against a failure-mode taxonomy (so we can count "we had 3 hallucination incidents this quarter").

New failure modes always get added to the eval suite as part of the incident response — not "later when we have time."

We review and update the failure taxonomy at least monthly.

Users can report bad AI outputs (in-product widget or via support).

Reported bad outputs flow into a triage queue with a named owner.

Triaged bad outputs get added to eval suites as new test cases.

We measure time-from-report-to-eval-coverage as a tracked metric.

Eval improvements ship in a regular weekly cadence — not ad-hoc batches when someone has time.

We have a written list of things our AI must refuse to do (out-of-scope, unsafe, regulatory).

The refuse policy is reflected in the system prompt (not just tribal knowledge).

Refuse behavior is tested by automated evals (the model is failed if it answers a question it should refuse).

The refuse policy has been reviewed by legal or compliance.

Compliance reviews are documented, dated, and re-run on a recurring cadence.