Why Your AI Feature Is Stuck in Review (And Why a Bigger Model Won't Fix It)

The demo was flawless. The model produced exactly what the product team hoped for, someone in the room said the word "ship," and everyone left optimistic. Months later the feature is still behind a flag and the standup update has not changed: we are working through the review backlog. Somewhere in that gap a team usually decides the problem is model quality and starts pricing out something bigger. Often, it isn't. What stalled is not generation. It is the judgment step that comes after generation, and no model swap has ever fixed a missing definition of "good."

Key Takeaways

• Once generation becomes cheap, the constraint moves to the human hour spent deciding whether an output is acceptable. That hour did not get cheaper.
• If two qualified reviewers disagree about the same output, the gap is usually in your acceptance criteria rather than in the model. A larger model just produces the disagreement faster.
• An evaluation harness is your acceptance criteria written down and scored repeatably on every change, producing a result you can gate a release on. Not a benchmark score, not a vibe check.
• Build the rubric and a golden set from your own real failures before you evaluate any model, and long before you consider custom training.
• Model upgrades fix capability gaps. They do not fix an undefined quality bar, missing domain context, or a review process with no exit condition.

Generation got cheap. Judgment did not.

The cost curve everyone watched was inference. Price per token fell, context windows grew, and the practical question shifted from "can we generate this" to "how much of this can we generate." That part worked.

The cost that did not move is the one that now sets your release date. A human reviewer might clear dozens of outputs in the time your pipeline generates thousands. The exact ratio varies enormously by task, and a clinician checking generated clinical material is not comparable to an editor scanning short product descriptions. The structure does not vary: generation scales faster than expert judgment, and that gap sits directly in front of your release gate.

The review queue is where AI features actually die

Stalled AI features share a recognizable set of symptoms. If three or more of these are true, you probably have a review system problem, regardless of whether the model could also improve:

• A growing backlog of outputs marked "needs a look," with no policy for what happens when it stops shrinking.
• Two reviewers score the same output differently, and the tiebreak is seniority rather than a written criterion.
• Nobody can state whether this week's output is better or worse than last week's, because there is no measurement that spans both.
• The definition of acceptable lives in one or two people's heads, and those people are now the release dependency.

This is a delivery pipeline problem wearing an AI costume, which is why an ordinary software audit often surfaces it faster than another round of prompt engineering. Derek Gallo described the same discovery from the production side on the podcast: once his team got generation fast enough to scale AI video output, the constraint relocated to review cycles, and the real engineering work became defining what "good enough" meant so that not every asset needed a person. How Derek Gallo Defended "Good Enough" Quality to Scale AI Video Production.

A faster model makes this worse, not better: more output per hour into a queue that clears at a fixed human rate is simply a longer queue.

What an evaluation harness is, and what it is not

The word "evals" has been stretched to cover three different things, and the confusion is expensive.

It is not a public benchmark score. Aggregate leaderboard numbers compress wildly different behaviors into one figure. Stanford's HELM project exists largely to make that point: models that look close on a headline metric diverge once you evaluate them scenario by scenario. Your scenario is not on the leaderboard.

It is not a vibe check. Reading twenty outputs and feeling good is a sampling method with no denominator and no record. It cannot tell you whether last Tuesday's prompt change helped.

An evaluation harness is four concrete things: a fixed set of representative inputs, a written definition of acceptable behavior, repeatable scoring against that definition, and results recorded on every run. Some of that scoring is deterministic, some is model graded, and some cases will still need a human. The goal is not to remove people from evaluation. It is to stop spending expert attention on the outputs you could have cleared automatically. The harness belongs in your delivery pipeline next to your test suite, which makes it an architecture decision rather than a data science side project. Same triggers as CI tests, same visibility, same authority to block a release.

One distinction is worth making early, because teams routinely collapse it. The same rubric can serve two different gates. Offline evals decide whether a system change (a new prompt, a model swap, a retrieval tweak) is safe to release. Runtime checks decide whether one particular production output can proceed automatically or needs a human. Do not confuse the two, but build both from the same written definition of acceptable behavior.

Where "acceptable" is defined externally rather than by your product taste, this stops being optional. A team building clinical or healthcare software does not choose its quality bar. It inherits one from regulators, payers, and standards of care, and has to demonstrate on demand that the bar is met. The MEASURE function of the NIST AI Risk Management Framework frames evaluation the same way: a governance requirement with documented methods and repeatable results, not a nicety you get to after launch.

Write the rubric before you touch the model

Mona Chitnis described this on the podcast when explaining why her team built an evaluation capability before deciding whether foundation models were sufficient or custom models were justified: How Mona Chitnis Defended Building Custom AI Models When Foundation Models Weren't Enough. The evaluation came first precisely so the model decision could rest on evidence instead of preference. Most teams run that order backwards, pick a model on reputation, then argue about quality without a yardstick.

The sequence that works puts the least glamorous work first. Concretely:

Harvest real failures. Pull the last two hundred outputs a human reviewed. Sort them into accepted and rejected, then write one plain sentence per rejection explaining what was wrong. Do not generalize yet.

Cluster those sentences into named criteria. Most teams land on four to six: factual correctness, domain fit, completeness, safety or compliance, tone or format. Give each a binary or three point scale with a written description of each level. Vague words like "accurate" are where rubrics fail, so replace them with checkable statements: every claim traces to a retrieved source, no dosage stated without units, the required disclosure paragraph present verbatim.

Test the rubric on humans before you test it on a model. Have two reviewers independently score the same twenty outputs. If they disagree frequently, the rubric is ambiguous and automating it would only encode the ambiguity. Fix the wording and repeat until independent reviewers apply it consistently. This step routinely exposes that two departments were holding different quality bars the whole time.

Build a golden set of twenty to thirty cases. Not random samples. Stratify them: roughly half typical cases that must always pass, a quarter hard cases from your rejection pile, a quarter adversarial or edge cases including malformed input, missing context, and the request the feature should refuse. Thirty well-chosen cases you actually maintain can be more useful than three hundred scraped examples nobody understands.

Score deterministically wherever you can. Schema validity, presence of a citation, forbidden terms, numeric tolerance, and required fields are ordinary assertions. Reserve model graded scoring for genuinely subjective criteria, and validate the grader against human labels first. An unvalidated model grader is a vibe check with better production values.

This is the same discipline regulated financial software teams already apply to model risk: if a team cannot explain why an output is acceptable, it cannot build a defensible release process around it. The same principle applies to generative AI. Before you optimize the model, build the yardstick.

Where a bigger model actually helps, and where it does not

Sometimes an upgrade is exactly right. Model swaps fix capability gaps: reasoning depth, instruction following, long context handling, language coverage. If your failures are cases where the model plainly could not do the task, upgrade.

They do not fix three other things, and in a stalled feature those are the usual culprits. Not an undefined quality bar, because a better model just produces better output nobody is authorized to approve. Not missing domain context, because a model cannot infer your internal policies or last quarter's exception; that is a retrieval and system optimization problem. And not a review process with no exit condition, because the queue is a process artifact.

You can tell these apart in about a day. Hand twenty rejected outputs to your two best reviewers with one question: could a knowledgeable person have produced the correct answer using only the information the model was given?

• Mostly no. You have a context problem. Better retrieval, and more of your own domain data reaching the model, beats a better model.
• Mostly yes, but the reviewers disagree about what correct means. You have a criteria problem. A rubric beats both retrieval work and a model upgrade, and is the cheapest of the three to build.
• Yes, the reviewers agree on the right answer, and the output is still wrong. Now you have a capability gap, and a model upgrade is the highest leverage move available.

A thirty day path to getting the feature unstuck

For a bounded feature with an existing review history, a first useful version can often be built inside a month without pausing feature work. It needs one engineer plus part time access to two domain reviewers. A regulated domain, or a feature with no review history to mine, will take longer.

1. Week one: harvest. Collect two hundred reviewed outputs, label them accepted or rejected, and write the one line reason for every rejection. Nothing gets designed this week.
2. Week two: rubric. Cluster the reasons into four to six criteria with written level descriptions, then run the two reviewer check on twenty outputs and revise until they apply it consistently.
3. Week three: encode. Build the golden set and implement scorers, deterministic first, model graded only where necessary.
4. Week four: gate. Wire the harness into CI so it runs on every prompt, model, or retrieval change, and set a pass threshold that blocks release. Then apply the same rubric at runtime, so the outputs reaching a human are the ones the checks could not confidently clear.

The outcome is not that humans stop reviewing. It is that they spend their attention on the cases the harness cannot confidently clear instead of manually reviewing every output, and every prompt change now arrives with a number attached. That number is what turns "it feels better" into a release decision.

If your feature has been one review cycle away from launch for a while, that is the diagnostic. We build this measurement layer with teams and get stalled features through the gate as part of our AI engineering work. The first conversation is usually about your rejection pile, not about which model you picked.