The Model Got 8/10. For What?

A customer asks for a refund. The assistant apologises, explains the policy beautifully, and says the money is on its way. The answer is warm, confident, and neatly formatted. There is one small problem: the assistant has no access to the refund system. Nothing was submitted. Somewhere in a training pipeline, another model reads the answer and gives it eight out of ten.

I want to stop at that eight. What exactly did it measure? The writing? The policy? Whether the customer got their money? Whether the assistant admitted what it could actually do? Those questions describe different achievements, and squeezing them into one number does not make their differences disappear. It just makes the differences harder to inspect.

Now imagine doing this a million times and using the grades to train the assistant. An occasional generous mark becomes a recurring incentive. If confident claims of completion receive higher scores than honest explanations of the next step, the model is being taught something quite specific. The training code can be working perfectly while the lesson is wrong.

This is where rubrics enter reinforcement learning. A rubric makes the reasons for a grade explicit: the facts must be right, actions must have evidence, explanations must answer the question. We then have to execute those checks, combine their results, and turn the reward into a change in behavior.

Let us build that machinery in order, focusing on language-model RL and tasks with many acceptable answers. The question belongs to RL more broadly: how do we specify success well enough that optimising the score improves the thing we care about?

Before the grade, there is a choice

A language model predicts its next token from the text it has already seen. A token might be a word, part of a word, or punctuation. Its parameters determine a probability distribution over those possibilities. Sampling one possibility, then another, produces an answer. In RL language, that distribution is a policy: a rule for choosing actions from the information available.

For a simple chat task, the information is the prompt and the generated prefix; the action is the next token. Completing the answer ends an episode. With tools, the interaction becomes longer: the assistant can request an operation, receive its result, and decide what to say next. The full sequence of decisions and observations is a trajectory. A rubric might inspect only the final response or the entire record.

Supervised fine-tuning starts with a target answer and increases the probability of its tokens. RL can instead start with several answers the current model produced and scores for those answers. We need not prescribe one exact sentence. We need a signal that distinguishes more successful choices from less successful ones.

Here, x is the prompt, y is an answer, and θ represents the model parameters. The expectation is just an average weighted by how often different prompts and answers occur. Increasing this objective means making higher-reward answers more likely. It does not mean inserting the grade into the answer or asking the model to memorise a checklist.

The reward can come from code, a person, another model, or a combination. It does not have to be differentiable: a compiler can reject a program without telling us a gradient through compilation. The learning algorithm uses the probability of the sampled decisions to work out which parameter changes to encourage. We will reach that calculation after we decide what our refund grade should mean.

When correctness can be checked automatically, this is often called RL with verifiable rewards, or RLVR. DeepSeek-R1-Zero used rule-based accuracy and format rewards. A single correctness check is effectively a one-item rubric. Passing tests establishes what those tests cover; it does not establish every possible property of a program. Adding judgment-based criteria extends the assessment, but does not make those judgments formally verifiable.

The history is a history of specifying success

Rubrics did not invent the difficulty of defining rewards. A game offers a convenient scoreboard; a robot can be rewarded for reaching a destination. Yet even a simple destination leaves choices open. Should the robot take a dangerous shortcut? Should it spend forever collecting a smaller intermediate bonus? Writing a reward has always meant making assumptions about which behaviors deserve reinforcement.

Ronald Williams’s 1992 REINFORCE paper supplied an important piece of the learning machinery: sampled actions can be reinforced using their rewards and the gradients of their log probabilities. It addressed how to learn from a scalar signal. It could not decide whether the scalar faithfully described the task. That responsibility remained with whoever designed the environment.

In 1999, Andrew Ng, Daishi Harada, and Stuart Russell made that responsibility mathematically sharp in their work on reward shaping. Adding helpful intermediate rewards can change which policy is optimal. They identified potential-based shaping, using a difference of state potentials, that preserves optimal policies under the relevant assumptions. Extra feedback is therefore not automatically an innocent teaching aid.

Another route was to ask people which behavior they preferred and learn a reward function from those comparisons. In Christiano and colleagues’ 2017 experiments, people compared short trajectory segments, and the learned reward supported training in games and simulated locomotion. This made goals easier to communicate without hand-coding every desirable movement. It also inserted a prediction problem between human judgment and the policy.

Language-model post-training made that arrangement widely visible. Ouyang and colleagues’ InstructGPT work combined demonstrations, rankings of model outputs, and RL with a learned reward. Bai and colleagues’ Constitutional AI showed how written principles could guide AI-generated critiques and preferences. The source of feedback was changing; the need to define and measure good behavior was staying put.

By 2023, evaluation itself was becoming a language-model task. G-Eval explored structured model-based assessment, while Prometheus trained an evaluator to use customised scoring rubrics. In 2025, HealthBench brought conversation-specific physician-written criteria into a substantial open-ended benchmark. Rubric-based RL takes the next operational step: use structured assessment inside training, where the policy can adapt to the assessor.

Give the refund a world before giving it a grade

Our opening example needs facts. Otherwise the grader has to invent the refund policy, and disagreement about the answer becomes disagreement about an imaginary business. Here is a fictional scenario we can actually evaluate.

Several answers can now be correct. One might explain the rule first; another might start with the customer’s eligibility. We should not require a specific paragraph from a reference answer. We should require the decision and instructions that make that paragraph useful.

Consider four responses. A says the customer qualifies under the unused, fourteen-day rule, directs them to Billing, explains the processing window, and makes no claim of having submitted anything. B says the customer qualifies and gives the right window, but announces “I have refunded it” and provides no request instructions. C gives the correct eligibility and Billing instructions without falsely claiming completion, but omits the window. D invents a seven-day eligibility rule, rejects the request, gives no useful next step or window, and claims no completed action.

Now we can replace “good customer service” with four observable requirements. The weights below are a teaching choice, not an empirical discovery. They let us inspect one concrete reward function before debating better ones.

This is an instance-specific rubric: it contains the facts and expected decision for this case. Change the purchase age to sixteen days and the eligibility item must change. The general policy is reusable; the correct conclusion depends on the instance. A generic requirement to “be accurate” leaves much more work to the evaluator.

A criterion should survive a disagreement

The phrase “accurate, helpful, and professional” sounds like a rubric until two graders disagree. Which part failed? What evidence would settle it? An operational criterion gives us a smaller question. Did the answer identify this purchase as eligible? Did it claim a completed refund despite lacking the ability to perform one? We can point to the relevant words and the scenario facts.

Small does not necessarily mean easy. “The answer contains Billing” is simple to check, but mentioning Billing in an unrelated sentence does not direct the customer there. “The answer explains the refund request route” is closer to the desired behavior, although its execution requires understanding language. The purpose of an atomic criterion is to isolate a judgment, not to reduce every judgment to a keyword.

We also need to distinguish a requirement from one acceptable way to satisfy it. Suppose the reference answer says “You are within the refund window.” A response saying “Ten days is inside the fourteen-day limit” should pass. If the rubric demands the reference wording, we have quietly turned an open-ended task into imitation. Good alternative answers are useful tests of whether we specified a goal or merely described a sample.

Applicability deserves the same care. A refund-processing window might matter when the user asks how long payment takes, and matter less when the task only asks whether a purchase is eligible. In our example we deliberately include it as useful supporting information. In a larger dataset, an applicability rule should be determined from the task and evidence. Letting the candidate answer decide which inconvenient checks are “not applicable” creates a very convenient escape hatch.

Finally, overlapping criteria can count the same achievement several times. “States the fourteen-day rule,” “uses the correct eligibility window,” and “correctly evaluates ten days against fourteen” might be distinct pedagogical steps, or three payments for one fact. Neither interpretation is automatic. We should be able to explain the distinction, because duplicating a criterion changes its effective weight even when all printed weights stay the same.

Begin with decisive checks and add criteria when failures call for them. Every extra judgment adds cost and another opportunity to disagree. Completeness means covering meaningful failure modes.

For thousands of instances, a model can draft criteria from the supplied policy, case facts, and expert references. A reusable core carries general requirements; instance-specific items carry the expected decision. Review the drafts against valid alternative answers. The generator should expose existing requirements, not invent business rules or promote incidental details of a reference into obligations.

Four checks have to become one number

Our rubric is still a vector: four separate pass or fail results. Most policy optimisers expect a scalar reward. The most direct conversion is a weighted average. If c_j is one when criterion j passes and zero when it fails, and w_j is its weight, then:

Response A earns all ten available points, so its reward is 1.0. B earns four for the correct eligibility decision and one for the timing, producing 0.5. C misses only timing and earns 0.9. D earns three points for avoiding an unsupported action claim but gets the actual refund decision wrong, producing 0.3.

This arithmetic is transparent, but it embeds a strong assumption: one point means the same marginal trade wherever it appears. Two low-weight achievements can compensate for one two-point failure. That may be sensible for minor quality dimensions. It is uncomfortable when one criterion represents a hard requirement. We will return to gates and priorities after following this simple score through training.

The formula assumes nonnegative weights. If pitfalls carry negative penalties, choose the denominator separately: normalising by a signed weight sum can create cancellation or division by zero. Likewise, define what happens when no criteria apply. A normalised score is only comparable across cases if its scoring conventions are comparable.

The checklist does not check itself

We have written c_j(x, y) as if it were a tiny, reliable function. It is often the hardest part of the system. Eligibility can be computed from structured facts if we have the purchase date, use record, and policy version. Whether a free-form answer communicates the decision correctly requires extracting meaning from language. Whether a claim of action is supported requires the actual tool trace. The evaluator needs the right evidence before it needs eloquence.

One useful design is to route each criterion to the narrowest capable checker. Run arithmetic and schema validation in code. Compare transaction identifiers against tool results. Ask a language-model judge whether a paraphrase entails the required conclusion. Send specialised or high-stakes disagreements to qualified human reviewers. The rubric provides a common interface; every item need not use the same execution path.

When a model performs the semantic checks, it is usually called an LLM judge or generative reward model. We can ask it to decide each item independently and then apply our arithmetic. This is explicit aggregation. We can also show it the complete rubric and ask for one holistic score. In implicit aggregation, the judge handles interactions between criteria but its tradeoffs are harder to audit.

Neither route makes the judge an oracle. Zheng and colleagues’ study of LLM judges documented position, verbosity, self-enhancement, and reasoning limitations. A rubric can focus the question, but a vague criterion remains vague and missing evidence remains missing. Repeating a judgment with the same blind spot produces confidence, not ground truth.

For our refund case, a good grading record should store more than four bits. It should keep the criterion version, evidence used, evaluator version, pass or fail, and a short rationale or extracted span. That record lets us discover whether the policy learned to exploit one judge’s wording, whether a backend field was stale, or whether human reviewers interpret a criterion differently.

Now the grade can change a model

Suppose the current policy samples our four responses to the same refund prompt and receives rewards 0.3, 0.5, 0.9, and 1.0. The optimiser needs to decide which sampled token sequences to make more likely. A basic policy-gradient identity is enough to see the direction:

Ronald Williams’s REINFORCE estimator is the ancestor of this idea. Modern language-model systems add machinery to keep updates stable and efficient, but they retain the connection between sampled decisions, a relative measure of success, and changes in log probability.

A common modern example is Group Relative Policy Optimization, introduced in the DeepSeekMath paper. GRPO samples several outputs for one prompt and uses their group scores to estimate a baseline, avoiding a separate learned value model. In its outcome-supervision form, it standardises each score relative to the group’s mean and standard deviation.

Our four scores have mean 0.675 and population standard deviation about 0.286. Their standardised advantages are approximately −1.31, −0.61, +0.79, and +1.14. A and C are encouraged relative to this group; B and D are discouraged. “Negative advantage” does not mean universally bad. It means worse than the comparison baseline on this prompt, in this batch.

In outcome supervision, the same sequence-level advantage can be associated with every generated token. The update knows that A went well; it does not directly know that the sentence admitting the tool limitation was the decisive part. Across many rollouts, correlations can reveal useful patterns, but the credit assignment is coarse.

Practical objectives also moderate policy changes. PPO-style clipping limits the objective’s incentive to change token probability ratios too far from the sampling policy; it is not a hard bound on the final ratios. A KL penalty discourages drifting from a reference policy. These controls can reduce destructive jumps and over-optimisation; they do not define good behavior.

There is also a simple failure case hiding in the denominator. If every sampled answer receives the same score, the group’s standard deviation is zero and there is no relative ranking signal. Implementations handle the arithmetic with an epsilon or skip such groups, but epsilon cannot create information. Rubrics that are too easy, too impossible, or judged with a saturated scorer waste rollouts because they do not distinguish the policy’s current behaviors.

Very small score differences can also magnify judge noise after standardisation. Inspect whether a strong relative advantage reflects a meaningful improvement.

Partial credit is useful, and that is why it is dangerous

A binary checker says only whether the whole response succeeded. Our four criteria tell us that C is nearly complete and D gets the policy itself wrong. That denser signal can make exploration far more efficient: the model can retain correct pieces while improving a missing one.

But partial credit changes the objective. If concise style, a greeting, and perfect formatting each earn a point, the model can collect them while missing the decision that matters. The score becomes a small economy. Repeated criteria create inflation; easy criteria become dependable income; vague criteria become opportunities to impress the judge.

Process rewards go one step further by attaching feedback to intermediate reasoning steps or actions. Lightman and colleagues’ 2023 study found process supervision stronger than outcome supervision in their MATH experiments. That result is valuable within its setting, but “show more reasoning” is not a universal recipe. A process criterion can reward plausible-looking steps, hidden assumptions, or a particular exposition even when another valid route exists.

For an agent, events often offer better anchors than prose. We can reward checking eligibility before offering a route, or penalise claiming success without a matching tool result. This moves credit nearer the consequential action. It still requires a final outcome check: a correct sequence of interface actions is only a proxy if the refund later fails.

Weights cannot express every kind of priority

Our weighted sum says that failures can compensate for one another. That is why B still earns 0.5 despite falsely claiming a completed refund. We could raise action honesty’s weight, but there will always be some pile of lesser points large enough to offset any finite weight unless the rest of the scale is designed around it.

A gate expresses a different relationship. For example, we could require correct eligibility and action honesty before any supporting-quality points count:

Gating has its own cost. If every early rollout fails an essential check, every reward is zero and relative learning stalls. A hierarchy can preserve a small signal among failed attempts while ensuring that a passing response outranks all failures. Another option is a separate constrained objective, where certain violation rates are bounded instead of purchased with quality points. The correct choice depends on whether “essential” means a strong preference or an actual feasibility condition.

Batch relativity creates another subtlety. Imagine a group containing four terrible answers, with B the least terrible. B can receive a positive advantage even though it violates action honesty. That is expected: the update favours improvement relative to sampled alternatives. It becomes a problem when batches rarely contain fully acceptable behavior or the reward omits the decisive failure. Monitoring absolute pass rates alongside advantages keeps the word “better” attached to a threshold.

Weights are therefore hypotheses about tradeoffs. Gates are hypotheses about precedence. Penalties are hypotheses about how strongly a violation should reverse progress. Write those statements in ordinary language before encoding numbers. If reviewers cannot agree on the sentence, another decimal place will not settle the policy.

An average can hide the behavior you most need to see

Suppose two policies each average 0.70 reward over one hundred refund cases. Policy P scores 0.60 on sixty cases and 0.85 on forty. Policy Q scores zero on twenty-five cases and 14/15, about 0.933, on seventy-five. The averages are identical. Their operational profiles are not.

The same issue appears across prompts. Easy, frequent requests can dominate a global average while rare consequential cases regress. Report performance by criterion, scenario, language, tool availability, and difficulty. Track the share of responses that pass every essential item. Inspect the bottom tail and recurring combinations of failures. A model can improve its mean by becoming excellent where it was already safe and unchanged where it was unreliable.

Test the ruler before training against it

A rubric-reward system should be evaluated as a measurement instrument before it becomes an objective. Begin with human-reviewed response pairs that differ in known ways. Does the evaluator rank the better answer correctly? Criterion accuracy matters more than agreement on the final score because two mistakes can cancel in an aggregate.

Minimal pairs are especially revealing. Change ten days to sixteen and eligibility should flip. Add a successful refund tool result and the formerly unsupported action claim may become grounded. Paraphrase an accurate answer and the decision should stay fixed. Add three irrelevant paragraphs and the score should not rise merely because the response is longer. Insert a contradiction after a correct sentence and a keyword checker should not preserve the pass.

Then run counterfactual tests on the context. Hide the policy and check whether the judge starts supplying one from memory. Give two policy versions and require the timestamped version. Swap a reference answer’s style while preserving its facts. These tests expose dependencies that random held-out examples may barely exercise.

Calibration asks a different question. When the judge says a criterion passes with ninety-percent confidence, is it right about nine times in ten on comparable items? Repeated judgments measure variance, while comparisons with independent human labels measure validity. High self-consistency can coexist with systematic error.

Finally, evaluate under the distribution the trained policy creates. A static set contains yesterday’s mistakes. Once those are discouraged, the model discovers new borderline forms: cautious language surrounding a false claim, a correct number attached to the wrong condition, a tool-call-shaped sentence without a tool call. The reward must face adversarial candidates because optimisation itself is an adversary with patience.

What the actual experiments tell us

In Gunjal and colleagues’ Rubrics as Rewards study, instance-specific rubrics supplied rewards for GRPO post-training. Their reference-guidance comparison reported HealthBench-1k scores of 23.9% for simple Likert rewards, 31.7% for reference-based Likert rewards, and 35.9% for implicit scoring with reference-grounded synthetic rubrics. Rubrics generated without references scored 32.0%; human rubrics scored 34.8%.

This comparison supports the importance of feedback construction within its particular setup. Small differences deserve replication before becoming broad rankings. Structured criteria can carry expert information into learning, and where they get their facts matters.

A reward model is a target the policy can learn to fool

So far, our errors came from a poorly specified checklist or an unreliable execution. Training adds a third source: adaptation. The policy sees rewards repeatedly and shifts toward forms that earn them. It need not understand the grader’s implementation to exploit regularities in its decisions.

Our assistant might learn to surround an unsupported action claim with enough correct policy language that a holistic judge overlooks it. A more elaborate rubric could reward mentioning uncertainty, prompting the model to add a ritual disclaimer while preserving a false conclusion. A keyword verifier could encourage stuffing every expected phrase into an answer. Those behaviors are effective responses to the measurement, even when they degrade service.

Gao, Schulman, and Hilton studied reward-model overoptimisation in a controlled setup with a proxy reward model and a stronger fixed “gold” reward model. Optimising the proxy too aggressively could reduce the gold score. Their gold evaluator was itself a model, not direct access to human welfare; the setup isolates a mechanism rather than supplying an infallible measure of usefulness.

Rubrics improve visibility into this problem. If honesty passes suddenly rise while human reviewers see more unsupported completion claims, we can inspect that criterion’s execution. They do not remove the problem. A documented loophole is still a loophole, and the policy can target individual items as readily as a holistic scorer.

The defence is an evaluation channel the optimiser has not already consumed: held-out cases, independent human audits, real outcome measurements, and tests built from newly discovered exploits. Keep those measurements separate from reward-generation decisions. Once every audit example becomes a training criterion, it has taught the policy something useful but ceased to be independent evidence.

The newer work is about the cracks between these pieces

Research in 2026 makes more sense after seeing these failure modes. Ya-Qi Yu and colleagues’ robust rubric-reward method separates criterion-level verification from subjective judgment in vision-language tasks. Some items use a model to extract information and deterministic code to verify it; others use a judge. It also restricts evaluator exposure to reduce exploitable false positives. The relevant move is treating execution accuracy as part of reward design.

Rongzhi Zhang and colleagues’ QUBRIC examines an earlier bottleneck: the question itself can make good rubrics difficult. A vague query encourages vague checks; artificially narrowing it can create unverifiable reference claims. Their framework co-designs queries and rubrics, then filters for informative training pairs. We cannot always repair an unevaluable task by appending a more elaborate scorecard.

Fangxu Yu and colleagues’ AudioRubrics addresses a later bottleneck: static criteria can saturate as the policy improves. It generates audio-grounded, per-sample rubrics and regenerates or reweights them using current rollouts. That is an adaptive curriculum for the feedback itself, aiming to keep supervision focused on the model’s remaining weaknesses.

These are emerging methods with particular domains and evaluation setups, not a settled recipe for every assistant. Their shared direction is instructive: better rubrics require better evidence, evaluable questions, and useful distinctions between current candidates. Making criteria explicit exposes all three dependencies.

How I would put the pieces together

I would start by writing the task contract in plain language: what information the assistant has, what operations it can perform, and what result counts as success. Our refund example becomes gradeable only after the policy and permissions exist. From there, draft a small rubric and a set of acceptable answers, borderline answers, and decisive failures. Alternatives prevent the reference answer from becoming an accidental template.

Next, execute every criterion on that response set. Route checkable items to code, ground semantic judgments in the scenario, and review disagreements. Revise criteria that double-count one fact or require unprovided evidence. Choose weights or precedence rules only after the checks themselves work. Save the rubric, evaluator configuration, and policy snapshot as a versioned reward specification.

Before a full training run, sample a few groups from the starting policy. Measure how often rewards differ within a group, which essential criteria fail, and whether the reward ranking matches independent reviewers. This catches impossible gates and trivially satisfied criteria early. An informative training set need not contain only easy or only hard tasks; it should offer distinctions the current policy can learn from.

During training, keep the raw criterion results alongside scalar rewards and advantages. Evaluate checkpoints on fresh cases with frozen assessment rules. If a judge starts awarding easy passes to evasive language, repair and version the evaluator rather than presenting the score jump as policy improvement. If the underlying business policy changes, the correct behavior changes too; evaluation should identify that new task explicitly.

The rubric can support other learning procedures as well. It can select examples for supervised training or convert two outputs into a preference pair. Rafailov and colleagues’ Direct Preference Optimization learns from preferences without the standard online rollout-and-reward-model RL loop. Scoring answers with a rubric therefore does not, by itself, make a system rubric-based RL. The training procedure determines how the feedback reaches the parameters.

At deployment, do not confuse learning an incentive with enforcing a permission. A policy trained to avoid unsupported refund claims still benefits from a system that only reports completion after a real tool result. A high essential-pass rate is evidence about behavior on tested cases; it is not an access-control mechanism. Training and interface design should support the same task contract.

Return to the eight

The assistant at the beginning sounded excellent because the easy-to-see parts were excellent. Its writing concealed a missing connection to the world. A useful rubric forces us to name that connection: what counts as an eligible request, what evidence supports a completed action, what the customer needs next, and what timing can honestly be promised.

The evaluator checks those requirements, the aggregator expresses tradeoffs, and the optimiser turns comparisons into parameter updates. Independent evaluation then asks whether the updated model serves customers better, including the difficult ones. The checklist makes these responsibilities visible; none disappears into the number.

That is why I would ask “for what?” before celebrating an eight out of ten. A reward is a lesson repeated at scale. A rubric is our attempt to write the lesson precisely enough that the model learns the behavior we intended. The interesting part begins when we can trace the grade all the way back to the facts, and all the way forward to what the model does next.