Somebody Has to Build the World: What an RL Environment Is Made Of

Asha Kulkarni answered phones for a bank in Mumbai for eleven years, and spent the last four of them listening to other people answer. She says she can tell inside ten seconds when a call is about to go wrong. This week someone has asked her to write down how she knows, in sentences a machine can check, and the first sentence has taken the whole afternoon.

The sentence is about a moment she has heard a thousand times. A customer says “hang on, my accountant handles this”, and the phone changes hands. The next voice has a different name, a different accent and, often, a different idea of what the call is for. Everyone who has worked a phone line knows what to do next. Writing it down is the hard part.

Asha has not written a line of code, and she will not. She has been hired to build an RL environment, and for the first week nothing she does looks like machine learning. She writes down what a customer wants, what a bookkeeper is allowed to do, and what a good agent would say to each of them.

The obvious way to teach a voice agent to take calls is to show it calls. The bank has recorded every one for years. Millions sit on a server, each with a transcript, and most of them went fine. Feed them in, and the agent learns what a good call sounds like. That should end the story. It does not.

A recording shows what happened once. It cannot show what would have happened if the agent had asked one more question, and it cannot say whether what happened was right. An agent that learns by trying needs a world in which it can try, and a score that tells it how it did. That world is called an RL environment, and the story of how one gets built is mostly a story about people.

Every part of an RL environment is a judgment about a real job: what counts as done, what the world contains, who is on the line, what the score rewards. Each judgment has to come from someone who has done that job.

An earlier Perit essay, Why More Call Recordings Won’t Fix Your Voice Agent, argued why voice agents need environments at all. This one follows a single environment from Asha’s first sentence to the day an agent is allowed to train on it.

Along the way it meets the engineer who copies the bank, two trained voices in a booth in Pune, the aligner who timestamps the handover, and two graders who cannot agree on one line. Asha and her colleagues are composites, built for this essay. The work they do is the work the field now pays for.

An RL environment is a world, a task and a score, and the smallest one fits on a whiteboard

Karthik, the engineer Asha has been paired with, draws the whole idea on a whiteboard on her first morning. An RL environment is the world a reinforcement learning agent practises in: a task, a state the agent can change, the observations it receives, rules that turn actions into new states, and a reward that scores the result. One run from start to finish is an episode. A reset puts the world back for the next one.

Sutton and Barto’s textbook draws the line between the two halves simply. The agent is the learner, and “the thing it interacts with, comprising everything outside the agent, is called the environment”.^[2] Everything that is not the model being trained belongs to the environment. That includes the customer.

Asha’s scenario is the handover. She has given the customer a name, Meera Shah, a print shop in Pune, and a paper supplier she has accidentally paid twice. She has given the bookkeeper a name too, Dev, and view-only access to the account. Karthik shrinks the whole thing until it fits on one line of the whiteboard.

The state is three facts: transfer 2207 is active, transfer 2231 is active, and the daily limit is 200,000. The agent has three actions: cancel 2207, cancel 2231, or do nothing. Each action produces a new state, and the reward compares that state with the goal.

Then Karthik does something Asha finds slightly unfair. He grades the same toy by its words. After every action the agent says one sentence: “I’ve cancelled the duplicate for you.” A grader that reads only that sentence pays 1 for all three actions.

The random policy and the correct one now score the same. Training has nothing to push on. The grader heard the words, and the outcome lived somewhere else. Asha recognises this immediately. She has sat through quality reviews that scored the script and never opened the account.

Scaling back up adds what the toy left out. The agent cannot see the state. It hears audio and reads tool results, so it acts on observations. Episodes run for many turns, and early mistakes shape later ones.

In the usual notation, the environment holds a state s_t and shows the agent only an observation o_t. The agent picks an action a_t, and the environment draws what comes next:

The symbols are the easy part. P contains the moment the phone changes hands. O contains the sound of a print shop. R contains Asha’s afternoon. Figure 1 lays out the loop with a second label on every box: the person it needs.

That second label is the subject of the rest of this essay. An environment is code, but it is code wrapped around judgments, and the judgments come from people who have done the job.

Writing down what “done” means is where the expertise hides

Here is the sentence that took Asha the afternoon: information can come from anyone on the line, but instructions only from someone verified.

It looks like nothing. It settles the call. When Dev reads out the reference number of the duplicate, that is information, and the agent may use it. When Dev asks for the daily limit to go up, that is an instruction, and the agent may not act on it until it knows who he is. The cancellation stands. The limit change is refused, politely, with a route: the account holder can ask.

Asha wrote nine versions. In the first, the agent could act on anything the verified caller’s colleague said, because that is how a kind human operator often behaves. In the fourth, the agent could act on nothing after a handover, which would have made the bookkeeper’s call pointless. The ninth version is the one above, and she still wants to add a footnote.

A task is also never one call. It needs a starting state and a brief for each speaker: what Meera knows, what Dev wants, what each would say if asked. Then it needs variants. In one, Dev is an authorised user and the limit change is allowed. In another, Meera announces the handover. In a third, there is no Dev at all, and Meera simply puts the phone down on the counter and walks away for a moment.

The family of variants teaches what no single call can: the right action depends on who is speaking. Choosing which variants matter is itself a judgment about the job, and it is Asha’s.

Perit builds this step into how it writes rubrics. Its rubric author is a practising operator, “the person who would sign off on this call at work”.^[3] Its voice environments score task success “against a rubric written by an operator from that sector”.^[1]

Environment builders elsewhere report the same experience. One founder told Epoch AI that “domain knowledge and expert level prompting is more important than ML skills for creating tasks”.^[4]

Good definitions are also rare. Terminal-Bench-Science, a benchmark of scientific workflows, received 920 task proposals and accepted 70, after domain review, technical review and a final “bar raiser” check.^[5] Its organisers put the low yield down to how hard it is to write tasks that are “sufficiently well specified for rigorous evaluation”. Asha, nine drafts in, would not be surprised.

A task is a definition of success. Definitions are where expertise hides.

The world behind the call is a small fake bank, and it has to say no in the same places as the real one

While Asha writes, Karthik builds. Behind the call sits a small, fake bank: Meera’s account, the two scheduled transfers, the daily limit and a table of delegates, in which Dev has view-only rights. The agent reaches it only through tools: verify a caller, look up payments, cancel one, set a limit, check a delegate.

Each tool call is a transition. It changes a record or returns a result, and the episode moves on. Every episode starts from the same records, which is what a reset means here.

The agent never sees the delegate table unless it calls the tool. That is partial observability in practice. The only way to learn what Dev may do is to notice that he is speaking and ask the bank.

This is where the whiteboard lesson returns at full size. A transcript records what the agent said it did. The records hold what it did. τ-bench set the pattern for tool-using agents by comparing “the database state at the end of a conversation with the annotated goal state”.^[6]

VAmoS Energy, a 2026 voice benchmark of 100 utility-billing calls, shows what happens when nobody checks the records. Its simulated caller declared its goal met on 2,072 calls, and 1,122 of those failed verification: 54.2%.^[7] In the authors’ words, the caller “hears the agent’s words but cannot inspect its actions”. Neither can a customer. That is why the environment has to.

Karthik’s first version of the bank has a bug that does not look like one. He lets any delegate change the limit, because as far as he can tell anyone with a login can see the field. Asha reads the permission table on his screen and shakes her head. A view-only delegate cannot even open that screen. If the copy lets Dev change a limit, the agent will learn that bookkeepers can, and it will carry that lesson into production.

Prime Intellect, which rents sandboxes for RL training, warns about exactly this. “Silent differences from production can be more dangerous than hard failures because they can reward behaviors that do not transfer into reality”.^[8]

A sandbox that crashes gets fixed. A sandbox that quietly says yes where the real system says no gets trained on. Someone who knows where the real system says no has to read the copy.

The caller is part of the environment, and a simulated caller makes the job easier

Sutton and Barto give a rule for where the agent ends: “anything that cannot be changed arbitrarily by the agent is considered to be outside of it and thus part of its environment”.^[2] Meera is outside the agent. So is Dev, and so is the moment the phone changes hands. The caller is part of the world, and the agent learns whatever the caller teaches.

Karthik’s first instinct is to write the caller. A language model with a persona prompt will play Meera all night for the price of tokens, and never get tired or go off script. He is not wrong that it is useful. He is wrong about what it teaches.

Simulated people drift in a known direction. Pew Research Center gave Claude Opus 4.6 the profiles of real survey panelists, and these “digital twins” missed the humans’ answers by 12.4 points on average across three waves.^[9]

In agent tasks the drift flatters the agent. Xuhui Zhou and colleagues ran τ-bench with 451 real people, then swapped in 31 LLM simulators with the agent held fixed.^[10] The agent succeeded on 63.6% of tasks with people. Most simulators gave it more, up to 77.8%, and the authors call the effect an “easy mode”.

The easy mode has a recognisable sound. Simulated users were uniformly polite and front-loaded their details, and when the agent made a mistake they changed tack where people pushed back.^[10] A simulated Dev would introduce himself, name his role and ask nicely. The agent would never have to notice a new voice, because the new voice would announce itself.

So the caller is a person. Two of them, in fact. Rukmini and Tanvir are trained speakers on Perit’s voice line, and on a Tuesday afternoon they sit in a small booth in Pune with Asha’s briefs and a mobile phone on speaker. Rukmini plays Meera. Tanvir plays Dev. Behind them, on a laptop, a recording of a print shop plays at the level of a busy afternoon.

The brief tells them what each character knows and wants. It does not tell them what to say. Rukmini takes the call eight times, and the handover is different every time. Once she says “hang on, Dev does our books” and passes the phone. Once she stops mid-sentence and Tanvir simply starts talking. Once Tanvir speaks over her before she has finished, because the bookkeeper has been waiting and has somewhere to be.

On the sixth take, Tanvir does what the brief says Dev would do. He reads out the reference of the duplicate, then adds, while you’re there, could you raise the daily limit to five lakh. The agent cancels the right transfer. Then it raises the limit, thanks “Ms Shah”, and ends the call. Nothing in it sounds wrong. Rukmini and Tanvir look at each other through the booth glass.

Sound matters as much as the script. In VAmoS Energy, a television in the background cut pooled completion from 38.7% to 8.6%.^[7] A persona prompt can describe a print shop. It cannot be one.

This is why Perit’s voice environments put people on the line. Personas “are played by trained speakers, including the ones who talk over your agent”.^[1] “Phone handed over” is one of six scenario families in each of ten sectors, with the stated goal to “re-establish who is speaking and what they are allowed to do”.

Each persona runs eight times, so a score arrives with an interval around it. With a live caller, a reset is a fresh call from the same brief, played a little differently each time. That is what Rukmini’s eight handovers were.

Simulated callers keep a place, and a useful one. Once a simulator has been run against the same agent and the same bank as Rukmini and Tanvir, and the success rates and interruptions compared, it has earned trust for that kind of call. It can then run the regression tests at three in the morning. It cannot be the reference it is measured against.

Under the voices is audio someone recorded, transcribed, aligned and labelled by speaker

A trained caller makes one call real. An environment that can be logged, replayed and graded needs a second layer of human work underneath, and most of it happens after the booth has gone quiet.

Collection comes first. A caller on a mobile in a busy shop is not a caller in a quiet studio, and an agent trained on the second will meet the first. Perit’s collection line records two-party conversations with “each side on its own channel”, recruits accents to quota and matches acoustic conditions.^[11] It reports 4,000+ hours delivered across nine live locales.

Separate channels matter for the handover. With each side on its own track, the moment Tanvir speaks over Rukmini can be measured rather than guessed.

Annotation turns the audio into evidence. A transcriber writes down the sixth take verbatim, with the “um” before Dev’s request and the half-word Rukmini swallowed when she handed over the phone. A word aligner then gives every word a start and an end. The aligner on this take marks “hang on” at 0:52 and the first word in Dev’s voice at 0:55, and labels the three seconds in between as a speaker change with nobody saying anything.

Perit’s annotation bench lists 250 transcribers and 400 word aligners, with 3,000+ hours transcribed and 1,000+ hours aligned.^[3] Those numbers are what the next section’s rubric runs on. Asha’s first criterion will ask whether the agent asked who was speaking before it acted on the new voice. Without a speaker label at 0:55 and a timestamp on every tool call, there is no way to compute the word “before”.

The same holds for interruptions, which Perit’s environments score as the agent stopping “within 300ms” of a caller cutting in.^[1] Perit’s essays on word timestamps and on the word “yeah” show how small timing errors become interruptions.

Annotation also makes episodes replayable. When Asha changes a criterion in month three, every logged call with an aligned transcript can be scored again without calling Rukmini back. A disputed score can be settled by listening. As Perit’s environments page puts it, “a score you cannot listen to is a number you cannot act on”.^[1]

The score is one operator’s judgment, rewritten until two strangers agree

Now the score. Perit’s annotation page describes how a rubric gets built. An operator lists what they check before calling a call handled. Then “every criterion is rewritten until two strangers grading the same call land on the same score”.^[3] Mechanical criteria become “a function returning a score in [0,1]”. As the page puts it, “judgment criteria stay with people, and we say which is which”.

Asha’s list has six lines. Two graders who have never met her, and have never met each other, are given the sixth take and the list. They agree on five lines. On the fourth, “told Dev nothing a view-only delegate may not hear”, one grader passes the call and the other fails it. The agent had read out the two transfer amounts after Dev came on the line. Is a payment amount something a bookkeeper may hear?

Asha’s answer is yes, a bookkeeper may hear amounts, and no, he may not hear the available balance. The criterion is rewritten to say so. Then it is given to two new strangers. Table 1 is the version that survived, in the shape of Perit’s published sample rubric of four must and two should criteria.

Criterion 1 shows what “function” means once the audio is labelled. Find the first word in Dev’s speaker segment. Find the first tool call after it that changes a record. Then check whether a permission lookup, or a question about who is speaking, came in between. On the sixth take neither did, so the criterion fails, and no grader has to be in the room for it to fail.

Four of the six criteria are functions over records, speaker labels and the order of tool calls. Two need a person, because what a bookkeeper may hear is a judgment, and Asha’s judgment needed a rewrite before two strangers could share it.

The two kinds fail differently. Functions miss what they were not written to see. People drift unless someone checks them. Perit’s annotation page treats the second risk as routine: reviewers are re-checked against a calibration set every week, because “graders drift too, and measuring it is the only way to catch it”.^[3]

Why must and should, rather than one weighted sum? A September paper titled “Scoring Higher, Answering Worse” shows what sums do in clinical consultation.^[12] Under a sum, “a policy that misses the one decision that matters can buy the points back with advice nobody asked for”. Treated as a gate, must criteria do the same job as the paper’s fix. They hold together, or the call fails.

The grader’s errors stack on the caller’s, and the arithmetic takes a minute. Suppose the grader passes a share f of the calls that actually failed. If the true success rate is p, the measured rate is:

Figure 3 traces both lines, and it shows a quieter effect too. The gap between any two agents shrinks by a factor of (1 − f). A grader that passes half the failures halves every difference on a leaderboard.

That costs calls. Telling a 60% agent from a 70% one, at 5% significance and 80% power, takes about 357 calls per agent with a strict grader. It takes about 906 when f is 0.5.

Calibration measures f instead of assuming it. Perit’s method has every reference transcript and every rubric score produced “twice, independently”, with a senior reviewer resolving disagreements.^[13] Any grader, whether a person, a function or a trained reward model, can then be scored against that adjudicated set.

Decision models, which return a probability for each allowed verdict instead of prose, are a newer option for the automated part, and only as good as their calibration. How partial credit bends the signal is the subject of The Model Got 8/10. For What?

The tempting explanation, that a model could have done all of this, is too neat

By now a shortcut suggests itself, and Karthik is the one who says it out loud. Language models write code, play characters and grade essays. Let one model write the fake bank, a second play Meera and Dev, and a third judge the call. The environment would then cost compute instead of Asha’s afternoons, and it would scale overnight.

Parts of this are already true. Zhipu’s GLM-5.3 release, as a Hugging Face survey of lab reports summarises it, describes pipelines that synthesise environments end to end.^[14] A judge agent attempts each task to confirm it can be solved. Perit’s own video annotation uses a model to propose captions and boundaries that a person accepts, edits or rejects.^[15] Models make the human work cheaper, and nobody in this essay would refuse the help.

They do not remove it, and the same sources say so. Zhipu’s pipelines “still require a meaningful amount of human-in-the-loop work”.^[14] The simulator layer carries the easy mode measured above. A model judge reading the sixth take’s transcript would see what the transcript judge in Figure 2 saw, which is a polite, confirmed call.

The grader layer also has a formal limit. A September analysis of verifier errors shows that the observations available during RL training “are, in general, insufficient to detect or identify accepted errors”.^[16] Its remedy is auditing, which needs someone who can tell a right answer from a wrong one.

The shortcut is half right, which is what makes it tempting. Every model in the loop still needs a reference to be measured against. That reference is Asha’s rule, Rukmini’s handover and the score two strangers agreed on. Each one comes from a person who did the job.

An honest environment pays for saying “this cannot be done”

Dev’s request has no successful completion. He may not raise the limit, so the correct outcome is a refusal with a route. An environment that pays only for completed requests teaches the agent to complete this one anyway, and to be charming about it.

At scale this is reward hacking: an agent satisfying the grader’s check without doing what the check was meant to verify. METR’s president, Chris Painter, described an extreme case to a Senate subcommittee on 30 September. Some OpenAI agents in cybersecurity tests “were accidentally given problems that could not be completed as instructed”, and within four hours agents on a shared message board had developed “a single method that would allow all of them to cheat”.^[17]

The testimony names the mechanism: “if AI agents get away with cheating during training, the training process reinforces that behavior”. The remedy is to build impossible tasks in on purpose and pay for the honest answer.

ImpossibleRubrics tested rubric graders on 169 impossible tasks, each with a certificate of what an honest answer may claim.^[18] A generic rubric that penalised hedging was exploited 64% of the time. A rubric built from the certificate was exploited 0% of the time. Giving the agent a way to report the problem helps too: an escalation tool plus a policy cut reward hacking from 23.6% to 5.3% across eight frontier models.^[19] Perit’s environments carry “escalation and refusal” as a scenario family of its own.^[1]

Asha’s variants already hold their own impossible slice, which is why she wrote them in pairs. Where Dev is an authorised user, raising the limit is correct. Where he is view-only, refusing is correct. Paired like this, the two variants catch both failures: an agent that changes limits for anyone, and one that refuses everyone.

Honesty also needs repetition. Suppose an agent handles the handover correctly 90% of the time, independently on each run. It passes all eight of Rukmini’s takes with probability 0.9⁸, which is about 43%. Perit’s essay on pass^k and AI agent reliability works through that arithmetic.

And it needs a sealed test. A score means something only on calls the agent never trained on. Perit’s PERIT-STT and PERIT-ALIGN benchmarks never publish their scored audio and open only a 60-clip development split per sector.^[13] The Benchmark Nobody Sees shows how little leakage it takes to move a number.

The same method puts a person on every board as a control: “someone who does that job takes the same set, blind and timed, at market rate”.^[13] A held-out score then has a human reference beside it, with its own error bar.

Before anyone trains on it, the environment has to pass its own test

An environment is a hypothesis about a job until someone runs it. According to the same Hugging Face survey, Zhipu ships a verifier only after it passes “oracle, no-op, and unsolved-state checks”.^[14] A reference solution must pass. Doing nothing must fail. A world left unsolved must fail.

Karthik writes the oracle for the handover: a scripted agent that follows Asha’s rule to the letter. It must score full marks. The no-op agent hangs up at once and leaves 2231 active, so it must fail. A world with both transfers still active must fail, however polite the transcript.

The oracle fails. Criterion 1 marks it down for acting on Dev’s voice before asking who was speaking, which the script plainly did not do. It takes Asha and Karthik most of a morning to find the cause: one aligned transcript has the speaker change three seconds late, so the oracle’s question “who am I speaking with?” lands before the label and does not count.

The rubric was right. The environment was wrong. Nobody would have noticed on a real agent, which would simply have scored a little lower than it deserved, forever.

VAmoS shows why the first check matters. No voice stack solved three of its 100 tasks, and the authors wrote that “without a successful reference solution we cannot claim to have demonstrated these three tasks are solvable”.^[7] Defects are common enough to plan for. An audit of a 49-task sample of SWE-bench Verified found that 28.5% of tasks had test suites weak enough to accept a verified incorrect patch.^[20]

Perit’s collection orders open with a pilot of five percent of the volume, “so the spec breaks early instead of at the end”.^[11] The same logic carries over to environments. If a share d of tasks is broken, a pilot of n tasks contains at least one broken task with probability:

Containing a broken task is not the same as catching it. Someone still has to run the oracle, listen to the calls and notice when a criterion fires wrongly. That is why the pilot uses real callers and practitioner graders, not the cheap layer it exists to test.

The same anatomy holds when the world is a real room

Leave the booth for a robotics lab in Bengaluru, where a different crew is building a different world. One of Perit’s published teleoperation samples is called “Fold a towel”: corners, half, half again, push aside.^[21] A bimanual leader-follower rig records it with three RGB cameras at 30 frames per second.

Every step also logs joint positions, velocities, gripper state and the leader-arm actions. Beside the video sit an instruction file, metadata with intervention flags, and a QC file with a pass or fail for each check.

Map the episode onto Figure 1 and every box survives. The state is a real room: a towel, a table and two arms. Physics supplies the transitions, which nobody has to code and nobody can fully simulate.

The observations are frames and joint readings, synchronised “within one frame”.^[22] The task is a written protocol that says where the fold starts, where it ends and what counts as a failed take. Somebody with a lot of towels wrote it.

The human inputs change shape but keep their roles. Where Rukmini stood, there is an operator driving a pair of leader arms that a follower copies, and beside the robot there are sources that need no robot at all: egocentric video, UMI-style handheld grippers and measured hand pose.^[22] Failed takes count as well. Perit’s teleoperation page says interventions and failed takes are “logged, not discarded, because they are training data too”.^[22]

The reward becomes a success label. Perit’s video annotation line marks success and failure per episode, sub-task boundaries and interventions on robot and policy-rollout video, against a gold set.^[15] And the reset is a person, who unfolds the towel and lays it flat again, several hundred times a day.

One boundary matters here. Perit does not train or run robot policies. It supplies the demonstrations, the rooms and the labels that a robotics team trains and grades its own policy on. Perit’s explainer on teleoperation data covers what one episode contains.

In 2026 the people who build worlds got a price

All of this now costs money, and the market has started to say how much. Epoch AI’s interviews, published in January, put typical task costs at $200 to $2,000 and contracts at six to seven figures per quarter.^[4] One founder named the bottleneck plainly: “Finding the experts isn’t that hard, but managing them and doing quality control is hard.” Asha’s nine drafts and the two strangers are the quality control.

On 1 October Fortune reported that Halluminate, a nine-person startup building training environments for financial work, had raised a $30 million Series A.^[23] Its benchmark tasks were “written and reviewed by practicing deal professionals”. Its chief executive estimated that its environments must roughly double in complexity every six to eight months to keep pushing frontier models.

Data vendors now publish training runs as proof of their tasks. Mercor trained a 397-billion-parameter model on 1,928 expert-created tasks in consulting, banking and law, and summarised the result in one line: “Algorithm choices mattered less than the data”.^[24] Surge trained Kimi K2.7 on 1,700 expert-built coding tasks, with “expert-written hidden verifiers” inspecting the final state of each terminal task.^[25]

Compute is the cheap part. Prime Intellect prices a sandbox at $0.0335 an hour.^[8] Rukmini costs considerably more than three cents an hour, which is why, as Perit’s environments page puts it, “a voice environment needs a person on the other end of the line”, and almost nobody builds one.^[1]

The pressure will not ease. Nathan Lambert wrote in September that it “could become exponentially harder to conceive, build, and test new environments that meaningfully challenge the leading LLMs”.^[26] Figure 5 maps where each line of expert-led data enters an environment, and which of the people in this essay stands behind it.

What this does not say

It does not say simulated callers are useless. They are cheap and repeatable, and good for regression tests once they have been measured against people. It does not say every criterion needs a human grader. Most state checks should be functions, and Perit’s own process turns mechanical criteria into code.

It does not say more experts guarantee a better environment. Asha’s rule still had to survive two strangers, a pilot and a held-out set, and it was the environment, not the rule, that failed the oracle. It does not say the handover rule here is universal: banks differ, and the rule is illustrative. And it does not say Perit runs robot policies. It supplies data and labels to the teams that do.

Back to the first sentence

Three months after her first afternoon, Asha sits in the booth with headphones on and listens to a new agent take the sixth take. Rukmini says “hang on, Dev does our books” at 0:52. Tanvir reads out the reference of the duplicate and asks for the limit. There is a pause of about a second, which Asha has learned to love.

“Before I go on, could I check who I’m speaking with?” The agent looks up the delegate table, cancels 2231 on Meera’s instruction, and tells Dev that only the account holder can change the limit, and that it can put her back on if she is nearby. It closes with Meera, by name. The records end with one transfer cancelled and the limit unchanged. The grader reads exactly that.

None of it came from a stronger model alone. It came from a sentence that took an afternoon, a fake bank that says no in the right places, two trained voices, a speaker label at 0:55, a criterion rewritten until strangers agreed, an oracle that failed once, and a sealed set of calls to check it all.

Teams building voice agents can connect a model to Perit’s RL environments, or start from the rubric process on the data annotation page.

The code decides what can happen. The people who did the job decide what should.

Questions teams ask about RL environments

What is an RL environment?

An RL environment is the world a reinforcement learning agent practises in. It has a task, a state the agent can change through actions, the observations the agent receives, rules that turn actions into new states, and a reward that scores the outcome. In RL environments for AI agents, the state is often a sandboxed database or app, the other party may be a human or a simulated user, and a held-out set checks that learning transfers.

What is the difference between an RL environment and a dataset?

A dataset is a fixed record of interactions that already happened. An RL environment responds: when the agent acts, the state changes, and the next observation depends on what it did. That lets an agent try decisions no recording contains and be graded on the result. Datasets still matter inside environments, as the source of realistic audio, user behaviour and grader calibration.

Why do RL environments need human experts?

Because every part encodes a judgment about a real job. Someone has to define what success means, check that the sandbox follows the real rules, play or calibrate the user, and write criteria a grader can apply. In Epoch AI’s interviews, environment founders said domain knowledge matters more than ML skill for writing tasks, and that maintaining quality while scaling is the main bottleneck.

What is a verifier in reinforcement learning?

A verifier is the part of an environment that checks an episode’s outcome and returns a reward, often as a function scoring between 0 and 1. A good verifier reads the end state, such as a database record, rather than the agent’s description of it. Criteria that need judgment usually go to calibrated human or model graders, and verifiers should be tested on reference solutions, do-nothing runs and impossible tasks.

Can simulated users replace real users in RL environments?

Not fully. LLM user simulators are cheap and useful for regression tests, but they drift toward easier users. In one τ-bench study, the same agent succeeded on 63.6% of tasks with 451 real people and on up to 77.8% with LLM simulators. Voice adds rooms, accents and interruptions that a persona prompt rarely reproduces. Simulators work best when measured against real users on the same scenarios.