Foundry is where the work happens: transcribers who write what was actually said, aligners who put a timestamp on every word, reviewers who sample and score every batch. Audio is what has shipped. Image, video and text run through the same gate.
Two crafts, both running today. Transcribers write what was actually said, fillers and all. Aligners put a start and end time on every word.
50 on fixed-term contracts, the rest freelance — and hiring against every open sector.
Freelance bench, timestamped to the word so a model can be scored on when it heard something, not just what.
The guide, the training module and the test are the same shape whatever the file is. Audio carries real numbers because it has shipped; the rest are open on the same bench.
Verbatim transcription, word-level alignment, speaker attribution and entity spans, to one written convention per project.
Boxes, polygons, masks and attributes, with the guide and the gold set that make two annotators land on the same answer.
Temporal segmentation, tracking and captions — including the embodied tasks on the Physical Intelligence pages.
Preference judgments, rubric scores and span labels by people who work in the sector, against a rubric a stranger can apply.
Every annotator clears the same gate, on every project, however long they have been on the bench.
Join the bench on FoundryOne written convention per project — casing, tags, how a number is written out, what counts as an edge case. Versioned; the version ships with the batch.
About thirty samples with the answers visible, so the guide stops being abstract before anyone is paid for it.
A held-back set, graded against the guide. No pass, no production access — the test can be retaken after re-reading, not after arguing.
Work goes out with QA sampling behind it. Below threshold routes back to calibration, and rejected work is not billed to the customer.
A rubric is not a list of nice qualities. It is what one operator checks before they call a call handled, written so that a stranger reaches the same verdict.
We sit with an operator and write down what they check before they call a call handled. That list is the rubric's first draft.
Every criterion is rewritten until two strangers grading the same call land on the same score.
The mechanical criteria become a function returning a score in [0,1]. Judgment criteria stay with people, and we say which is which.
Nothing is delivered without the numbers it was produced at: QA-sample accuracy, WER against the adjudicated reference, inter-annotator agreement, and what was reworked.
Card dispute · caller reports two charges they do not recognise
Two strangers, same call, same score — or the criterion gets rewritten. Agreement on every criterion is measured and reported per batch.
Every project starts from one written convention, agreed with the customer and versioned. Disagreement about what a tag means is a guide bug, and it gets fixed in the guide.
Training on about 30 samples, then a held-back test graded against the guide. Nobody reaches paid production work without clearing it.
Delivered batches carry accuracy against the guide on a QA sample, WER against an adjudicated reference, and inter-annotator agreement on the double-passed portion. All three ship with the batch, whether they flatter us or not.
When IAA drops on a task, we treat it as the guide being ambiguous before we treat it as an annotator being wrong. Agreement is measured per batch and per reviewer, so drift shows up as a trend rather than a surprise.
Work below threshold is redone before delivery and is not billed. The rework rate goes in the batch report, because hiding it would make the accuracy number meaningless.
Customer delivery is QA-sampled. Benchmark reference transcripts are stricter: written twice, independently, with a senior reviewer adjudicating every disagreement — nothing else is defensible as a scoring baseline.
Nobody touches production data before passing a calibration set. Reviewers are re-checked against that same set every week — graders drift too, and measuring it is the only way to catch it.
| Role | Bar to enter | What they do |
|---|---|---|
| Annotator | Sector experience, passes a calibration set before touching production data | Transcription, entity spans, turn boundaries, preference judgments |
| Reviewer | Sustained QA-sample accuracy above threshold, across batches | Second pass, adjudication on disagreement, agreement reporting, batch sign-off |
| Rubric author | Practising operator — the person who would sign off on this call at work | Writes what a good answer is, in criteria a stranger can apply |
| Calibration lead | Reviewer who has held threshold while running a guide revision | Weekly reviewer checks, gold-set maintenance, guide revisions |