{"id":877,"date":"2026-09-29T14:03:38","date_gmt":"2026-09-29T14:03:38","guid":{"rendered":"https:\/\/perit.ai\/blogs\/?p=877"},"modified":"2026-09-30T18:42:45","modified_gmt":"2026-09-30T18:42:45","slug":"pilot-economics","status":"publish","type":"post","link":"https:\/\/perit.ai\/blogs\/pilot-economics\/","title":{"rendered":"Order 5% First: How a Data Collection Pilot Works"},"content":{"rendered":"\n\r\n
\r\n
A spec that read fine on paper, the batch that proved it wasn’t, and the arithmetic behind ordering five percent first<\/div>\r\n

The brief said \u201crecord two-party calls in a noisy shop.\u201d Ten contributors followed it, and the pilot arrived with shopping-centre music, till beeps and a PA announcement about detergent. The model was failing on compressor hum: the low drone of a walk-in fridge, thirty decibels quieter. The brief had left \u201cnoisy\u201d open to interpretation. The pilot exposed the gap.<\/p>\r\n

Perit delivers five percent of a collection order, or ten to thirty episodes for embodied data, before the main run. That first batch tests whether the written spec produces the evidence the model needs.<\/p>\r\n\r\n

Testing the written brief<\/h2>\r\n

A collection or annotation brief describes who to record, under which conditions, doing what, and how to label the result. Those instructions must be specific enough for a contributor to follow without asking the person who wrote them what \u201cnoisy\u201d means.<\/p>\r\n

The writer usually starts with a failed clip, an example of poor lighting or an accent the model misses. Translating that example into capture instructions can lose detail. \u201cCompressor hum\u201d becomes \u201cmall ambience\u201d, and the mismatch may stay hidden until delivery.<\/p>\r\n

The pilot runs a small slice through recruiting, capture or annotation, review and delivery. The team receives that slice and its quality report while changes to the rest of the order are still cheap.<\/p>\r\n\r\n

The arithmetic of catching it early<\/h2>\r\n

Perit’s published contributor rate for approved two-party speech is $20 per audio hour<\/strong>. Take a mid-sized order: 2,000 hours of accent-matched call recordings for a new locale. Contributor payouts alone come to:<\/p>\r\n\r\n

2,000 hours \u00d7 $20\/hour = $40,000<\/span> contributor pay only; review, QA and delivery overhead are quoted separately<\/span><\/div>\r\n

Suppose the team misses the noise-condition gap until all 2,000 hours arrive. It can accept data that does not address the failure, or revise the spec and repeat much of the order. A full repeat means another $40,000 in contributor pay and the time needed to record 2,000 hours again.<\/p>\r\n

A 5% pilot of the same order is 100 hours. The contributor cost of finding the gap is:<\/p>\r\n\r\n

100 hours \u00d7 $20\/hour = $2,000<\/span> the cost of finding out the spec was wrong, instead of the cost of the spec being wrong at full volume<\/span><\/div>\r\n

That is $2,000 to discover the flaw before committing a $40,000 batch that might need substantial rework.<\/p>\r\n\r\n

When the pilot pays for itself<\/h2>\r\n

A pilot has a cost even when the spec is correct. Its expected value depends on how often it catches a gap and how much rework that gap would cause.<\/p>\r\n

Suppose, illustratively, that a first-time spec for a new locale, device or condition has roughly a one-in-three chance of containing a definition gap serious enough that a meaningful share of the batch would need to be redone if it went undiscovered until full delivery. Compare the expected cost of the two strategies:<\/p>\r\n\r\n

E[no pilot] = p \u00d7 (full redo cost) = 0.33 \u00d7 $40,000 \u2248 $13,300<\/span> expected cost of skipping the pilot across both flawed and clean specs<\/span><\/div>\r\n
E[with pilot] = (pilot cost) + p \u00d7 (cost of respecifying before the other 95% runs)<\/span> the pilot cost is paid on every project; the correction cost only applies to the flawed fraction, and it is much smaller once caught early<\/span><\/div>\r\n

The pilot costs $2,000 in either case. If it catches a gap, the team can revise the schema and rerun a small slice before committing the other 1,900 hours. Under the illustrative one-in-three failure assumption, this costs less on average than risking a second $40,000 order. The calculation includes pilot costs on the two-thirds of projects whose specs were already sound.<\/p>\r\n\r\n

\r\n
A line chart with the probability of a spec flaw on the horizontal axis from 0 to 100 percent, and expected cost in dollars on the vertical axis. The no-pilot line rises steeply and linearly from zero. The with-pilot line starts near two thousand dollars and rises much more slowly, crossing below the no-pilot line almost immediately and staying well under it across the whole range. Expected cost with and without a 5% pilot Illustrative curves for the 2,000-hour example at $20 per audio hour With 5% pilotNo pilot $0 $10k $20k $30k $40k 0% 25% 50% 75% 100% Chance the first-draft spec has a costly gap Expected cost (USD) No pilot With 5% pilot Pilot floor \u2248 $2,000paid on every order<\/div>\r\n
Figure 1. Illustrative costs for the 2,000-hour order. The pilot costs something even when the spec is correct. When a flaw is caught early, the remedy is a revised spec and a small rerun rather than another full order.<\/figcaption><\/figure>\r\n

Collection pilots and annotation samples<\/h2>\r\n

Perit uses different checks for collection and annotation. A collection pilot is 5% of ordered volume. An annotation sample has a fixed size: ten minutes of audio, 100 images or one page of text, returned within five working days with an agreement report.<\/p>\r\n

The collection pilot checks whether the recruiting and capture instructions produce the right inputs across the required conditions. The annotation sample checks whether the team can apply your guide to files you already recognise. An order that needs both stages uses both checks: first a 5% collection pilot, then annotation sampling on that slice before scaling.<\/p>\r\n\r\n

Choosing the pilot size<\/h2>\r\n

The size of the pilot is itself a decision worth defending, not a round number chosen for convenience.<\/p>\r\n

On the 2,000-hour order, a 1% pilot is 20 hours. That may involve only a few contributors and one or two recording days. It can miss a phone model, dialect or room type that becomes common in the full run, leaving a gap undiscovered until hour 400.<\/p>\r\n

A 20% pilot is 400 hours and costs $8,000 in contributor pay. It may catch the same flaw, but costs four times as much as the 5% pilot. Once the sample covers the conditions the team needs to inspect, adding volume has diminishing value.<\/p>\r\n

Five percent of a multi-thousand-hour order still includes a hundred-plus hours, several contributors and multiple recording days. For embodied data, the pilot uses ten to thirty episodes across several operators and task attempts. It runs in the second week of a four-phase process that starts with a one-page brief.<\/p>\r\n\r\n

\r\n
Step 01<\/span>Day 0: Brief<\/strong>Task, environment, conditions, device or rig, and volume agreed in one page.<\/span><\/div>\r\n
Step 02<\/span>48 hours: Scope and quote<\/strong>Quantity, locales, conditions and schema translated into a concrete plan.<\/span><\/div>\r\n
Step 03<\/span>Two weeks: Pilot<\/strong>5% of volume, or 10\u201330 episodes, delivered with a quality report.<\/span><\/div>\r\n
Step 04<\/span>Weekly: Full delivery<\/strong>Protocol refined against pilot findings, then a steady delivery cadence.<\/span><\/div>\r\n<\/div>\r\n
\r\n

An egocentric-video example: the brief asks for chest-mounted footage of kitchen cleanup. The tasks and rooms are correct, but half the operators mount the camera high enough that their hands leave the frame when reaching into overhead cupboards. Across thirty pilot episodes, the team can fix the SOP with a rig-height requirement and a reference frame for camera angle. Across three thousand episodes, the same omission means substantial re-shooting.<\/p>\r\n\r\n<\/div>\r\n

Testing recruiting, capture and review<\/h2>\r\n

It’s tempting to think of the pilot as checking whether contributors can follow instructions. It’s actually checking something broader: whether the brief, the recruiting quota, the recording protocol, the review process and the delivery format all fit together the way the plan assumed they would.<\/p>\r\n

The pilot can reveal that an accent is harder to recruit than expected, or that review takes longer than the delivery schedule allows. A written spec cannot establish those facts. Running the workflow can.<\/p>\r\n

Speech, video, image, text and the eight embodied-data types all use the same four-phase process. The capture instructions vary; the pilot still checks them before the main commitment.<\/p>\r\n\r\n

A pilot is not a smaller version of the order. It’s a test of whether the order, as written, actually means what you think it means.<\/div>\r\n

What a pilot report should actually tell you<\/h2>\r\n

Ask for the same accuracy, agreement and rework measures that will accompany full delivery. A small batch without those results tells you little about whether the pipeline is ready to scale.<\/p>\r\n

Check which episodes or hours passed QC, why any failed, and whether the cause was contributor error or an ambiguous spec. Would the failure recur if the brief stayed unchanged? A clean pilot is useful too: it shows the instructions worked on the sample tested.<\/p>\r\n

Passing the pilot does not end QA. The full run may introduce new contributors, recruiting pressure and edge cases that the pilot did not contain. Weekly quality reports check whether the approved spec continues to hold as volume grows.<\/p>\r\n\r\n

The short version<\/strong> A brief is a guess about what real people, in real conditions, will actually produce. The pilot is the cheapest way to test that guess before the full order depends on it being right. In this example, spending $2,000 on the pilot can avoid a $40,000 repeat.<\/div>\r\n
\r\n

Start with the failure, not the full order<\/h2>\r\n

Every Perit collection and annotation programme runs a pilot before scaling: 5% of volume for speech, video, image and text, or 10 to 30 episodes for embodied data. Send the failing clip, photo or task<\/a> and get a scope and quote within 48 hours, with the pilot landing two weeks later.<\/p>\r\n\r\n<\/div>\r\n