{"id":775,"date":"2026-09-20T13:24:26","date_gmt":"2026-09-20T13:24:26","guid":{"rendered":"https:\/\/perit.ai\/blogs\/?p=775"},"modified":"2026-09-30T18:42:42","modified_gmt":"2026-09-30T18:42:42","slug":"collection-annotation-or-both","status":"publish","type":"post","link":"https:\/\/perit.ai\/blogs\/collection-annotation-or-both\/","title":{"rendered":"Collection, Annotation or Both?"},"content":{"rendered":"\n\r\n
\r\n
One failed input, three doors, five lines on the annotation bench, and the gate every annotator must clear<\/div>\r\n

A customer gives an amount over a noisy phone line. The speech model records the wrong number, and the rest of the system processes it correctly. In the meeting that follows, someone asks for several hundred thousand rows of \u201cmore data\u201d. The failed clip is twelve seconds long. It is worth examining before choosing the order size.<\/p>\r\n

\u201cMore data\u201d can mean three different jobs.<\/p>\r\n

The team may need new recordings because the acoustic condition, accent or interaction that breaks the model is absent from its files. It may already own the right recordings, in which case it needs those files transcribed, aligned or otherwise labelled. Or it may know only the failure, the twelve-second clip, and need someone to turn that failure into a collection protocol, an annotation guide and a licensable delivery.<\/p>\r\n

Those routes are related, but they are not interchangeable. Annotation cannot create the warehouse recording you never captured. Collection does not automatically tell you which word was spoken at 7.4 seconds. And \u201ctranscription versus alignment\u201d is not the first fork at all: both are jobs inside annotation.<\/p>\r\n

Before choosing a service, check whether the inputs your model needs already exist.<\/p>\r\n

\r\n
Do you already have the inputs that contain the failure?<\/div>\r\n
\r\n
No<\/span> Data Collection<\/strong>\r\n

Record or author the missing situations, people, devices, rooms or language.<\/p>\r\n<\/div>\r\n

Yes<\/span> Data Annotation<\/strong>\r\n

Turn files you own into transcripts, timestamps, masks, tracks, spans or judgments.<\/p>\r\n<\/div>\r\n

I only know the failure<\/span> Request Data<\/strong>\r\n

Start with one bad input; scope collection and annotation together.<\/p>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n

Check what is missing before choosing a service<\/h2>\r\n

Suppose your model misses product names when a shopkeeper speaks beside a refrigerator. \u201cWe need annotated audio\u201d sounds like a reasonable brief. It is not yet enough to choose a service.<\/p>\r\n

Open the files you already have. If they contain the same shops, accents, devices and background hum, the missing ingredient may indeed be labels: a verbatim reference transcript, word-level times, speaker turns or entity spans around the product names. If the files are mostly clean office speech, better labels will make the clean office data more precise. They will not introduce the refrigerator, the handset microphone or the shopkeeper.<\/p>\r\n

Start with the failed input. Does that condition appear in your files? What should a person label? Which measurement would show that the new batch helped?<\/p>\r\n

This is why all three Perit routes use nearly the same opening language: send a clip, photograph, paragraph or task the model handles badly. You do not need to solve the data specification before the first conversation. You do need to bring the failure close enough that somebody can inspect it.<\/p>\r\n

A service name is a conclusion. The failed input is the evidence that gets you there.<\/div>\r\n

Collect the missing situations<\/h2>\r\n

Choose Data Collection<\/a> when the model needs experience your current files do not contain. That missing experience might be a locale, an accent, a room, a camera angle, a handset, a lighting condition, a document type or a way people phrase the same intention.<\/p>\r\n

Collection is not restricted to speech. Perit lists four modalities on the same recruiting and QA machinery: speech and audio; video; image; and text and conversation. A contributor might record a two-party call with each speaker on a separate channel, film an everyday task on a chest camera, photograph the same shelf under two lighting conditions, or write a dialogue in the register of the sector they work in. The protocol and equipment change. Recruiting to quotas, capturing consent, running a pilot, reviewing quality and delivering weekly remain the same.<\/p>\r\n

Speech is currently the deepest line: 4,000+ hours delivered, nine live locales and capacity for 50 hours a day<\/strong>. Those figures establish throughput, but the collection spec must still name the people, conditions, devices and outcomes that belong in your batch.<\/p>\r\n

That is the purpose of the pilot. After the brief, Perit returns a scope and quote; then five percent of the planned volume<\/strong> is delivered before the main run. The pilot exists so a vague instruction becomes an obvious problem on a small batch. Perhaps \u201cnoisy shop\u201d produces shopping-centre music when the model actually fails on compressor hum. Perhaps \u201cHindi speakers\u201d is too broad for a regional failure. It is much cheaper to discover that at five percent than at one hundred.<\/p>\r\n

\r\n

A collection-shaped failure:<\/strong> Your image model performs well on catalogue photographs but misses products on crowded shelves photographed by customers at night. You have labels for clean product images; you do not have the user device, shelf density or light. The missing thing is not another pass over the old boxes. It is new imagery collected across the conditions in which the model is expected to work.<\/p>\r\n<\/div>\r\n

Collection can include labels where ordered. That does not blur the distinction; it means one work order can cross it. The people and situations are collected first, then the resulting files move through an annotation specification. \u201cCollection\u201d describes how the raw evidence comes into existence, not a requirement that it remain raw.<\/p>\r\n

Annotate the inputs you already have<\/h2>\r\n

Choose Data Annotation<\/a> when the useful inputs already sit in your bucket but the answer is not attached to them. This is also the direct answer to a question the FAQ<\/a> makes explicit: yes, Perit annotates data customers already own; in fact, most annotation work is performed on customer audio, images or video.<\/p>\r\n

The work runs through Foundry<\/a>. The published bench has 650 people<\/strong>: 250 transcribers, including 50 on fixed-term contracts, and 400 freelance word aligners. It has delivered more than 3,000 transcribed hours and more than 1,000 aligned hours across nine locales. The service page reports 95%+ QA-sample accuracy.<\/p>\r\n

That last number deserves its full name. It is QA-sample accuracy, checked against the project standard; it is not a promise that every possible label on every possible dataset is \u201c95% accurate\u201d. For audio, word error rate is reported separately against an adjudicated reference. Keeping those measures separate is useful because a batch can follow its guide consistently while the guide itself is incomplete, or produce a good transcript while still placing a timing boundary poorly.<\/p>\r\n

Graders stream the files rather than copy them, and the project is pinned to the agreed region. What remains is to decide what answer should be attached to each input.<\/p>\r\n

Choosing the annotation task<\/h2>\r\n

This is where transcription and alignment finally become the right question. Audio happens to split into two distinct crafts, while image, video and text each need their own label geometry. A useful way to choose is to finish this sentence: After annotation, I need to know\u2026<\/em><\/p>\r\n

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Bench line<\/th>\r\nThe question it answers<\/th>\r\nWhat comes back<\/th>\r\nThe common mix-up<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
Transcription<\/td>\r\nWhat was actually said?<\/td>\r\nVerbatim text, with fillers, repeats and false starts retained under the written convention.<\/td>\r\nAsking for a \u201cclean\u201d transcript, then trying to evaluate a model on disfluencies that were edited away.<\/td>\r\n<\/tr>\r\n
Alignment<\/td>\r\nWhen did each word begin and end?<\/td>\r\nWord-level start and end timestamps placed on top of a transcript.<\/td>\r\nTreating alignment as a substitute for transcription. A timestamp needs a word to belong to.<\/td>\r\n<\/tr>\r\n
Image labels<\/td>\r\nWhere is the object, and what is true of it?<\/td>\r\nBoxes, polygons, masks, attributes, classes or extracted document fields.<\/td>\r\nChoosing a box when the model must learn an exact boundary, or a mask when a coarse location is enough.<\/td>\r\n<\/tr>\r\n
Video labels<\/td>\r\nWhat happened, when, and to which object?<\/td>\r\nTask boundaries, tracks across frames, segment captions and episode-level success or failure.<\/td>\r\nLabelling isolated frames when the model needs the continuity of an action.<\/td>\r\n<\/tr>\r\n
Text labels<\/td>\r\nWhich answer is better, which rule passed, or which span carries the meaning?<\/td>\r\nPreferences, rubric scores, entity and intent spans, taxonomy classes or policy labels.<\/td>\r\nUsing a vague rating such as \u201cgood\u201d where a stranger needs an applicable criterion.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

Transcription writes the words<\/h3>\r\n

Use transcription when the target is a reliable textual account of the audio. On Perit’s default verbatim convention, \u201cuh\u201d, repeated words and abandoned starts remain because they happened. Speaker attribution, diarization and entity or intent spans can sit beside the transcript when the model needs to distinguish who said what or which tokens carry the transaction.<\/p>\r\n

Then the edge cases arrive. Is \u201ctwenty-one\u201d one token or two? How should a partially audible account number appear? A transcription is not merely somebody typing; it is somebody applying the same written decisions thousands of times.<\/p>\r\n

Alignment puts time under those words<\/h3>\r\n

Use alignment when correctness depends on when<\/em> the system heard something, not only what<\/em> it heard. Each word receives a start and an end. That makes it possible to inspect latency, subtitle timing, turn-taking or the exact interval around an error.<\/p>\r\n

Alignment presupposes a transcript. If you already have a trusted transcript, it can go directly to the alignment line. If you have only audio, the sensible order may contain both: transcribe the utterance, adjudicate the words, then align those words to the waveform. Precisely timestamping the wrong transcript produces precise wrongness.<\/p>\r\n

Images need geometry; video needs continuity; text needs a rule<\/h3>\r\n

For an image, the decision is often about the shape of the answer. A bounding box says roughly where an object is; a polygon or mask traces it more closely; an attribute says something about it. The most detailed label is not automatically the best one. Detail that the model never uses adds cost and creates extra boundary decisions for annotators to disagree about.<\/p>\r\n

Video adds time. A box may need to follow the same object across frames. A long episode may need task and sub-task boundaries, captions for each segment, and a final success label. For embodied data, the difference between \u201cthe gripper touched the cup\u201d and \u201cthe cup was lifted and remained stable\u201d is the difference between an event and an outcome.<\/p>\r\n

Text usually makes the hidden standard visible. A preference label says which of two replies is better. A rubric score says which specific criteria a response passed. A span label identifies the words that carry an entity or intent. If two careful people cannot apply the criterion the same way, the first suspect should be the criterion, not the people.<\/p>\r\n

Scope collection and annotation together<\/h2>\r\n

Request Data<\/a> is for the team that cannot honestly answer the earlier questions yet. It knows the model fails on one call, one shelf, one form or one style of conversation. It does not know how many examples will expose the pattern, which conditions need quotas, whether the delivery needs transcripts or masks, or how those labels should be judged.<\/p>\r\n

Request Data covers collection and annotation together. Perit turns the failure into a volume, locale and condition plan, collects the inputs, annotates them to the agreed depth and licenses the delivery. The team reviews a scope and quote, checks the five-percent pilot, then receives weekly batches with quality reports and reviewer records.<\/p>\r\n

\r\n

A request-shaped failure:<\/strong> Your support agent mishandles callers who change the requested amount halfway through a sentence. You have three examples and no stable taxonomy for them. The right opening request is not \u201c50,000 aligned utterances\u201d. It is the three failed calls, the correct behaviour and the question you need the set to answer. Collection volume and annotation depth can then follow from the evidence.<\/p>\r\n<\/div>\r\n

Not knowing the specification is not the same as having no requirements. You still know what failed and what a successful model should do. Request Data is useful precisely in the space between those two facts, where a data programme has to be designed.<\/p>\r\n

Calibrating contributors before production<\/h2>\r\n

Once a route has been chosen, the more important trust question begins: how does a stranger learn to make the same decision your team would make?<\/p>\r\n

Each data type uses the same calibration process. The waveform or image changes; the contributor still trains against a guide, passes a test and receives production review.<\/p>\r\n

\r\n
01<\/span>Read the guide<\/strong>One versioned convention defines the labels and edge cases.<\/span><\/div>\r\n
02<\/span>Train on ~30 samples<\/strong>Answers are visible, so the written rule becomes concrete.<\/span><\/div>\r\n
03<\/span>Pass a held-back test<\/strong>No pass means no access to paid production work.<\/span><\/div>\r\n
04<\/span>Produce under sampling<\/strong>QA continues; work below threshold returns to calibration.<\/span><\/div>\r\n<\/div>\r\n

The guide comes first because agreement cannot be inspected without an agreed answer. Training uses about thirty examples with answers visible; the test withholds them. Production access follows only after a pass, and production itself is sampled rather than trusted forever.<\/p>\r\n

Four numbers then make the process inspectable. QA-sample accuracy<\/strong> asks whether reviewed items match the guide. WER<\/strong>, where the work is audio, compares the transcript with an adjudicated reference. Inter-annotator agreement<\/strong> asks whether independent people reach the same answer on the double-passed portion. Rework rate<\/strong> records how much fell below threshold and had to be redone. The published policy is to ship these numbers with the batch whether or not they flatter the vendor.<\/p>\r\n

A clean accuracy number after silent corrections tells you less than the same number beside the amount of correction required. Rework is part of the account of how delivered quality was produced.<\/p>\r\n

The gate also explains why the taxonomy is not the deepest decision. Collection, transcription, alignment and masks are different crafts. They can still fail in the same way: an ambiguous guide lets two sensible people create two incompatible datasets. The line on the invoice tells you what work was ordered. Calibration tells you whether the work means the same thing from one item to the next.<\/p>\r\n

Most real projects occupy more than one box<\/h2>\r\n

A clean decision tree is helpful at the entrance. Real work often becomes a pipeline.<\/p>\r\n