{"id":823,"date":"2026-09-24T12:28:37","date_gmt":"2026-09-24T12:28:37","guid":{"rendered":"https:\/\/perit.ai\/blogs\/?p=823"},"modified":"2026-09-30T18:42:46","modified_gmt":"2026-09-30T18:42:46","slug":"what-95-accuracy-means-in-data-annotation-qa","status":"publish","type":"post","link":"https:\/\/perit.ai\/blogs\/what-95-accuracy-means-in-data-annotation-qa\/","title":{"rendered":"What “95% Accuracy” Means in Data Annotation QA"},"content":{"rendered":"\n\r\n
\r\n
A refund call, three numbers hiding inside one quality report, and what each is actually catching<\/div>\r\n

A quality report arrives with \u201c95%+ QA-sample accuracy\u201d at the top. Before accepting that number, the team needs to know what was scored. A batch can follow its annotation guide, match a transcription reference or maintain quality in production. Those are separate checks.<\/p>\r\n

Take a thirteen-word sentence from a refund call: \u201cthe balance owing on the account is two hundred and forty seven dollars.\u201d A transcriber writes \u201cthe balance owing on the account is two forty seven dollars.\u201d The words \u201chundred\u201d and \u201cand\u201d are missing. To judge that error, we need the transcription reference and the scoring convention.<\/p>\r\n

Three measurements help explain the headline: word error rate, inter-annotator agreement and rework rate. Each catches a different failure.<\/p>\r\n\r\n

A single number is doing three jobs<\/h2>\r\n

Start with what each measurement actually asks.<\/p>\r\n

Word error rate compares the annotator\u2019s transcript with an adjudicated reference. It measures the edits needed to turn one string into the other. That makes it useful for transcription; it is less suited to judgments without a reference answer.<\/p>\r\n

Inter-annotator agreement measures how often two trained people independently reach the same answer. Product-name spans, preferences between model replies and rubric scores may all require judgments. Agreement helps show whether the guide gives people enough detail to apply those judgments consistently.<\/p>\r\n

Rework rate tracks the share of sampled production work that fails QA and has to be redone. It helps reveal drift after calibration, including a decline in quality from someone who passed the original test.<\/p>\r\n\r\n

\r\n
\r\n\r\n01<\/span>WER<\/strong>\r\n

Distance from an agreed reference. Answers: was the transcript correct?<\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n02<\/span>Inter-annotator agreement<\/strong>\r\n

Distance between two independent annotators. Answers: is the task well-specified?<\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n03<\/span>Rework rate<\/strong>\r\n

Share of sampled production work sent back. Answers: is quality holding up over time?<\/p>\r\n\r\n<\/div>\r\n<\/div>\r\n

High agreement does not guarantee useful labels. Two annotators can follow the same rigid guide and both miss what the client needed. Likewise, a transcript can match every reference word while reviewers disagree about whether the call was resolved. Read the metric that corresponds to the task.<\/p>\r\n\r\n

What WER is actually counting<\/h2>\r\n

WER counts the minimum substitutions, deletions and insertions needed to turn a hypothesis transcript into the reference.<\/p>\r\n\r\n

WER = (S + D + I) \/ N<\/span> S = substitutions, D = deletions, I = insertions, N = words in the reference<\/span><\/div>\r\n

Take the refund sentence. The reference has twelve words:<\/p>\r\n\r\n

the balance owing on the account is two hundred and forty seven dollars<\/span> reference: 13 words<\/span><\/div>\r\n

The reference is \u201cthe \/ balance \/ owing \/ on \/ the \/ account \/ is \/ two \/ hundred \/ and \/ forty \/ seven \/ dollars\u201d: thirteen words. The hypothesis deletes \u201chundred\u201d and \u201cand\u201d; nothing else changes.<\/p>\r\n\r\n

the balance owing on the account is two hundred<\/mark> and<\/mark> forty seven dollars<\/span> hypothesis: two deletions, zero substitutions, zero insertions<\/span><\/div>\r\n

With S = 0, D = 2, I = 0 and N = 13, WER = 2\/13 \u2248 15.4%. The sentence may look plausible on a quick read, but two missing words out of thirteen give it a double-digit error rate.<\/p>\r\n

Now change one thing: suppose the guide does not require filler words to be transcribed, and the only difference between hypothesis and reference is that the caller said “um” once and the annotator correctly dropped it. A guide that scores fillers would count that as a deletion; a guide that excludes fillers scores it as zero error on the same audio. The number moves by a guide decision, not by annotator skill, which is why the report has to name the guide version next to the score.<\/p>\r\n\r\n

\r\n
Two bars. Top: a clip with fillers included in scoring shows a higher WER of 18.7%. Bottom: the same clip with fillers excluded from scoring shows a lower WER of 6.9%, because the scoring convention changed while the annotator\u2019s work stayed the same. The same clip scored under two guide conventions Same clip, same annotator, same transcript; different scoring convention Scoring convention fillers scored fillers excluded 18.7%6.9%0%10%20%30%Word error rate (WER) on the same clip<\/div>\r\n
Figure 1. Illustrative scores for audio with frequent fillers. The annotator\u2019s work is identical; only the guide\u2019s definition of an error changes.<\/figcaption><\/figure>\r\n

This is why a WER on its own, without the guide version it was scored against, is close to unreadable. The number is real, but the ruler it was measured with is a choice.<\/p>\r\n\r\n

Using disagreement to improve the guide<\/h2>\r\n

Span boundaries, response preferences and call-resolution labels often require judgment. The guide defines what counts as a correct label. Agreement then tests whether a second trained annotator can apply that definition the same way.<\/p>\r\n

Take a span-labelling task: mark the product name in “I want the Samsung Galaxy S24 Ultra in the black colour.” One annotator, trained to capture the full retail name, marks the span as “Samsung Galaxy S24 Ultra.” A second annotator, reading the same guide slightly differently, marks only “Galaxy S24 Ultra,” treating “Samsung” as the brand rather than part of the product name.<\/p>\r\n

A simple way to score the overlap is token-level agreement: the number of shared tokens divided by the number of tokens either annotator included:<\/p>\r\n\r\n

agreement = |A \u2229 B| \/ |A \u222a B|<\/span> A = annotator one’s span tokens, B = annotator two’s span tokens<\/span><\/div>\r\n

The first span is {Samsung, Galaxy, S24, Ultra}, four tokens. The second is {Galaxy, S24, Ultra}, three tokens. Their intersection has three tokens and their union has four, so agreement is 3\/4 = 75%.<\/p>\r\n

Here the 75% agreement exposes a guide gap. Neither annotator was told whether the brand name belongs in the span. Both decisions are defensible, so the next step is to clarify the guide.<\/p>\r\n

This is the practical use of inter-annotator agreement: it is measured on a double-passed slice of the work specifically so that low agreement can be traced back to guide language before it spreads through the rest of the batch.<\/p>\r\n\r\n

A WER tells you whether the work matched the reference. Agreement tells you whether two people reading the same instructions would have produced the same reference in the first place.<\/div>\r\n

Rework rate is the number that catches drift<\/h2>\r\n

WER and agreement describe a batch, or a double-passed sample, at a particular time. They do not establish that an annotator who passed calibration in week one still meets the standard in week six.<\/p>\r\n

Rework rate measures sampled production items that fail ongoing QA and return to the annotator or trigger recalibration. These are items produced after the original calibration test.<\/p>\r\n\r\n

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Metric<\/th>\r\nMeasured on<\/th>\r\nCatches<\/th>\r\nMisses<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
WER<\/td>\r\nEvery transcribed item, against an adjudicated reference<\/td>\r\nTranscription errors relative to the agreed guide<\/td>\r\nWhether the guide itself is well-specified<\/td>\r\n<\/tr>\r\n
Inter-annotator agreement<\/td>\r\nA double-passed sample<\/td>\r\nAmbiguous or under-specified guide language<\/td>\r\nDrift that develops after calibration passes<\/td>\r\n<\/tr>\r\n
Rework rate<\/td>\r\nAn ongoing sample of production work<\/td>\r\nIndividual or bench-wide drift over time<\/td>\r\nErrors too subtle for the sampling rate to surface<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

A rising rework rate alongside a stable WER needs investigation. The transcripts checked for WER may still match their references while more production items fail other reviews. Check which annotators, conditions and error types account for the increase before changing sampling or recalibration.<\/p>\r\n\r\n

\r\n

Audio example: a batch reports 96.5% QA-sample accuracy, 4.1% WER against the adjudicated reference, 91% agreement on the double-passed sample and 2% rework. Those figures describe guide compliance, transcript errors, consistency between annotators and production failures. If rework rose to 14%, the team would need to investigate even if WER and agreement stayed stable.<\/p>\r\n\r\n<\/div>\r\n

The same checks apply outside audio, with a different reference metric. For bounding boxes, intersection-over-union measures how closely the box matches an adjudicated reference. For video success\/failure labels, use accuracy against reviewed outcomes. Agreement and rework remain useful in both cases.<\/p>\r\n\r\n

\r\n

Image example: a shelf-detection batch has 0.88 mean IoU against the reference, 94% agreement on the double-passed sample and 3% rework. The averages still need a condition breakdown. If low-light, crowded shelves account for the weak boxes, that matters to a team whose model already fails on those photos.<\/p>\r\n\r\n<\/div>\r\n

Which metric to check first<\/h2>\r\n

The three numbers do not deserve equal attention on every project, because they are sensitive to different things going wrong.<\/p>\r\n\r\n