Engineering Quality in AI Data: Combining Human Expertise with Automated QA

Where Does Quality Break Down in AI Data?

Some problems are easy to identify, such as a missing label or an incorrect value. Others are less obvious. Two annotators might look at the same image and choose different labels, or an unclear part of a transcript might be interpreted differently depending on the context.

These problems generally fall into three areas:

When a dataset is small, these issues can often be found through manual review. As the volume increases, checking everything by hand becomes slower and harder to manage. This is where automated checks and human review can complement each other, with each handling the types of problems it is better suited to catch.

Why Small Errors Matter at Scale

The impact of an individual annotation error may seem small when looking at a single record. The situation changes when the same type of mistake appears repeatedly across a large dataset.

For example, if an annotation guideline is interpreted differently by different reviewers, the resulting inconsistencies can spread across thousands of records. Similarly, a validation rule that is too broad may repeatedly flag valid annotations, while a rule that is too narrow may allow certain errors to pass unnoticed.

This makes dataset quality more than a matter of finding individual mistakes. The process also needs to identify patterns in those mistakes and determine why they are happening.

A useful QA process therefore needs to answer two questions:

Which annotations need attention, and what is causing those issues to appear in the first place?

Automated checks can help identify large numbers of potentially problematic records. Human reviewers can then examine the cases where context, interpretation, or project-specific knowledge is needed.

Automated QA at Scale

When AI datasets grow to thousands or millions of records, checking every annotation manually is not practical. Automated QA provides a first layer of validation by applying the same rules across a large volume of data.

For example, an automated system can flag a missing label, an invalid category, a duplicate annotation, or a bounding box that extends beyond an image. These are problems that can be described through clear rules and checked consistently.

Within Perit’s data annotation workflow, automation can help identify cases that need a closer look. Instead of replacing human reviewers, it can narrow down where their attention is needed.

This creates a practical division of work. Automation handles repeatable checks, while human reviewers focus on cases that require context or judgment. The approach becomes particularly useful when the same quality standards need to be maintained across large and varied datasets.

The Role of Human Review

Automation is effective when an issue can be defined by a clear rule. However, not every annotation can be judged that way. Some cases require a reviewer to understand the surrounding context before deciding whether an annotation is actually incorrect.

Consider a dataset containing 1,000 annotations. An automated QA system flags 80 of them as potential issues. A human reviewer examines those 80 cases and finds that 60 are genuine errors, while the remaining 20 are valid annotations.

We can use precision to measure how many of the flagged cases were actually errors.

In this example:

This means that 75% of the annotations flagged by the automated system actually required correction.

The example also shows why human review remains important. An automated system may identify a pattern, but it cannot always determine whether an unusual case is genuinely wrong or simply an exception that the project guidelines allow.

Human review can also expose recurring problems. If reviewers repeatedly find the same type of annotation error, the issue may point to unclear instructions or missing examples in the annotation guidelines.

Measuring Data Quality

Finding errors is only one part of QA. For large datasets, teams also need a way to understand how frequently those errors occur and whether the annotation process is being applied consistently.

One simple measure is the error rate, which represents the proportion of annotations that require correction.

For example, if a batch contains 2,000 annotations and 50 are found to contain errors:

This gives a simple view of how much incorrect work was identified in the batch. However, error rate alone does not explain whether the annotation guidelines are being interpreted consistently.

That is where other measures can be useful.

Turning QA Results Into Measurable Signals

Inter-annotator agreement looks at how often independent annotators reach the same decision when working on the same data. When reviewers consistently reach different conclusions, it can indicate that a guideline is unclear or that certain cases need better examples.

Precision provides another perspective by showing how many of the cases flagged by an automated check turn out to be genuine errors after review.

These measurements answer different questions. Error rate helps show how much incorrect work was found, precision helps show how useful an automated flagging process is, and inter-annotator agreement helps show how consistently the annotation standard is being applied.

For projects handled through Perit’s data annotation workflow, these measurements can be considered alongside sampling and human review to understand how a batch is performing.

The important point is that no single metric can describe the complete quality of a dataset. Looking at several signals together provides a more useful picture of where the process is working and where it needs attention.

A Hybrid QA Workflow

The real value of combining automation with human review comes from how the two stages work together.

A typical workflow can start with annotated data passing through automated validation. Records that meet the defined checks can continue through the process, while potential issues are flagged for closer inspection. Human reviewers can then determine whether the flagged cases are genuine errors, valid exceptions, or cases where the guideline needs clarification.

The workflow can be represented as:

At Perit, this type of quality-focused process can be supported through structured annotation, calibration, sampling, and review. Its Foundry workspace provides the environment for annotation and related review activities.

The workflow does not have to end when an annotation is corrected. Findings from human review can also be used to improve the next stage of the process. A repeated error may require a clearer guideline, while a validation rule that generates too many unnecessary flags may need refinement.

This turns QA from a final inspection into an ongoing part of the data production process.

Learning From QA Results

A useful QA system should not only identify what went wrong. It should also help prevent the same problem from appearing repeatedly.

Suppose reviewers keep correcting the same type of label. The issue may not necessarily be with the individual annotators. The guideline might need a clearer definition, an additional example, or a better explanation of an edge case.

The same applies to automated checks. If a rule repeatedly flags valid annotations, the rule may need to be adjusted. If a particular type of error is repeatedly missed, a new validation check may need to be introduced.

This creates a feedback loop:

For Perit, calibration and review are important parts of maintaining consistent annotation standards. QA findings can provide additional information about where those standards or processes may need refinement.

The goal is not simply to correct more errors in the current batch. It is to reduce the chance of the same errors appearing in future batches, making the overall process more consistent over time.

Perit’s Approach to Reliable AI Data

For complex AI projects, quality assurance is most useful when it is built into the data workflow rather than treated as a final inspection.

Automated validation can handle repeatable checks across large datasets, while human reviewers can focus on cases where context and judgment matter. Measurements such as error rate, precision, and inter-annotator agreement can then provide a way to understand how the process is performing.

Perit brings these elements together through its data annotation workflow, supported by calibration, sampling, review, and quality measurement. Its Foundry workspace supports the annotation process and related review activities.

The approach can be summed up simply:

For teams working with large and complex AI datasets, combining automated validation with human expertise provides a practical way to make quality more consistent, measurable, and scalable.