When AI Gets It Right Most of the Time: The Problem With Rare Cases

The Data Distribution Is Not the Real World

A dataset does not become representative just because it is large. A model can be trained on millions of examples and still have very little experience with the situations it will eventually face in the real world.

Consider an autonomous driving system trained on millions of hours of road footage. Most of that data might show ordinary daytime driving: clear roads, visible lane markings, familiar intersections, and predictable traffic. But only a small portion may contain situations such as a pedestrian suddenly stepping out from behind a parked vehicle, heavy rain obscuring lane markings, or a cyclist appearing from an unexpected direction. These situations may be rare in the dataset, but they can matter enormously when the system encounters them on the road.

This creates a gap between what the model sees most often and what it might actually encounter. AI systems learn from the patterns present in their training data, so situations that appear rarely—or not at all—give the model much less experience to learn from. NIST research on data coverage has also examined how the frequency of different input combinations can affect AI performance. [1]

The gap can become even wider after deployment. The people, environments, behaviors, and conditions a model encounters in the real world are not always the same as those represented in its training data. A model might perform well on a carefully prepared test set and still struggle when it encounters situations that were barely represented during training. [2]

A large dataset gives an AI system more examples to learn from. It does not necessarily give the model the right examples.

Why Rare Cases Are So Difficult to Learn

Rare cases are difficult for AI systems for a simple reason: the model just does not get to see them often enough.

If a model sees thousands of examples of the same situation, it has plenty of opportunities to learn what matters. But unusual situations might appear only a few times in the entire training set. And when they do appear, they may not look exactly alike. One example might involve poor lighting, another an unusual camera angle, and another a combination of several conditions the model has never encountered together.

Take an autonomous driving system. It can see millions of examples of normal roads, pedestrians, traffic signals, and vehicles. But what happens when heavy rain, poor visibility, an unexpected obstacle, and an unusual road layout occur at the same time? Each of these conditions may be familiar on its own, but their combination may be almost absent from the training data.

This is what makes the long tail difficult. [3] A relatively small number of common situations account for most of the data, while a much larger number of unusual situations appear only occasionally. The model gets plenty of practice with the first group and very little with the second.

There is another complication: rare cases are often not simply new categories. They can be new combinations of things the model already knows. A system might correctly recognize a pedestrian, a vehicle, and a road obstruction individually, yet still struggle when all three appear together in an unfamiliar situation.

That is why rare does not necessarily mean unimportant. A case can appear only a handful of times in a dataset and still become one of the most important cases to handle correctly once the AI is deployed in the real world.

When Accuracy Hides the Problem

One of the easiest ways to miss rare-case failures is to look at the overall accuracy of a model.

Imagine a system tested on 100,000 examples. If 98,000 of those examples represent common situations and the model performs extremely well on them, it can still achieve an impressive overall score even if it performs poorly on the remaining 2,000 rare cases. The final percentage looks good, but it does not tell us where the model is struggling.

Finding the Cases the Model Has Missed

Once a model’s weak spots become visible, the next challenge is figuring out what is missing from the data. Simply adding more examples is not always the answer. If the new data looks almost identical to what the model already knows, the dataset becomes larger without becoming much more useful.

A better approach is to look at where the model repeatedly struggles. For an autonomous driving system, that could mean collecting more examples from heavy rain, poorly lit roads, unusual intersections, or situations where pedestrians and vehicles behave unpredictably. For a voice system, it could mean looking at uncommon accents, background noise, overlapping speech, or phrases that the model regularly misinterprets.

This turns data collection into a more targeted process.  [5] Instead of asking “How can we get more data?”, teams can ask “What kind of data would help the model handle this failure better?”

Production data can be particularly valuable here. Every unexpected prediction, correction, or failure can reveal a situation that was poorly represented during training. Those examples can then be reviewed, labelled, and added back into the training or evaluation process where appropriate.

The goal is not to collect every possible situation. That would be impossible. The goal is to continuously identify the gaps that matter and make sure the dataset becomes better at representing them over time.

More Data Is Not Always Better Data

Once a gap has been identified, the obvious response is often to collect more data. But adding data indiscriminately can make a dataset larger without making it more useful.

What matters is what the new data adds. If an autonomous driving system already has millions of examples of clear-weather highway driving, collecting another million similar examples will do little to prepare it for heavy rain, poorly marked roads, or pedestrians appearing from unexpected locations. A smaller, carefully selected set of those missing scenarios could provide much more useful coverage. [6]

This is why data collection increasingly needs to be guided by the behaviour of the model itself. Teams can use failed predictions, production feedback, and evaluation results to identify which situations deserve more attention. The aim is not simply to increase the number of examples, but to improve the range of situations the model has learned to handle.

The same principle applies to annotation. Rare and difficult examples often require more careful review because a small labelling mistake can make an already limited signal even harder for the model to learn from. Human experts can help determine which cases are genuinely important, how they should be labelled, and whether similar examples should be added to the dataset.

In other words, the value of data depends not only on how much you have, but on what it teaches the model that it did not already know.

What Reliable AI Looks Like Beyond the Average

AI systems will always have cases they have never seen before. The goal is not to build a dataset that contains every possible situation, because no dataset can do that. The goal is to understand where the gaps are and keep reducing the ones that matter.

That means looking beyond overall accuracy and paying attention to the situations where models struggle. A rare example may seem insignificant when viewed on its own, but a collection of these examples can reveal a pattern that a conventional benchmark would miss.

For teams building AI for real-world environments, this makes data quality a continuous process. Models need to be evaluated against changing conditions, their failures need to be investigated, and the resulting insights need to feed back into data collection and training.

References