The Robot Didn’t Fail. The Dataset Did.

When More Data Still Isn’t Enough

A warehouse robot can sort the same type of package thousands of times and still struggle when the shelf is rearranged, the lighting changes, or another package partially blocks its view. From the outside, it looks like the robot simply made a bad decision. But sometimes the real problem happened much earlier, during data collection. If most of its training demonstrations showed the same layout, the same objects, and the same predictable conditions, the robot may have learned to perform the task only when the world behaves exactly as expected.

This is where the idea that more data automatically means better AI starts to fall apart. A dataset can contain thousands of demonstrations, millions of frames, and hundreds of hours of recordings while still representing only a small slice of the situations a robot will encounter in practice. If the same environments, object configurations, camera viewpoints, lighting conditions, and successful actions keep appearing, the dataset becomes larger without becoming meaningfully broader.[1]

For physical AI, the real bottleneck is increasingly not how much data a robot has seen, but how much of the world that data actually represents. Robots operating outside controlled environments have to deal with clutter, changing conditions, unfamiliar objects, unexpected movements, and people performing the same task in different ways. When those variations are missing from the training data, the answer may not be a larger model or another round of fine-tuning. Sometimes, the robot simply needs a dataset that exposes it to a wider and more realistic version of the problem.

The Hidden Cost of Repetition

Collecting robotics data takes time, money, equipment, and people, so it makes sense for teams to stick with a setup that is already working. If a particular environment is producing good demonstrations, it is easy to keep recording there and watch the dataset grow. The numbers look encouraging: more videos, more demonstrations, more hours. But if most of those examples are being collected under almost the same conditions, the dataset may be growing faster than the knowledge it adds.

Repetition itself is not a bad thing. Seeing the same action performed several times can help a model understand what a successful behavior looks like and make it less sensitive to small amounts of noise. The problem starts when repetition leaves very little room for variation. If a robot keeps seeing the same task in the same type of environment, with similar objects and similar conditions, it can become very good at that particular setup without really learning how the task changes when something unexpected happens.

That is what makes dataset growth slightly deceptive. The number of demonstrations can keep climbing while the range of situations covered by those demonstrations barely changes. Instead of asking only how many new recordings were added this week, robotics teams need to ask a more useful question: What did the robot get to see this time that it had not seen before? That shift, from counting data to understanding what the data actually contains, is where better dataset design begins.[1]

What Does Good Coverage Actually Look Like?

Having lots of different videos does not automatically mean a robotics dataset has good coverage. Two recordings might look different but still teach the robot almost exactly the same thing. What really matters is whether the data captures the kinds of changes that can affect what the robot needs to do. A new camera angle, for instance, might be useful, while changing only the background may add very little.

There are many ways a real-world task can change. Objects may have different shapes, sizes, materials, or positions. The surrounding environment can look completely different even when the task itself stays the same. Lighting can shift, people can approach the task differently, and the robot may have to start from a different position or deal with an object that is partly blocked. These may seem like small differences to a person, but together they can make a familiar task much harder for a robot.[2]

That is why good coverage should be intentional, not just random. The goal is not to collect every strange situation imaginable. It is to understand where the robot will actually be used, identify the changes it is likely to encounter, and make sure those changes are reflected in the training data. A useful dataset is not simply varied for the sake of looking diverse; its variation should have a reason behind it.[3]

The Data Nobody Thinks to Collect

When teams collect robotics data, they usually start with the task going right. The robot needs to pick something up, move it, sort it, or complete a sequence of actions, so the easiest thing to record is a clean demonstration where everything works as expected. That makes sense as a starting point, but it can also leave a big gap in the dataset: what happens when the task does not go exactly according to plan?

Real environments rarely cooperate that neatly. An object may be partly hidden, a person may suddenly move into the scene, the robot may approach from an awkward angle, or an action that normally works may fail on the first attempt. Even small changes in the surroundings can force the robot to react differently. If its training data contains mostly perfect demonstrations, it has little experience to draw on when one of these situations appears outside the training environment.

That is why some of the most valuable data can be the data that shows where a task becomes difficult. This does not mean deliberately making every demonstration chaotic or filling a dataset with random failures. It means understanding which situations are likely to cause problems and making sure the robot gets enough exposure to them. A difficult example can sometimes teach the model more about real-world behavior than another perfectly executed demonstration under conditions it has already seen many times.[2]

From Model Failures Back to Data Collection

A robotics dataset should not be treated like a finished product that is collected once and then forgotten. Once a robot starts being tested, its mistakes can tell the team a lot about what the dataset is still missing. If it keeps struggling when objects are partly hidden, when the layout changes, or when a task starts from an unusual position, those failures are useful clues. They can show where the training data is too narrow or where certain situations simply have not been represented enough.[4]

That also makes the next round of data collection much more focused. Instead of deciding to record another 10,000 demonstrations and hoping they improve the model, a team can first look at what the robot is actually getting wrong. Maybe there are not enough examples of different object placements, perhaps most recordings come from one type of environment, or the model has rarely seen what happens when an action does not work on the first attempt. Once those gaps are clear, new demonstrations can be collected around them rather than adding more examples of situations the robot already handles well.

Over time, this turns data collection into a continuous feedback loop. The model shows where it struggles, those failures reveal what the dataset is missing, and the next collection cycle is designed around those gaps. The point is not to make the dataset endlessly larger. It is to make each new batch of data add something the robot genuinely needs to learn.[4]

Why Dataset Composition Can Matter More Than Dataset Size

When people talk about robotics datasets, the easiest thing to compare is size. A dataset with hundreds of thousands of demonstrations naturally sounds more useful than one with a much smaller number. But the number alone does not tell us much about what the robot has actually learned. If most of those demonstrations come from the same few environments, involve similar objects, and follow nearly identical task patterns, the dataset can become much larger without becoming much broader.[1]

Consider two teams collecting data for the same robotic task. One records a very large number of demonstrations in a controlled environment, repeating the task under conditions that change very little. The other collects fewer demonstrations but spreads them across different environments, object arrangements, viewpoints, operators, and situations where the task does not always go perfectly. The second dataset may be smaller, but it gives the model a wider picture of what the task can actually look like. The point is not that smaller datasets are better. It is that what is inside the dataset matters just as much as how much of it there is.[3] [5]

This difference becomes much easier to see once the robot leaves the environment where it was trained. A task that looked straightforward during collection can become harder when the surroundings change, an object appears in an unfamiliar position, or another person interacts with the scene differently. If those kinds of changes were barely present in the training data, adding more demonstrations of the original setup may not solve the problem. The robot may simply need examples that cover the situation it is struggling with.

That is why dataset size should not be treated as the only measure of progress. Teams need to look at what each new batch of data is actually adding: a new environment, a new type of object, a different way of performing the task, or a situation the robot previously handled poorly. A dataset becomes more valuable when new data expands the range of situations the model can understand, rather than simply giving it more versions of something it has already seen.

Building Better Data for the Real World

Once teams start looking at coverage instead of just dataset size, the way they collect robotics data has to change. The first question is no longer simply how many demonstrations can be recorded. It becomes a question of what those demonstrations should contain. Before collection starts, teams need to think about where the robot will operate, what can change in those environments, which situations are likely to cause problems, and what kinds of examples are still missing from the dataset.

That is not always easy to manage in practice. Real-world data collection involves people, different environments, different objects, changing conditions, and a lot of small details that are easy to overlook. Someone has to make sure the instructions are clear, the recordings are usable, and the data actually reflects the situations the robot is expected to encounter. Just as importantly, the process has to leave room for learning from the robot’s mistakes. If a model repeatedly struggles with a particular situation, that information should influence what gets collected next.[4]

This is one area where a specialized data partner such as Perit can become useful. The difficult part of training an AI system is often not finding a way to collect more examples. It is building a process that consistently produces the right examples, checks their quality, and adapts as the requirements change. For physical AI in particular, that means paying attention to the environments and conditions represented in the data, rather than treating every additional recording as equally valuable.

A simple way to think about this is that three things need to work together:

This is not meant to be a formal scientific equation. It is a useful way of looking at the problem. A huge dataset does not help much if most of it repeats the same conditions. A highly varied dataset is also difficult to rely on if the recordings are inconsistent or poorly reviewed. The real value comes from having enough data to learn from, enough variation to prepare the model for different situations, and enough quality control to make sure those examples are actually useful.

The same idea extends to the infrastructure around data collection and management. Foundry is part of this broader approach, providing a structured environment for managing AI data workflows and annotation. As models become more capable, the data pipeline has to become more deliberate too: collect, review, evaluate, identify gaps, and then collect again with those gaps in mind.

The next generation of robotics datasets will not be defined simply by how many demonstrations they contain. What will matter more is whether those demonstrations capture the situations a robot is actually likely to face: different environments, different objects, different people, changing conditions, and the moments when a task does not go as planned. The goal is not to collect every possible scenario. It is to make sure the important ones are not missing.

When a robot fails in the real world, the first instinct is often to look at the model. Maybe the architecture needs to change, the model needs more parameters, or another round of fine-tuning is required. Sometimes that is true. But sometimes the more useful question is much simpler: what did the robot never get to see? The answer may be sitting in the dataset long before the failure appears in deployment.

The Dataset Is Part of the Model

For robotics teams, the practical takeaway is simple: do not treat dataset size as the finish line. Before starting another large collection cycle, look at where the robot is actually failing. Group those failures by environment, object, viewpoint, task condition, human interaction, and outcome. The patterns will often reveal gaps that are much more useful than another round of random data collection.

From there, the next collection cycle can be much more targeted. Add examples that cover the missing conditions, include difficult and unsuccessful attempts where they reflect real deployment, and review whether the new data actually improves performance on those cases. This creates a much more useful loop: evaluate the model, find the gaps, collect for those gaps, retrain, and evaluate again.

Teams should also keep track of what their dataset represents, not just how large it has become. A simple coverage review can show which environments or situations are heavily represented and which ones barely appear. That makes it easier to decide where the next batch of data should come from and prevents the collection process from quietly producing thousands of variations of the same example.

In the end, better robotics data does not mean collecting everything. It means being deliberate about what is collected, why it is collected, and what the model still needs to learn. When a robot fails, the next question should not always be “How do we make the model bigger?” It should be “What data would have helped it handle this situation?” That question can turn a model failure into the starting point for a better dataset—and, ultimately, a more reliable robot.

References

[1] Data Scaling Laws in Imitation Learning for Robotic Manipulation
— ICLR, 2025

[2] Efficient Data Collection for Robotic Manipulation via Compositional Generalization
— RSS, 2024.

[3] Is Diversity All You Need for Scalable Robotic Manipulation?
— 2025.

[4] What Matters in Learning from Large-Scale Datasets for Robot Manipulation
— 2025.

[5] Diversity You Can Actually Measure: A Fast, Model-Free Diversity Metric for Robotics Datasets
— 2026.