{"id":936,"date":"2026-10-01T12:03:24","date_gmt":"2026-10-01T12:03:24","guid":{"rendered":"https:\/\/perit.ai\/blogs\/?p=936"},"modified":"2026-10-01T12:03:24","modified_gmt":"2026-10-01T12:03:24","slug":"ai-is-running-out-of-data-what-happens-when-the-internet-stops-scaling","status":"publish","type":"post","link":"https:\/\/perit.ai\/blogs\/ai-is-running-out-of-data-what-happens-when-the-internet-stops-scaling\/","title":{"rendered":"AI Is Running Out of Data: What Happens When the Internet Stops Scaling?"},"content":{"rendered":"

The next AI bottleneck may not be compute<\/h2>\n

For the last few years, the recipe for building better AI has looked fairly straightforward: make the models bigger, give them more computing power, and train them on more data.<\/p>\n

That approach has worked remarkably well. But there is a problem hiding in the third part of that recipe.<\/p>\n

How much more data is actually available?<\/strong><\/p>\n

It is easy to think of the internet as an endless source of training material. There are billions of webpages, books, articles, images, videos, conversations, and lines of code online. But AI models need useful training data, not just a larger pile of files.<\/strong> As models become more capable, the amount of high-quality data needed to train them grows too.<\/p>\n

Research from Google DeepMind made this relationship clearer. Their compute-optimal training work showed that simply making a model larger was not enough: a smaller model trained on significantly more data could outperform a much larger one under a similar compute budget. [1]<\/a><\/p>\n

Compute only goes so far if you do not have enough good data to use it effectively.<\/strong><\/p>\n

\n
\n
\n
Larger Models<\/div>\n
More capacity<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
More Compute<\/div>\n
More training power<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
More Data<\/div>\n
More training examples<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Data Pressure<\/div>\n
Useful human data is finite<\/div>\n<\/div>\n<\/div>\n
More model scale \u2192 more demand for useful training data<\/div>\n<\/div>\n

And that creates a different kind of scaling problem.<\/p>\n

GPUs can be manufactured. Data centers can be expanded. Training runs can be made larger.<\/p>\n

But genuinely new human knowledge does not appear on demand.<\/strong><\/p>\n

The public internet may keep getting bigger, but much of that growth comes from duplicated, outdated, low-quality, restricted, or AI-generated content. More content does not necessarily mean more useful human-generated information<\/strong>.<\/p>\n

AI is not literally running out of data today. The problem is that useful human-generated data is finite<\/strong>, and finding enough of it could become a much bigger constraint as models continue to scale.<\/p>\n

\n

Scaling has a data bill<\/h2>\n

Making an AI model bigger sounds like a problem for GPUs. But scaling comes with another bill: data<\/strong>.<\/p>\n

As models become larger, they need more training data to make use of that extra capacity. The problem is that high-quality human-generated data does not grow as quickly as the appetite of these models. The internet keeps getting bigger, but that does not mean it is producing an endless supply of useful training material.<\/p>\n

Researchers have started putting numbers around this problem. A 2024 study by Villalobos and colleagues estimated that, depending on the assumptions, demand for human-generated training data could begin approaching its available supply somewhere between 2026 and 2032<\/strong>. [2]<\/a><\/p>\n

\n
When training-data demand starts catching up<\/div>\n

<\/p>\n

\n
Available human-generated data<\/div>\n
<\/div>\n
Large, but ultimately finite supply<\/div>\n<\/div>\n

<\/p>\n

\n
Training-data demand<\/div>\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Growing as models require more training data<\/div>\n<\/div>\n

<\/p>\n

\n

<\/p>\n\n\n\n
2024<\/td>\n2026<\/td>\n2028<\/td>\n2030<\/td>\n2032<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/p>\n

<\/div>\n

<\/p>\n

Projected pressure window: 2026\u20132032<\/div>\n<\/div>\n

<\/p>\n

Conceptual view based on the study’s projections. The exact timing depends on the assumptions used.<\/div>\n<\/div>\n

That is not a prediction that the internet will suddenly run out of text. It is about whether the supply of usable human-generated data can keep pace with what increasingly large models need.<\/p>\n

But raw volume can be misleading. A huge collection of files does not necessarily mean a huge collection of useful information.
\nSo when we talk about the amount of data available to AI, the important number is not everything sitting on the internet.<\/p>\n

It is the amount of new, usable information that can actually improve a model.<\/strong><\/p>\n

As training datasets reach billions or even trillions of tokens, finding another huge collection of files is one thing. Finding genuinely useful new information is another.<\/p>\n

The problem is not that the internet is becoming small. It is that valuable new data may not be growing fast enough.<\/strong><\/p>\n

So if the internet keeps producing more content, why can’t AI companies simply keep collecting it?<\/p>\n<\/div>\n

\n

The internet isn’t an infinite dataset<\/h2>\n

At first, the solution seems obvious: if AI needs more data, just collect more of the internet.<\/p>\n

There is certainly a lot to collect. New articles, posts, videos, comments, code, and reviews appear online every day. But more content does not automatically mean more useful training data.<\/strong><\/p>\n

\n
More content does not mean more useful data.<\/div>\n
The internet can keep getting bigger without providing the same amount of genuinely new information. A large dataset may contain repeated pages, outdated material, low-quality content, restricted sources, or information that has already appeared in previous training datasets. What matters is not simply how much content exists, but how much of it adds something new and useful to the model.<\/div>\n<\/div>\n

Take duplication and quality. The same information can appear across hundreds of websites, while carefully researched material sits alongside spam, outdated pages, automatically generated text, and plain mistakes.[3]<\/a> Counting all of it makes the dataset look bigger, but it does not give the model the same amount of new information.Counting all of it makes the dataset look bigger, but it does not give the model the same amount of new information. [4]<\/a><\/p>\n

There is also the question of what the model has already seen. As datasets get larger, researchers have to check for contamination<\/strong>\u2014for example, when benchmark or evaluation material accidentally ends up in the training data. That can make a model appear better than it really is. [5]<\/a><\/p>\n

There is also the question of access. Not everything on the internet is available for unrestricted use.<\/strong> Copyright, licensing, privacy, and website terms can all affect whether material can actually be included in a training dataset.<\/p>\n

So the amount of content online is very different from the amount that can become useful training data.<\/p>\n

The data problem is therefore becoming less about how much content exists<\/strong> and more about how much of it is new, useful, diverse, and actually usable.<\/p>\n

And increasingly, some of that new content is being created by AI itself.<\/strong><\/p>\n

\n

Synthetic data has a ceiling<\/h2>\n

This is where the data problem gets a little more complicated.<\/p>\n

If good human-generated data becomes harder to find, the obvious idea is to create more of it ourselves. AI can already generate text, images, code, conversations, and all kinds of other examples at a speed people simply cannot match.
\nOn paper, that sounds almost like an unlimited supply of training data.<\/p>\n

And synthetic data can genuinely be useful. It can create situations that are difficult to collect in the real world, generate rare examples, and expand a dataset when gathering real data is expensive or impractical.<\/p>\n

But there is a catch.<\/p>\n

AI-generated data ultimately comes from what earlier models learned from.<\/strong><\/p>\n

Imagine a model trained mostly on human-created information. It generates millions of pieces of new content, and some of that content ends up being used to train the next model. That model then produces more content, which becomes training material for another model.<\/p>\n

\n
\n
\n
Human Data<\/div>\n
Original information<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
AI Model<\/div>\n
Learns from the data<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Synthetic Data<\/div>\n
Model-generated content<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Next Model<\/div>\n
Trained on the mixture<\/div>\n<\/div>\n<\/div>\n
Repeating the cycle can reduce the amount of fresh human information entering training.<\/div>\n
\n

Researchers have found that repeatedly training on model-generated data can lead to model collapse<\/strong>, where newer generations lose some of the variety and information present in the original human-generated data. [6]<\/a><\/p>\n

That does not make synthetic data useless. It can still be valuable for rare situations, controlled examples, edge cases, and tasks where collecting enough real-world data is difficult.<\/p>\n

The limitation is simple: generating more examples is not the same as generating more knowledge.<\/strong> A model can produce thousands of variations of something it already understands without adding thousands of genuinely new insights.<\/p>\n

Real-world data still brings something synthetic data struggles to reproduce: unpredictability. Human conversations, messy environments, sensor readings, and professional workflows contain details that are difficult to anticipate in advance.<\/p>\n

So the future is unlikely to be a choice between human and synthetic data. Real-world data can provide the foundation, while synthetic data can expand it and target specific gaps.<\/p>\n

That also changes where AI companies need to look next. The answer may no longer be another corner of the web, but people, specialised industries, software environments, sensors, machines, and the physical world.<\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
Synthetic data can expand knowledge, but it cannot endlessly create new knowledge.<\/div>\n
A model can generate thousands of new examples from patterns it has already learned. Those examples can be useful for practice, edge cases, and filling specific gaps. But generating more variations does not automatically introduce new information into
\nthe training process. Real-world data is still needed to keep the model grounded in information it could not simply generate from its existing knowledge.<\/div>\n<\/div>\n<\/div>\n
\n

The search for data beyond the web<\/h2>\n

If the public internet cannot keep supplying useful data forever, AI developers have to start looking elsewhere.<\/p>\n

One obvious source is specialised human-generated data<\/strong>. The web may have millions of pages about medicine, engineering, finance, or law, but that does not capture everything professionals actually do. Data collected from real workflows can contain details that are difficult to find through ordinary web scraping.<\/p>\n

The same applies to multimodal and real-world data. Audio, video, sensor readings, software interactions, and physical environments can capture information that text alone cannot.<\/p>\n

\n\n\n\n\n\n\n\n\n
Data source<\/th>\nWhat it adds<\/th>\nMain challenge<\/th>\n<\/tr>\n<\/thead>\n
Public web<\/td>\nHuge volume and broad coverage<\/td>\nDuplication, quality, licensing and contamination<\/td>\n<\/tr>\n
Human & domain data<\/td>\nExpert knowledge and real workflows<\/td>\nExpensive and harder to collect<\/td>\n<\/tr>\n
Synthetic data<\/td>\nControlled examples and rare edge cases<\/td>\nMay reproduce existing model patterns<\/td>\n<\/tr>\n
Real-world multimodal data<\/td>\nAudio, video, sensors and physical interactions<\/td>\nCollection, annotation and validation are complex<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The next data advantage may come from harder-to-collect sources rather than larger web-scale datasets. [7]<\/a><\/div>\n<\/div>\n

The physical world is another important source. Robotics, for example, needs data about how objects move, how surfaces react, how things go wrong, and how people interact with physical environments. Those details have to be learned from real interactions with the world<\/strong>, not just webpages.<\/p>\n

That means collecting data from real interactions with the world<\/strong>.<\/p>\n

The same pattern appears in scientific experiments, business operations, and human demonstrations. These sources can show a model how something actually happens<\/strong>, rather than simply describing it in text.<\/p>\n

The same pattern appears in scientific experiments, business operations, and human demonstrations. These sources can show a model how something actually happens<\/strong>, rather than simply describing it in text.<\/p>\n

That makes these datasets more expensive to build, but also harder to replace.<\/p>\n

The next generation of AI may depend less on how much data can be scraped from the internet and more on how much valuable data can be deliberately collected and validated.<\/strong><\/p>\n

And that shift is changing the data pipeline itself.<\/p>\n

\n

A new data pipeline is emerging<\/h2>\n

For a long time, AI training looked fairly simple: collect a lot of data, clean it up, and train the model.<\/strong><\/p>\n

That gets harder when the data comes from specialised industries, real-world interactions, or human demonstrations. You have to be more deliberate about what you collect, who provides it, and whether it is actually useful.<\/strong><\/p>\n

The process starts looking less like scraping and more like a proper pipeline:<\/p>\n

\n

<\/p>\n

\n
\n
Collect<\/div>\n
Gather raw data<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Filter<\/div>\n
Remove noise<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Curate<\/div>\n
Select useful examples<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Annotate<\/div>\n
Add context<\/div>\n<\/div>\n<\/div>\n

<\/p>\n

\u2193<\/div>\n

<\/p>\n

\n
\n
Validate<\/div>\n
Check quality<\/div>\n<\/div>\n
\u2190<\/div>\n
\n
Train<\/div>\n
Build the model<\/div>\n<\/div>\n
\u2190<\/div>\n
\n
Evaluate<\/div>\n
Find remaining gaps<\/div>\n<\/div>\n<\/div>\n

<\/p>\n

\n
<\/div>\n
Feedback from evaluation drives the next round of data collection<\/div>\n<\/div>\n<\/div>\n

The steps are not just cleanup. They decide what information actually reaches the model. And the process does not necessarily stop after training. If evaluation reveals a weakness, that failure can point to a gap in the data, sending the team back to collect and validate more examples. [8]<\/a><\/p>\n

That creates a feedback loop between data and model performance<\/strong>.<\/p>\n

This also makes high-quality data more valuable. A specialised dataset may contain far fewer examples than a massive web scrape, but those examples can be far more useful for a specific task.<\/p>\n

The data race is shifting from finding the biggest dataset to building datasets that are new, relevant, diverse, trustworthy, and grounded in the real world.<\/strong><\/p>\n

The bottleneck is shifting<\/h2>\n

Saying AI is \u201crunning out of data\u201d makes it sound like the internet is about to go empty. That is not what is happening. The problem is that the easiest sources of useful human-generated data are becoming harder to scale.<\/p>\n

Synthetic data can help, but it cannot fully replace information created through real human experience. The advantage may increasingly come from better, more diverse data that is difficult to collect or reproduce.<\/strong><\/p>\n

That could mean specialised industries, human interactions, scientific work, sensors, machines, and the physical world.<\/p>\n

The next data challenge may not be finding the largest dataset. It may be finding data that adds something genuinely new.<\/strong><\/p>\n

AI still has enormous amounts of information left to learn from. But getting to the valuable parts is becoming a more deliberate process. And that could make data collection, curation, and validation just as important to AI development as model architecture and computing power.<\/p>\n

In the end, the important question may not be \u201cWho has the most data?\u201d<\/strong><\/p>\n

It may be \u201cWho can find the data that everyone else cannot easily find?\u201d<\/strong><\/p>\n

\n
\n
\n
\n
\n
\n
\n
\n
humanise it<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
\n

What this means for the next generation of AI<\/h2>\n

The data problem does not mean AI has hit a wall. It means the way we get training data has to change.<\/strong><\/p>\n

For years, the internet seemed like an almost endless source of material. If a model needed more examples, there were always more webpages, articles, posts, and code to collect. But that becomes less useful as datasets get bigger and the easiest sources of human-generated content become repetitive, difficult to license, or simply less informative.<\/p>\n

The next phase is likely to involve a much wider mix of data.<\/p>\n

Human-created web content will still matter. Synthetic data will continue to be useful for filling specific gaps. But AI systems will also need more specialised, real-world data that has been deliberately collected for a purpose.<\/strong><\/p>\n

\n
\n
\n
Web Data<\/div>\n
Broad coverage
\nHuge volume<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Synthetic Data<\/div>\n
Fill specific gaps
\nCreate controlled examples<\/div>\n<\/div>\n
\u2192<\/div>\n
\n
Real-World Data<\/div>\n
Specialised knowledge
\nReal interactions<\/div>\n<\/div>\n<\/div>\n
The next generation of AI training will use a wider mix of data sources.<\/div>\n<\/div>\n

That changes the job of building a training dataset too. It is no longer just about collecting as much as possible. Someone has to figure out what the model actually needs, find the right examples, check whether they are reliable, and make sure the dataset reflects the real world.<\/p>\n

This becomes even more important as AI moves beyond text.<\/p>\n

A language model can learn from billions of words, but a robot needs information about movement and physical environments. A voice system has to deal with accents, interruptions, background noise, and the way people actually speak. A medical AI needs carefully selected clinical information, not simply another million webpages about medicine.<\/p>\n

Different AI systems need different kinds of reality.<\/strong><\/p>\n

That is why the future of AI training may be less about building one enormous dataset from the open internet and more about building many specialised sources of valuable data.<\/p>\n

The internet gave AI an enormous head start. But the next wave of progress may depend on information that cannot simply be scraped.<\/p>\n

And that could make how data is collected, curated, and validated<\/strong> just as important as the models and hardware used to train AI.<\/p>\n

\n

The data problem is becoming a collection problem<\/h2>\n

AI is not going to suddenly wake up one day with nothing left to learn from.<\/p>\n

The problem is more subtle. The internet is still full of content, but finding new, useful human-generated data<\/strong> is getting harder. Adding another million webpages to a dataset does not help much if most of them repeat information a model has already seen.<\/p>\n

That means the work is shifting.<\/p>\n

Instead of simply collecting more content from the web, AI developers may need to spend more time collecting data directly from people, experts, software environments, machines, and the physical world. They may need to record real interactions, capture unusual situations, and build datasets around specific problems.<\/p>\n

Synthetic data will have a role too. It can create extra examples and help cover situations where real data is difficult to collect. But it works best when it is grounded in real information rather than being used as an endless replacement for it.<\/p>\n

So the next stage of AI training may look less like \u201cfind more data\u201d<\/strong> and more like \u201cfind better sources of data.\u201d<\/strong><\/p>\n

The internet gave AI an enormous amount to learn from. What comes next may depend on data that takes more effort to collect, understand, and validate.<\/p>\n

And that could make the people and systems building those datasets just as important as the models being trained on them.<\/p>\n

\n

References<\/h2>\n
\n

[1]<\/strong> Training Compute-Optimal Large Language Models<\/em><\/a>
\nHoffmann, J., et al. (2022). arXiv:2203.15556.<\/p>\n

[2] <\/strong>Position: Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data<\/em><\/a>
\nVillalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2024). Proceedings of Machine Learning Research, 235, 49523\u2013\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 49544.<\/p>\n

[3] <\/strong>The Pile: An 800GB Dataset of Diverse Text for Language Modeling<\/em><\/a>
\nGao, L., et al. (2020). arXiv:2101.00027.<\/p>\n

[4] <\/strong>Deduplicating Training Data Makes Language Models Better<\/em><\/a>
\nLee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021). arXiv:2107.06499.<\/p>\n

[5] <\/strong>Benchmark Data Contamination of Large Language Models: A Survey<\/em><\/a>
\nXu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024). arXiv:2406.04244.<\/p>\n

[6] <\/strong>AI Models Collapse When Trained on Recursively Generated Data<\/em><\/a>
\nShumailov, I., Shumaylov, Z., Zhao, Y., et al. (2024). Nature<\/em>, 631, 755\u2013759.<\/p>\n

[7] <\/strong>The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text<\/em><\/a>
\nKandpal, N., Lester, B., Raffel, C., et al. (2025). arXiv:2506.05209.<\/p>\n

[8] <\/strong>Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research<\/em><\/a>
\nSoldaini, L., Kinney, R., Bhagia, A., et al. (2024). arXiv:2402.00159.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

The next AI bottleneck may not be compute For the last few years, the recipe for building better AI has looked fairly straightforward: make the models bigger, give them more\u2026<\/p>\n","protected":false},"author":1,"featured_media":940,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,3],"tags":[],"class_list":["post-936","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","category-tech-update"],"_links":{"self":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/936","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/comments?post=936"}],"version-history":[{"count":4,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/936\/revisions"}],"predecessor-version":[{"id":941,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/936\/revisions\/941"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/media\/940"}],"wp:attachment":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/media?parent=936"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/categories?post=936"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/tags?post=936"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}