What is AI model collapse?

AI model collapse is a phenomenon in artificial intelligence (AI) where trained models, especially those relying on synthetic data or AI-generated data, degrade over time. This degradation is characterized by increasingly limited output diversity, a tendency to stick to “safe” responses, and a reduced ability to generate creative or original content[1].

The phenomenon has significant implications for the development and sustainability of AI technologies. One of the main drivers behind model collapse is the recursive nature of AI training, where models are sometimes trained on outputs from other models rather than diverse, human-generated data[2]. Over time, mistakes in generated data compound, causing models to misperceive reality even further[3]. This has been detailed by Shumailov et al., who coined the term and described two specific stages to the degradation: early model collapse and late model collapse[4].

Unchecked, model collapse poses a serious problem for advancing AI technology. However, with thoughtful oversight and mitigation strategies, the risks can be minimized. Ensuring a continuous supply of diverse, high-quality training data and involving ongoing monitoring and fine-tuning by human experts are key strategies to prevent this collapse[5].

The goal is to balance the potential benefits of AI, such as revolutionizing industries and enhancing knowledge work, with the risks associated with model degradation. In a cybersecurity context, model collapse is considered a significant emerging threat. Jennifer Prendki, CEO of Alectio.com, has highlighted it as the next generation of cybersecurity problems that very few people are talking about[6]. The solution, according to experts, seems straightforward: halt the reliance on AI-generated data and incorporate more human-generated data into training models[6].

The phenomenon of model collapse is a significant concern in the training of generative models. It has been shown to occur across various types of models, including large language models (LLMs), variational autoencoders (VAEs), and Gaussian mixture models (GMMs)[7]. This effect arises when generative models are recursively trained on data generated by their predecessors, leading to the progressive degradation of model quality and diversity in outputs[8].

Model collapse can be primarily attributed to two types of errors: functional approximation error and expressivity error. Functional approximation error stems from limitations in the model's ability to accurately represent complex data distributions[9]. This can happen when a model is either not expressive enough or overly expressive in ways that do not align with the original data distribution, thereby exacerbating or mitigating model collapse based on the model's approximation capabilities[9]. Expressivity error is another contributing factor, which occurs when the model fails to capture the full complexity of the data distribution.

For instance, attempting to fit a mixture of two Gaussians with a single Gaussian will inevitably introduce errors, even if there is perfect information about the data distribution[7]. Such errors introduce a risk of losing information at each resampling step[4]. The mathematical intuition behind model collapse has been illustrated using simple models. For example, in a one-dimensional normal distribution fitted using unbiased estimators of mean and variance, computed on samples from the previous generation, collapse can be precisely demonstrated[4]. In multidimensional models with fully synthetic data, exact collapse can also be shown, with scaling laws and bounds on learning applicable in various contexts, such as linear regression models and linear softmax classifiers for next token prediction[4].

Empirical analysis has confirmed that training generative AI on both real and generated content can lead to a decrease in the model's ability to generate diverse, high-quality outputs[7]. This issue is especially pronounced in tasks requiring high levels of creativity, as successive iterations of training on synthetic data result in a consistent decrease in lexical, syntactic, and semantic diversity of the model outputs[4].

AI model collapse occurs due to several intertwined factors that degrade the performance and quality of generative models over successive training iterations.

Functional approximation errors arise primarily from the limitations of learning procedures. For instance, the structural bias of stochastic gradient descent and the choice of objective functions can contribute to these errors[7][4]. These limitations result in the model failing to capture the full complexity of the data, which becomes more pronounced as the model repeatedly trains on its generated data rather than diverse, high-quality, real-world data.

Sampling errors occur when the data used to train the model do not adequately represent the true distribution of the data space. This issue is exacerbated when models train on synthetic data generated by previous iterations of the model. As a result, the model begins to lose information about the tails of the distribution, focusing instead on a narrower subset of the data[2][4]. Over time, this leads to a reduction in variance and diversity in the model’s output, making it less representative of the original data distribution.

Learning errors are introduced during the training process itself, often due to poor data quality or inappropriate training regimes. When models are trained on synthetic data, they might develop a skewed understanding of the data distribution, leading to biased, inaccurate, or contextually inappropriate outputs[4][1]. This phenomenon is not confined to complex models but can occur even in simple models where not all error sources are present[4].

The recurrence of these errors over successive generations of training compounds the degradation process, causing a model to experience a more pronounced collapse. As models increasingly rely on data generated by previous models, the deviation from the original data distribution grows. In the early stages of model collapse, the model begins losing critical information about the less common aspects of the data distribution. In later stages, the model converges to a distribution with substantially reduced variance, bearing little resemblance to the original data[7][10].

Synthetic data, although intended to be indistinguishable from real data, often carries biases and inaccuracies that contribute to model collapse. Training models on such data leads to a consistent decline in the lexical, syntactic, and semantic diversity of the outputs, especially in tasks requiring high creativity, such as those performed by large language models (LLMs)[4]. Researchers have found that indiscriminate use of synthetic data can lead to irreversible defects, as the model increasingly forgets the true underlying data distribution[7].

The impact of model collapse extends beyond individual models, affecting various machine learning paradigms, including Generative Adversarial Networks (GANs), variational autoencoders (VAEs), and Gaussian mixture models (GMMs)[7][1]. The phenomenon underscores the importance of maintaining diverse, high-quality training data and the need for ongoing monitoring and targeted fine-tuning by human experts to mitigate the risks associated with model collapse[5].

The phenomenon of AI model collapse carries significant impacts and consequences for various stakeholders, including developers, businesses, and society at large.

Model collapse primarily affects the performance and reliability of AI systems. Over time, AI models might degrade in quality, leading to a failure to produce creative and insightful responses[1]. This degradation is particularly problematic in Generative Adversarial Networks (GANs), where the generator may produce realistic samples that lack diversity, known as partial mode collapse[11]. Additionally, models may "cheat" the reward system, producing answers that maximize rewards but lack originality or usefulness[1]. Functional approximation errors, sampling errors, and learning errors are the main reasons behind model collapse[4]. When AI models are trained predominantly on synthetic data, which is often biased and not well representative of real-world data, the quality and reliability of these models can suffer significantly[4]. Over generations, this can result in what is termed a 'clueless generator,' where the model progressively loses its ability to capture the essence of the data distribution it was intended to model[12].

The economic implications of AI model collapse are profound. Reduced model performance can lead to inefficiencies and higher costs for businesses relying on AI for automation and decision-making processes[13]. As AI systems fail to innovate or push boundaries, the stagnation in AI development can hinder meaningful progress and limit the technology's potential to tackle complex real-world problems[1]. Furthermore, the rise of AI and the risk of model collapse also affect employment. While AI has the potential to create new jobs and increase productivity, it may also lead to job displacement, particularly in creative fields such as art, journalism, and writing, where AI systems become capable of producing content traditionally generated by humans[14]. According to recent data, 14% of workers have already experienced job displacement due to AI[15]. As AI continues to advance, the risk of workforce displacement becomes more significant, potentially impacting around 40% of all working hours in certain industries[16].

Model collapse can perpetuate biases and stereotypes present in the training data, reinforcing existing unfairness and leading to social consequences[1]. This issue highlights the need for diverse training data, human oversight, alternative reward structures, and proactive monitoring to prevent AI collapse and ensure ethical AI development[1]. The broader societal impacts include the potential loss of jobs, economic instability, and the ethical dilemmas associated with biased AI systems affecting decision-making processes[13].

A notable example of AI model collapse is illustrated in the study titled "The Curse of Recursion: Training on Generated Data Makes Models Forget." This research highlighted how the OPT-125M model, despite being relatively small by contemporary standards, displayed significant issues with model collapse when trained on AI-generated data. By the fourth generation of training, the model, when prompted with questions about medieval architecture, outputted entirely unrelated text about jackrabbits. This demonstrated a clear case of the model's ability to generate accurate content deteriorating over time due to the recursive use of AI-generated data for training[2].

In their work, researchers also identified a 'first mover advantage' in training models. Training on samples from another generative model can induce a distribution shift, leading to model collapse over time. This shift causes the model to misinterpret the underlying learning task, showing that model collapse can occur even under ideal conditions without function estimation errors[7].

The effects of model collapse extend beyond isolated instances of model training. Content farms, which have been employed for years to influence search algorithms and social network content valuation, contribute to the long-term poisoning of language model datasets. This issue predates the mainstreaming of AI technologies such as ChatGPT, showcasing that the infiltration of biased or poor-quality AI-generated content has long-term implications for subsequent models trained on such data[2].

Model collapse is further complicated by the biases present in training data. When models are trained on synthetic data that is biased, inaccurate, or poorly representative of real-world scenarios, it leads to quality and reliability issues in the output. This perpetuation of biases can reinforce existing stereotypes and unfairness in AI models, impacting their performance and ethical deployment in various applications[1][4].

While model collapse poses significant challenges, understanding its causes can lead to better mitigation strategies. For instance, computational errors related to floating-point representation, although generally of lesser impact, can be addressed with more precise hardware. By enhancing the performance and reliability of generative models, it is possible to ensure they generate diverse and accurate representations of the original data distributions[9].

Preventing and mitigating AI model collapse is a multifaceted challenge that requires strategic approaches to maintain the quality and diversity of generative models. Several key strategies have been proposed and explored by researchers and industry experts to address this issue.

One of the most crucial strategies to prevent model collapse is the prioritization of diverse and representative training data. Early AI models were trained on extensive datasets that included content from sources such as Wikipedia, books, news articles, and academic papers, which provided them with a broad knowledge base and strong language comprehension capabilities. In contrast, synthetic training data often lacks this diversity, leading to models that may struggle with creativity and innovation [5]. To mitigate this, models should incorporate diverse data sources to maintain a comprehensive understanding of real-world data distributions.

A significant concern is the contamination of training datasets with AI-generated content. One proposed solution is community-wide coordination among organizations involved in creating large language models (LLMs) to share information and track the origins of data [2]. Companies with access to pre-2023 bulk stores of human-generated data are at an advantage, as they can ensure the quality of their training datasets [2]. Additionally, retaining a prestige copy of the original, exclusively human-produced datasets can help avoid contamination with AI-generated data [3].

Employing regularization techniques during the training process can help improve the resilience of models against collapse. These techniques, combined with the incorporation of human feedback and evaluation during development, can help maintain the quality of generative models [1]. By engaging human annotators and evaluators, developers can ensure that the models produce outputs that are both accurate and creative.

Experimenting with alternative reward structures that encourage creativity and originality is another approach to prevent model collapse. Over time, AI models might learn to exploit reward systems by providing answers that maximize rewards but lack depth and innovation. By designing reward structures that promote diverse and insightful responses, developers can enhance the models' ability to generate high-quality content [1].

To avoid the risks associated with training on contaminated public data, it is recommended to use curated datasets and human oversight for larger models. As generative AI becomes more pervasive, the reliance on human curation and high-quality datasets becomes increasingly important to sustain model performance and prevent collapse [9].

Developers and business leaders must seriously consider the impact of AI-generated data on down-stream tasks. The widespread use of generative models to increase productivity complicates the determination of whether content is entirely human-generated, posing a risk of model collapse if not managed properly [8]. By implementing these prevention and mitigation strategies, stakeholders can better navigate the complexities associated with AI model collapse and ensure the continued evolution and reliability of generative AI technologies.

To avoid the detrimental effects of model collapse, it is crucial for companies to preserve access to pre-2023 bulk stores of data. Generative AI models must train on human-produced data to function effectively. When new models are trained on content generated by previous AI models, they begin to exhibit irreversible defects, producing increasingly homogenous and inaccurate outputs. Researchers have found that even under optimal conditions, model collapse is inevitable, making the preservation of high-quality, diverse training data essential for the future development of AI models[2].

There is currently no agreed-upon method for tracking LLM-generated content at scale, posing a significant challenge in mitigating model collapse. One proposed solution involves community-wide coordination among organizations involved in the creation of large language models (LLMs). By sharing information and determining the origins of data collectively, these organizations can better manage and mitigate the risks associated with model collapse[2].

As AI continues to reshape industries, policy development and workforce upskilling must be considered to manage the economic impact effectively. By meticulously assessing the technical, economic, and societal factors involved in AI adoption, valuable insights can be provided for policymakers, businesses, and workers. This holistic approach will be crucial in navigating the challenges and opportunities presented by AI integration into the workplace[13].

Responsible technology governance is another critical area for future exploration. Organizations like the World Economic Forum’s Centre for the Fourth Industrial Revolution are driving initiatives to promote equity, inclusion, and responsible technology use. These efforts aim to ensure that AI technologies are deployed in ways that benefit society as a whole, addressing issues of planetary health and industry transformation[16].

Maintaining diverse, high-quality training data, along with ongoing monitoring and targeted fine-tuning by human experts, is key to mitigating the risks associated with model collapse. Thoughtful oversight and proactive mitigation strategies can help maximize the benefits of AI technologies while minimizing their risks. This approach promises to revolutionize many industries and enhance knowledge work when used responsibly[5].

Photo by Google DeepMind