Researchers have issued a cautionary statement regarding the potential depletion of training data in the industry, which serves as the lifeblood for powerful AI systems, especially as artificial intelligence (AI) reaches the pinnacle of its popularity.
This scarcity could potentially alter the trajectory of the AI revolution and decelerate the advancement of AI models, particularly the larger language models.
However, considering the vast amount of data available on the internet, why is the potential shortage of data a concern? And is there a viable solution to mitigate this risk?
The Significance of High-Quality Data for AI
To cultivate robust, accurate, and top-notch AI systems, a copious amount of data is imperative. For instance, ChatGPT was trained on a staggering 300 billion words or 570 terabytes of language data.
Similarly, the diffusion algorithm, fundamental to various AI image-generating applications such as DALL-E, Lensa, and Midjourney, relied on the LIAON-5B dataset comprising 5.8 billion image and text pairs for its training. Inadequate data can lead to the generation of inaccurate or inferior outcomes by an algorithm.
Another critical aspect is the quality of the training data. While low-quality data like blurry images or social media posts may be readily available, they are insufficient for effectively training high-performance AI models.
Data sourced from social media platforms may be biased, prejudiced, or contain misinformation or illicit content that the model could inadvertently reproduce. For instance, Microsoft’s AI bot exhibited racist and misogynistic tendencies when trained on Twitter data.
Hence, AI developers seek high-quality data sources such as literary works, online articles, scientific papers, Wikipedia, and meticulously curated web content. To enhance the conversational abilities of the Google Assistant, it was trained on 11,000 romance novels from the self-publishing platform Smashwords.
Is the Data Adequate?
The emergence of high-performing models like ChatGPT or DALL-E 3 can be attributed to the AI industry training systems on progressively larger datasets. However, research indicates that the growth rate of online data is significantly slower than the data required to train AI.
If the current trends in AI training persist, a study from the previous year suggests that we may exhaust high-quality textual data reserves before 2026. Additionally, they projected a depletion of low-quality audio and image data between 2030 and 2050.
According to PwC, an accounting and consulting firm, AI has the potential to contribute up to US\(15.7 trillion (A\)24.1 trillion) to the global economy by 2030. Nevertheless, a dearth of suitable data could impede this growth.
Should We Be Alarmed?
Despite the initial apprehension that may arise among AI enthusiasts, the situation may not be as dire as it seems. There exist several strategies to address the looming data scarcity issue and numerous unanswered queries regarding the future evolution of AI models.
One avenue for AI developers is to optimize systems to extract maximum utility from existing data repositories.
In the forthcoming years, they might achieve the training of high-performance AI systems with reduced data requirements and potentially lower computational energy, consequently reducing the carbon footprint of AI technologies.
Furthermore, AI can be leveraged to generate synthetic data for training purposes, enabling programmers to tailor data to suit their specific AI models.
Synthetic data, often sourced from data-generating enterprises like Mostly AI, is already being integrated into various projects and is expected to become more prevalent in the future.
Developers are also exploring avenues to access data not readily available online, such as content from prominent publishers or offline archives. The vast reservoir of pre-digital era publications could serve as a valuable resource for AI projects if digitized and made accessible online.
News Corp., one of the largest news content providers globally, recently announced negotiations with AI developers for content licensing. While a significant portion of their content is behind paywalls, such agreements could compel AI firms to pay for training data, a departure from the prevalent practice of scraping data from the internet for free.
The unauthorized utilization of content for training AI models has sparked backlash from content creators, with some initiating legal action against companies like Microsoft, OpenAI, and StabilityAI. Compensating creators for their contributions could help address the power disparity between artists and AI enterprises.
