Is it possible for an AI to be trained just on data generated by another AI? It might sound like a harebrained idea. But it’s one that’s been around for quite some time — and as new, real data is increasingly hard to come by, it’s been gaining traction.
Anthropic used some synthetic data to train one of its flagship models, Claude 3.5 Sonnet. Meta fine-tuned its Llama 3.1 models using AI-generated data. And OpenAI is said to be sourcing synthetic training data from o1, its “reasoning” model, for the upcoming Orion.
But why does AI need data in the first place — and what kind of data does it need? And can this data really be replaced by synthetic data?
The importance of annotations
AI systems are statistical machines. Trained on a lot of examples, they learn the patterns in those examples to make predictions, like that “to whom” in an email typically precedes “it may concern.”
Annotations, usually text labeling the meaning or parts of the data these systems ingest, are a key piece in these examples. They serve as guideposts, “teaching” a model to distinguish among things, places, and ideas.
Consider a photo-classifying model shown lots of pictures of kitchens labeled with the word “kitchen.” As it trains, the model will begin to make associations between “kitchen” and general characteristics of kitchens (e.g. that they contain fridges and countertops). After training, given a photo of a kitchen that wasn’t included in the initial examples, the model should be able to identify it as such. (Of course, if the pictures of kitchens were labeled “cow,” it would identify them as cows, which emphasizes the importance of good annotation.)
The appetite for AI and the need to provide labeled data for its development have ballooned the market for annotation services. Dimension Market Research estimates that it’s worth $838.2 million today — and will be worth $10.34 billion in the next 10 years. While there aren’t precise estimates of how many people engage in labeling work, a 2022 paper pegs the number in the “millions.”
Companies large and small rely on workers employed by data annotation firms to create labels for AI training sets. Some of these jobs pay reasonably well, particularly if the labeling requires specialized knowledge (e.g. math expertise). Others can be backbreaking. Annotators in developing countries are paid only a few dollars per hour on average, without any benefits or guarantees of future gigs.
A drying data well
So there’s humanistic reasons to seek out alternatives to human-generated labels. For example, Uber is expanding its fleet of gig workers to work on AI annotation and data labeling. But there are also practical ones.
Humans can only label so fast. Annotators also have biases that can manifest in their annotations, and, subsequently, any models trained on them. Annotators make mistakes, or get tripped up by labeling instructions. And paying humans to do things is expensive.
Data in general is expensive, for that matter. Shutterstock is charging AI vendors tens of millions of dollars to access its archives, while Reddit has made hundreds of millions from licensing data to Google, OpenAI, and others.
Lastly, data is also becoming harder to acquire.
Most models are trained on massive collections of public data — data that owners are increasingly choosing to gate over fears their data will be plagiarized, or that they won’t receive credit or attribution for it. More than 35% of the world’s top 1,000 websites now block OpenAI’s web scraper. And around 25% of data from “high-quality” sources has been restricted from the major datasets used to train models, one recent study found.
Should the current access-blocking trend continue, the research group Epoch AI projects that developers will run out of data to train generative AI models between 2026 and 2032. That, combined with fears of copyright lawsuits and objectionable material making their way into open datasets, has forced a reckoning for AI vendors.
Synthetic alternatives
At first glance, synthetic data would appear to be the solution to all these problems. Need annotations? Generate ’em. More example data? No problem. The sky’s the limit.
And to a certain extent, this is true.
“If ‘data is the new oil,’ synthetic data pitches itself as biofuel, creatable without the negative externalities of the real thing,” Os Keyes, a PhD candidate at the University of Washington who studies the ethical impact of emerging technologies, told TechCrunch. “You can take a small starting set of data and simulate and extrapolate new entries from it.”
The AI industry has taken the concept and run with it.
This month, Writer, an enterprise-focused generative AI company, debuted a model, Palmyra X 004, trained almost entirely on synthetic data. Developing it cost just $700,000, Writer claims — compared to estimates of $4.6 million for a comparably-sized OpenAI model.
Microsoft’s Phi open models were trained using synthetic data, in part. So were Google’s Gemma models. Nvidia this summer unveiled a model family designed to generate synthetic training data, and AI startup Hugging Face recently released what it claims is the largest AI training dataset of synthetic text.
Synthetic data generation has become a business in its own right — one that could be worth $2.34 billion by 2030. Gartner predicts that 60% of the data used for AI and analytics projects this year will be synthetically generated.
Luca Soldaini, a senior research scientist at the Allen Institute for AI, noted that synthetic data techniques can be used to generate training data in a format that’s not easily obtained through scraping (or even content licensing). For example, in training its video generator Movie Gen, Meta used Llama 3 to create captions for footage in the training data, which humans then refined to add more detail, like descriptions of the lighting.
Along these same lines, OpenAI says that it fine-tuned GPT-4o using synthetic data to build the sketchpad-like Canvas feature for ChatGPT. And Amazon has said that it generates synthetic data to supplement the real-world data it uses to train speech recognition models for Alexa.
“Synthetic data models can be used to quickly expand upon human intuition of which data is needed to achieve a specific model behavior,” Soldaini said.
Synthetic risks
Synthetic data is no panacea, however. It suffers from the same “garbage in, garbage out” problem as all AI. Models create synthetic data, and if the data used to train these models has biases and limitations, their outputs will be similarly tainted. For instance, groups poorly represented in the base data will be so in the synthetic data.
“The problem is, you can only do so much,” Keyes said. “Say you only have 30 Black people in a dataset. Extrapolating out might help, but if those 30 people are all middle-class, or all light-skinned, that’s what the ‘representative’ data will all look like.”
To this point, a 2023 study by researchers at Rice University and Stanford found that over-reliance on synthetic data during training can create models whose “quality or diversity progressively decrease.” Sampling bias — poor representation of the real world — causes a model’s diversity to worsen after a few generations of training, according to the researchers (although they also found that mixing in a bit of real-world data helps to mitigate this).
Keyes sees additional risks in complex models such as OpenAI’s o1, which he thinks could produce harder-to-spot hallucinations in their synthetic data. These, in turn, could reduce the accuracy of models trained on the data — especially if the hallucinations’ sources aren’t easy to identify.
“Complex models hallucinate; data produced by complex models contain hallucinations,” Keyes added. “And with a model like o1, the developers themselves can’t necessarily explain why artefacts appear.”
Compounding hallucinations can lead to gibberish-spewing models. A study published in the journal Nature reveals how models, trained on error-ridden data, generate even more error-ridden data, and how this feedback loop degrades future generations of models. Models lose their grasp of more esoteric knowledge over generations, the researchers found — becoming more generic and often producing answers irrelevant to the questions they’re asked.
A follow-up study shows that other types of models, like image generators, aren’t immune to this sort of collapse:
Soldaini agrees that “raw” synthetic data isn’t to be trusted, at least if the goal is to avoid training forgetful chatbots and homogenous image generators. Using it “safely,” he says, requires thoroughly reviewing, curating, and filtering it, and ideally pairing it with fresh, real data — just like you’d do with any other dataset.
Failing to do so could eventually lead to model collapse, where a model becomes less “creative” — and more biased — in its outputs, eventually seriously compromising its functionality. Though this process could be identified and arrested before it gets serious, it is a risk.
“Researchers need to examine the generated data, iterate on the generation process, and identify safeguards to remove low-quality data points,” Soldaini said. “Synthetic data pipelines are not a self-improving machine; their output must be carefully inspected and improved before being used for training.”
OpenAI CEO Sam Altman once argued that AI will someday produce synthetic data good enough to effectively train itself. But — assuming that’s even feasible — the tech doesn’t exist yet. No major AI lab has released a model trained on synthetic data alone.
At least for the foreseeable future, it seems we’ll need humans in the loop somewhere to make sure a model’s training doesn’t go awry.
TechCrunch has an AI-focused newsletter! Sign up here to get it in your inbox every Wednesday.
Update: This story was originally published on October 23 and was updated December 24 with more information.
Hai, para pencinta slot! Pernah denger semboyan “slot gaco”? jika tidak, siap-siap jatuh cinta sama program ini. slot gacor merupakan mesin slots yang sering kasih win. Ya, slot-slot ini bisa dikatakan adalah jagoannya tuk bawa come back cuan. tapi, cemana sih
tekniknya jumpain slot gaco yang benar? Tenang Bro, kita beri santai aja di tempat ini
Game tergacor waktu ini satu-satunya di Indonesia yaitu pasti menyediakan return on Investment terbaik
Daftar di:
Informasi mengenai KING SLOT, Segera Daftar Bersama king selot terbaik dan terpercaya no satu di Indonesia. Boleh mendaftar melalui sini king slot serta memberikan hasil kembali yang paling tinggi saat sekarang ini hanyalah KING SLOT atau Raja slot paling gacor, gilak dan gaco saat sekarang di Indonesia melalui program return tinggi di kingselot serta pg king slot
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
situs slot terbaru
situs slot terbaru permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
slot terbaru
slot terbaru permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
sewaapartemen permainan paling top dan garansi imbal balik hasil besar bersama sewaapartemen.org
pipapvc permainan paling top dan garansi imbal balik hasil besar bersama pipapvc.org
pialaeropa2028 permainan paling top dan garansi imbal balik hasil besar bersama pialaeropa2028.net
pialadunia2026 permainan paling top dan garansi imbal balik hasil besar bersama pialadunia2026.asia
bahanbangunan permainan paling top dan garansi imbal balik hasil besar bersama bahanbangunan.org
gemini999 permainan paling top dan garansi imbal balik hasil besar bersama gemini999.org
havana888 permainan paling top dan garansi imbal balik hasil besar bersama havana888.net
indonesia88 permainan paling top dan garansi imbal balik hasil besar bersama indonesia88.info
inspirasi permainan paling top dan garansi imbal balik hasil besar bersama inspirasi.info
jokiwin88 permainan paling top dan garansi imbal balik hasil besar bersama jokiwin88.org
oli88 permainan paling top dan garansi imbal balik hasil besar bersama oli88.org
situs88 permainan paling top dan garansi imbal balik hasil besar bersama situs88.asia
transfermarkt88 permainan paling top dan garansi imbal balik hasil besar bersama transfermarkt88.com
rossi46 permainan paling top dan garansi imbal balik hasil besar bersama rossi46.com
pecco88 permainan paling top dan garansi imbal balik hasil besar bersama pecco88.com
pecco permainan paling top dan garansi imbal balik hasil besar bersama pecco.asia
pangkas permainan paling top dan garansi imbal balik hasil besar bersama pangkas.net
facebook88 permainan paling top dan garansi imbal balik hasil besar bersama facebook88.com
depo88 permainan paling top dan garansi imbal balik hasil besar bersama depo88.asia
bajaringan88 permainan paling top dan garansi imbal balik hasil besar bersama bajaringan88.com
baja ringan permainan paling top dan garansi imbal balik hasil besar bersama bajaringan.asia
tradingview88 permainan paling top dan garansi imbal balik hasil besar bersama tradingview88.com
trading88 permainan paling top dan garansi imbal balik hasil besar bersama trading88.org
tokopedia88 permainan paling top dan garansi imbal balik hasil besar bersama tokopedia88.net
kualifikasi permainan paling top dan garansi imbal balik hasil besar bersama kualifikasi.net
grosirboneka permainan paling top dan garansi imbal balik hasil besar bersama grosirboneka.net
situsgacor permainan paling top dan garansi imbal balik hasil besar bersama situsgacor.news
prediksi88 permainan paling top dan garansi imbal balik hasil besar bersama prediksi88.biz
keramat88 permainan paling top dan garansi imbal balik hasil besar bersama keramat88.biz
cnn88 permainan paling top dan garansi imbal balik hasil besar bersama cnn88.asia
cnbc88 permainan paling top dan garansi imbal balik hasil besar bersama cnbc88.com
bloomberg88 permainan paling top dan garansi imbal balik hasil besar bersama bloomberg88.com
antibocor88 permainan paling top dan garansi imbal balik hasil besar bersama antibocor88.com
buku88 permainan paling top dan garansi imbal balik hasil besar bersama buku88.org
dukun88 permainan paling top dan garansi imbal balik hasil besar bersama dukun88.info
indodax88 permainan paling top dan garansi imbal balik hasil besar bersama indodax88.net
investing88 permainan paling top dan garansi imbal balik hasil besar bersama investing88.net
makanan88 permainan paling top dan garansi imbal balik hasil besar bersama makanan88.net
pakaian88 permainan paling top dan garansi imbal balik hasil besar bersama pakaian88.com
rambo88 permainan paling top dan garansi imbal balik hasil besar bersama rambo88.info
santet88 permainan paling top dan garansi imbal balik hasil besar bersama santet88.net
sekolah88 permainan paling top dan garansi imbal balik hasil besar bersama sekolah88.org
uno88 permainan paling top dan garansi imbal balik hasil besar bersama uno88.info
starlink88 permainan paling top dan garansi imbal balik hasil besar bersama starlink88.org
solitaire88 permainan paling top dan garansi imbal balik hasil besar bersama solitaire88.net
roblox88 permainan paling top dan garansi imbal balik hasil besar bersama roblox88.org
pubg88 permainan paling top dan garansi imbal balik hasil besar bersama pubg88.info
motogp88 permainan paling top dan garansi imbal balik hasil besar bersama motogp88.net
motivasi88 permainan paling top dan garansi imbal balik hasil besar bersama motivasi88.com
mobilelegends88 permainan paling top dan garansi imbal balik hasil besar bersama mobilelegends88.com
ludoking permainan paling top dan garansi imbal balik hasil besar bersama ludoking.asia
kucing88 permainan paling top dan garansi imbal balik hasil besar bersama kucing88.asia
kucing permainan paling top dan garansi imbal balik hasil besar bersama kucing.info
klasmen88 permainan paling top dan garansi imbal balik hasil besar bersama klasmen88.com
imdb88 permainan paling top dan garansi imbal balik hasil besar bersama imdb88.net
higgsdomino permainan paling top dan garansi imbal balik hasil besar bersama higgsdomino.biz
gmail88 permainan paling top dan garansi imbal balik hasil besar bersama gmail88.org
gmail88 permainan paling top dan garansi imbal balik hasil besar bersama gmail88.net
chess88 permainan paling top dan garansi imbal balik hasil besar bersama chess88.info
callofduty88 permainan paling top dan garansi imbal balik hasil besar bersama callofduty88.com
semen88 permainan paling top dan garansi imbal balik hasil besar bersama semen88.com
seluler88 permainan paling top dan garansi imbal balik hasil besar bersama seluler88.com
ram88 permainan paling top dan garansi imbal balik hasil besar bersama ram88.net
pasir88 permainan paling top dan garansi imbal balik hasil besar bersama pasir88.asia
papan88 permainan paling top dan garansi imbal balik hasil besar bersama papan88.info
komputer88 permainan paling top dan garansi imbal balik hasil besar bersama komputer88.net
kayu88 permainan paling top dan garansi imbal balik hasil besar bersama kayu88.info
kabel88 permainan paling top dan garansi imbal balik hasil besar bersama kabel88.net
gaming88 permainan paling top dan garansi imbal balik hasil besar bersama gaming88.asia
games88 permainan paling top dan garansi imbal balik hasil besar bersama games88.info
beton88 permainan paling top dan garansi imbal balik hasil besar bersama beton88.lat
teknis88 permainan paling top dan garansi imbal balik hasil besar bersama teknis88.com
sarana88 permainan paling top dan garansi imbal balik hasil besar bersama sarana88.info
arena88 permainan paling top dan garansi imbal balik hasil besar bersama arena88.asia
viral888 permainan paling top dan garansi imbal balik hasil besar bersama viral888.asia
usdt88 permainan paling top dan garansi imbal balik hasil besar bersama usdt88.net
uang88 permainan paling top dan garansi imbal balik hasil besar bersama uang88.asia
twitter88 permainan paling top dan garansi imbal balik hasil besar bersama twitter88.asia
tahanlama88 permainan paling top dan garansi imbal balik hasil besar bersama tahanlama88.com
suka88 permainan paling top dan garansi imbal balik hasil besar bersama suka88.biz
sekali88 permainan paling top dan garansi imbal balik hasil besar bersama sekali88.org
plafonpvc permainan paling top dan garansi imbal balik hasil besar bersama plafonpvc.net
pinjol88 permainan paling top dan garansi imbal balik hasil besar bersama pinjol88.asia
mpox88 permainan paling top dan garansi imbal balik hasil besar bersama mpox88.com
daun88 permainan paling top dan garansi imbal balik hasil besar bersama daun88.asia
wordpress88 permainan paling top dan garansi imbal balik hasil besar bersama wordpress88.net
whatsapp88 permainan paling top dan garansi imbal balik hasil besar bersama whatsapp88.net
wechat88 permainan paling top dan garansi imbal balik hasil besar bersama wechat88.net
telegram88 permainan paling top dan garansi imbal balik hasil besar bersama telegram88.org
super888 permainan paling top dan garansi imbal balik hasil besar bersama super888.asia
skype88 permainan paling top dan garansi imbal balik hasil besar bersama skype88.com
racing88 permainan paling top dan garansi imbal balik hasil besar bersama racing88.asia
pupuk88 permainan paling top dan garansi imbal balik hasil besar bersama pupuk88.net
petani888 permainan paling top dan garansi imbal balik hasil besar bersama petani888.org
pabrik88 permainan paling top dan garansi imbal balik hasil besar bersama pabrik88.asia
organik88 permainan paling top dan garansi imbal balik hasil besar bersama organik88.com
online77 permainan paling top dan garansi imbal balik hasil besar bersama online77.info
ojek88 permainan paling top dan garansi imbal balik hasil besar bersama ojek88.info
nostalgia88 permainan paling top dan garansi imbal balik hasil besar bersama nostalgia88.com
michat permainan paling top dan garansi imbal balik hasil besar bersama michat.info
