TL;DR: Data poisoning is when attackers slip bad data into what an AI model learns from or looks up, so it gives wrong or attacker chosen answers. It takes far less than most teams think: 250 documents were enough to backdoor models up to 13 billion parameters, and 5 planted texts per question fooled RAG systems 90% of the time. The best defenses are knowing where your data comes from, locking down who can change it, and attacking your own AI before every update.
In one recent survey, 26% of US and UK organizations said they were hit by AI data poisoning in the past year. Nothing crashes when it happens. The model keeps answering, only now it answers the way an attacker wants.
What Is Data Poisoning?
Data poisoning is a cyberattack where someone corrupts the data an AI model learns from so the model behaves the way the attacker wants.
Picture a spam filter. If an attacker feeds it thousands of spam emails labeled “safe,” the filter learns to let spam through.
The poison can sit in training data, fine tuning data, or the documents a model looks up when it answers.
OWASP lists it as LLM04:2025, “Data and Model Poisoning,” in its Top 10 risks for large language model apps. The 2025 update widened the risk from training data alone to fine tuning and embedding data too.
How a Data Poisoning Attack in AI Works
A model is a mirror of its data. Change the data and you change the model.
Targeted vs Untargeted Attacks
A targeted attack goes after one behavior, like making a malware detector ignore one malware family.
An untargeted attack makes the whole model worse, usually by flooding it with noisy or mislabeled data.
In one lab study, poisoning cut accuracy by up to 27% in an image recognition model and 22% in a fraud detection model.
Why a Small Dose Is Enough
For years, people assumed attackers needed to control a big slice of the training data.
A 2025 study run with the UK AI Security Institute and the Alan Turing Institute showed otherwise. Researchers backdoored language models from 600 million to 13 billion parameters with only 250 malicious documents.
For the largest model, those documents were about 0.00016% of its training data.
So count suspicious samples, not percentages.
A tiny dose. A big result.
Attackers do not need to own your dataset. A handful of planted samples is often enough.
Defenders should count suspicious samples, not percentages. Small and steady beats big and noisy.
Types of Data Poisoning Attacks
Label Flipping
Attackers swap correct labels for wrong ones, like tagging malware samples as safe.
Data Injection
Attackers add brand new fake records that nudge the model toward a result they picked.
Backdoor Attacks
Attackers plant a hidden trigger, such as a rare word or a pixel pattern. The model acts normal until the trigger shows up.
Clean Label Attacks
The poisoned samples look correct to a human reviewer, so they slip past normal data checks.
Availability Attacks
Attackers pile in junk data until the model is too inaccurate to trust.
Data Poisoning in AI Examples
- Tay chatbot, 2016: A public chatbot that learned from Twitter users got flooded with toxic posts. It started repeating them and was pulled offline in less than 24 hours.
- Nightshade, 2023: University researchers built a tool that adds hidden changes to images. Image models that scrape those pictures learn the wrong links between words and images.
- Poisoned instruction tuning: Researchers showed that slipping 50 to 500 poisoned examples into user submitted training data could change how a language model acts across many tasks.
- Model hub uploads: Security researchers found a flaw on a popular AI model hub that let attackers upload malicious data, which anyone downloading from it could inherit.
Data Poisoning in RAG Systems
RAG (retrieval augmented generation) lets a model look up your own documents before it answers. That makes it more useful, and it turns those documents into attack surface.
In the PoisonedRAG study, presented at USENIX Security 2025, researchers planted 5 malicious texts per target question in a knowledge base holding millions of texts. The model gave the attacker’s chosen answer 90% of the time.
Defenses the researchers tested, like paraphrasing questions and flagging odd text, were not enough to stop it.
Any source your RAG pulls from can be poisoned: an internal wiki anyone can edit, a ticket queue, a shared drive, or a public web page. An insider can poison a private knowledge base as easily.
Five doors. One poisoned model.
Poison does not need to touch your model directly. It rides in through the data your model learns from or reads at answer time.
Most teams guard the model. Attackers go after the data supply chain because it is wider, softer, and rarely tested.
Data Poisoning Attacks Against Federated Learning Systems
Federated learning trains one shared model across many devices without moving the raw data. Each device trains on its own data and sends back an update.
That keeps data private, but it also means you cannot see what each device trained on.
Data Poisoning vs Model Poisoning in Federated Learning
In data poisoning, a bad device trains on tainted data and sends an honest looking update.
In model poisoning, a bad device skips the data step and tampers with the update itself.
A USENIX Security 2020 study showed that crafted updates could sharply raise error rates, even in federated systems built to resist bad clients.
A later NDSS study found its attacks cut model accuracy 1.5 to 60 times more than earlier known attacks.
Data Poisoning vs Prompt Injection
The difference comes down to timing.
Data poisoning corrupts what the model learned or what it retrieves, so the damage stays until you find and remove it.
Prompt injection hides instructions in what the model reads right now, like a chat message or an email it is asked to summarize. It usually ends with the session.
OWASP ranks prompt injection as LLM01:2025 and data and model poisoning as LLM04:2025.
RAG blurs the line, because a poisoned document can carry an injected instruction. Test for both.
Same goal. Very different timing.
Both attacks bend what an AI says or does. Poisoning hides in what the model learned. Prompt injection hides in what the model reads right now.
In RAG systems the line blurs. A poisoned document can carry an injected instruction, so test for both.
How to Detect Data Poisoning
Poisoning is hard to catch because a backdoored model can pass standard tests and still misbehave when its trigger appears.
Watch for these signs:
- Accuracy drops with no clear cause after a data or model update.
- Odd answers tied to specific words, names, or formats.
- New data sources or contributors that showed up right before a behavior change.
- RAG answers that cite documents nobody on your team recognizes.
- Outputs that drift away from a trusted baseline over time.
Keep a set of canary prompts with known good answers, and rerun them after every change.
How to Prevent Data Poisoning
Once a model learns from bad data, removing it often means retraining, so prevention costs far less than cleanup.
- Track provenance: know where every dataset and document came from, and keep an AI bill of materials.
- Lock down write access to training sets, fine tuning sets, and RAG sources.
- Version datasets next to model versions so you can roll back fast.
- Validate and clean data before it goes in, and flag outliers for review.
- Red team your AI before every model swap, new data source, or new tool connection.
- Monitor production outputs and alert on drift.
Red teaming is the step most teams skip. If you are starting from zero, this guide to AI red teaming shows how to scope the tests.
How Secure.com Helps
Data poisoning wins when nobody attacks the data supply chain before an attacker does. Secure.com runs Governed Defense, Powered by Offense: AI Teammates that attack your setup, harden what they find, and prove the fix, with a human approving every consequential action.
- Red Teammate: Runs attack simulations to show what an attacker could reach, including the paths into your AI data sources.
- AppSec Teammate: Flags risky code, dependencies, and data flows in the apps that feed and serve your models.
- Infrastructure Security Teammate: Spots loose write access and misconfigured storage around training data and RAG sources.
- SOC Teammate: Watches for unusual activity and routes alerts to analysts with the context already gathered.
- Governed by default: Consequential actions wait for human approval, and every step is logged as proof.
Attack. Harden. Prove. Repeat. No more grunt work. No more burnout.