TL;DR
AI red teaming is the practice of attacking your own AI systems to find out how they fail before an attacker does. It covers prompt injection, jailbreaks, data poisoning, model evasion, and the ordinary software bugs sitting underneath the model. The phrase describes two different jobs, which is why the search results feel so scattered: testing AI systems, and using AI to test everything else.
Microsoft’s AI red team has run more than 80 operations covering over 100 generative AI products. Their second published lesson was blunt: you do not need to compute gradients to break an AI system. The simple tricks kept working.
What is AI red teaming?
AI red teaming means attacking an AI system on purpose, inside an agreed scope, to learn how it breaks. Testers try to make the model leak data it should protect, ignore the rules it was given, produce harmful output, or take actions nobody approved.
The confusion starts with the name. One phrase covers two jobs.
Red teaming AI systems versus using AI to red team your network
In the first job, the AI is the target. A tester attacks a chatbot, a retrieval pipeline, or an agent that can call tools, and reports what they got it to do. Every major course and certification on the market teaches this version.
In the second job, the AI is the tester. Software runs offensive testing against your network, your cloud, and your applications, and reports what it reached.
Both are real work. When a vendor sells AI red teaming to a security operations team, they usually mean the second one. When a job board posts an AI red teamer role, it usually means the first. Check which one you are reading about before you budget for it.
Who is attacking whom?
Courses, certifications and job posts almost always mean the first one. Security vendors selling to a SOC almost always mean the second. Check which you are reading before you budget for it.
The model, the retrieval pipeline and the agent are the target. Testers try prompt injection, jailbreaks, data poisoning and evasion, then report what they got it to do.
The AI runs the offensive work. It attacks your cloud, your network and your applications inside an approved scope, then reports what it actually reached.
Same keyword, different purchase. One buys assurance about your AI. The other buys testing capacity for everything else.
Why AI systems break differently
Three things make this harder than the testing you already run.
The same input can produce a different answer twice. A test that passes this morning can fail this afternoon, so a single clean result proves very little.
The attack surface moves under you. Change a system prompt, connect a new data source, or give the model one more tool to call, and your old findings expire.
The model is only part of the target. Attackers go after the application wrapped around it, the data feeding it, the tools it can reach, and the servers hosting all of it.
What AI red teamers actually attack
Prompt injection and jailbreaks
OWASP ranks prompt injection first in its Top 10 for LLM Applications 2025, as LLM01. The attacker plants instructions the model treats as orders, either typed directly or hidden inside a document, a web page, or an email the model reads later.
Jailbreaking is the close cousin. The goal is to get the model past its own guardrails and into output it was built to refuse.
Data and model attacks
Poisoning corrupts the training or tuning data so the model learns something the attacker wants. Evasion crafts an input that slips past a classifier. Membership inference works out whether a specific record was in the training set, which matters a great deal when the training set held customer data.
The application and agent layer
Output handling is where a lot of damage happens. If the model returns text your application trusts, you can end up with cross site scripting, SQL injection, or code execution, delivered by your own AI feature.
Agents raise the stakes. An agent that can call tools, read files, and connect to other services gives an attacker a much longer list of things to abuse.
The plain infrastructure around the model
Microsoft’s team found an outdated FFmpeg component inside a video processing application, which opened a server side request forgery flaw. No model weights involved. Their conclusion was that AI security still needs basic security hygiene.
Scope your AI red team work to include the stack underneath. Some of the most useful findings are ordinary software bugs in a system nobody had tested since the AI feature shipped.
How AI red teaming compares to the testing you already buy
AI red teaming versus penetration testing as a service
Penetration testing as a service gives you a repeatable, scoped test on a schedule, often with a portal and a retest. Scope is the gap. A standard engagement checks the application and infrastructure, and stops at the model. Asking a tester to attack an AI feature without naming the model behavior, the tools it can call, and the data it retrieves gets you a report about the login page.
AI red teaming versus breach and attack simulation
Breach and attack simulation runs known attack behavior against your controls, over and over, to confirm your detections still fire. It gives you wide coverage of techniques that are already documented. AI red teaming works the other way: a narrow target, and a human trying things the library does not contain yet.
Teams that run both use simulation to watch for drift in detection coverage, and red teaming to find the failure nobody had written down.
Three tests that are not interchangeable
Teams often buy one and assume it covers the others. It does not. Here is what each one actually gives you.
Simulation watches for detection drift. Red teaming finds the failure nobody predicted. Most mature teams run both.
AI red teaming tools
PyRIT and Attack Success Rate
PyRIT is Microsoft’s open source toolkit for probing generative AI systems. It handles prompt generation, attack orchestration, and scoring of the responses.
Whatever tool you run, track Attack Success Rate: the share of attempts that got the result the attacker wanted. Break it down by technique and risk category so you can compare it across versions.
Attack Success Rate is a useful number for regression testing. Run the same scan after each release and watch whether the figure moves.
What tools cannot do for you
Microsoft’s own position is the clearest statement on this. Automation surfaces candidate risks, and expert human teams analyze them. Their team found that language models are unreliable judges of specialized content such as medicine, cybersecurity, and chemical or biological risk, so subject matter experts have to stay in the loop.
Cultural context and judgment do not automate either. Deciding whether a chatbot handled a user in distress correctly is not a scoring problem.
How to run your first AI red team exercise
- Write down what the system does, who uses it, and what it can touch. Scope the data, the tools, and the permissions, not only the model.
- Build a short threat model. Name the three outcomes that would actually hurt, such as customer data leaving, an agent spending money, or the model giving dangerous instructions.
- Start with simple techniques. Direct prompt injection, hidden instructions in retrieved content, and basic encoding tricks find more than exotic methods do.
- Test the application layer with normal application testing. Check how your code handles whatever the model returns.
- Record attempts and results, not just the wins. Attack Success Rate only means something if you log the denominator.
- Fix, then retest, then keep the before and after. Repeat on a cadence, because the system will change.
AI red teaming courses, certifications, and careers
The Hack The Box path
Hack The Box Academy runs an AI Red Teamer job role path of 12 modules, covering prompt injection, output attacks, data attacks, evasion, privacy, and defense. Finishing the path at 100% is a prerequisite for the HTB Certified Offensive AI Expert exam.
The exam is a hands on assessment in a simulated enterprise environment with model deployments and applications built on language models. You get a single 7 day window to run the engagement, submit flags, and upload a commercial grade report. Results come back within 20 business days.
The path itself was built with Google and follows Google’s SAIF. Hack The Box’s certifications page maps the wider curriculum to the OWASP ML Security Top 10, the OWASP Agentic Top 10, and the OWASP Top 10 for LLM Applications 2025.
The OffSec path
OffSec’s AI-300 course, Advanced AI Red Teaming, leads to the OSAI and OSAI+ certifications. It covers attacking language models, agent systems, retrieval pipelines, embeddings, and AI infrastructure, with roughly 65 hours of content.
The exam is 24 hours and proctored, run as a red team engagement against an AI enabled enterprise environment. OffSec recommends OSCP level experience or equivalent hands on background before you start. The course and certification bundle starts at $1,749.
Jobs and pay
Be careful with the salary numbers you see, because the title covers two very different jobs.
Most of what gets advertised as AI red teaming is contract work, brokered through platforms and paid by the hour or by the task. One job aggregator, measuring 34 advertised rates across 12 platforms in November 2026, put that work at $45 to $125 an hour with a median of $80. That figure is for independent contractor gigs with no guaranteed hours.
Full time AI security engineering is a different job with a different pay structure, usually base plus equity. There is no government occupation code for it yet and no salary survey with a disclosed sample size, so the six figure ranges circulating on career blogs are estimates rather than measured data. Read the required skills and the scope before you read the number.
What regulators expect
Article 15 of the EU AI Act requires high risk systems to be resilient against attempts by outsiders to alter their use, outputs, or performance. It names data poisoning, model poisoning, and adversarial examples directly, and calls for measures to prevent, detect, respond to, and control those attacks.
Read that carefully, because it stops short of naming a method. The Act does not order you to run red teaming. It asks you to show resilience against exactly the attacks red teaming is built to find.
The dates moved, so check them before you plan. The Digital Omnibus, in force since July 27, 2026, sets December 2, 2027 as the compliance date for standalone high risk systems, and August 2, 2028 for high risk systems built into regulated products.
How Secure.com helps
Finding AI weaknesses is the easy half. Proving you closed them, with a record of who approved what, is where most teams stall.
- The Red Teammate attacks inside the scope and permissions your team sets, so testing happens on your terms.
- It confirms what is actually exploitable and the path to get there, so your queue gets ordered by real risk instead of severity scores.
- Defensive teammates turn the finding into the work: a configuration change, a control improvement, a detection, or a fix.
- Every retest records the before and after state, plus the decision and approval trail, which is the evidence an auditor asks for.
- Secure.com runs above the tools you already own, goes live in days, and keeps human approval on consequential action.
Attack. Harden. Prove. Repeat.
Finding the gap is the easy half
A finding only counts once someone has fixed it and kept the record. This is the loop that turns an attack into evidence.
Each cycle feeds the next, so the next test starts smarter than the last one.
Frequently asked questions
Is AI red teaming the same as AI penetration testing?
Do I need a certification to get into AI red teaming?
Can AI red teaming be fully automated?
What is Attack Success Rate?
How often should we red team an AI system?
Recommended reads
- Attack Path Modeling vs Vulnerability Scanning: why prioritization should follow exploitability, not severity scores.
- AI Compliance and Human Approval Gates: how the approval gate works when you need to evidence governed action.
