What it means
- Deliberately try to break the AI system the way an attacker or a careless user would, and fix what you find.
- For GenAI: prompt injection, jailbreaks, getting the bot to leak system prompts or customer data, generating harmful or unlawful content, bypassing content labels.
- For predictive models: adversarial inputs, data poisoning, manipulated documents, gaming the model (for example applicants learning which inputs get approval).
- Frequency set by tier, plus extra runs triggered by major changes or new attack techniques.
Why it matters
- RBI FREE-AI recommends red teaming at least half-yearly for medium and high risk AI, at least before deployment for low risk, plus trigger-based runs, with findings documented for audit and supervisors.
- Normal testing checks the system works; red teaming checks it fails safely under attack.
- GenAI attack techniques change monthly; last year's test is stale.
Checklist
- Set red teaming frequency by tier (for example half-yearly for medium and high, pre-deployment for low) and list trigger events.
- Build or buy a test library covering prompt injection, jailbreaks, data leakage, harmful content, and model-specific attacks.
- Use people independent of the build team (internal security team or external specialists).
- Rate findings by severity and track fixes to closure; retest after fixes.
- Feed new attack patterns from threat intelligence into the test library.
- Keep reports for audit and supervisors.
Evidence examples
- Red team report for the customer service chatbot (date, scope, attack categories, findings, severity).
- Findings tracker with fix and retest dates.
- Red teaming schedule mapped to AI inventory tiers.
- External specialist engagement letter and scope.
Common pitfalls
- Treating a standard web application pentest as AI red teaming; it usually misses prompt-level attacks.
- One run before launch, never repeated.
- Findings fixed with a prompt tweak and not retested.
- Not testing vendor models in your configuration.
Satisfies
- BindingEU AI Act
- VoluntaryNIST AI RMF and GenAI Profile
- BindingTexas TRAIGA
- BindingKorea AI Basic Act
- VoluntaryRBI FREE-AI
Standards crosswalk
Clause references only. Check the standard itself for the exact wording.
ISO/IEC 42001
- 6.1.2 AI risk assessment
- A.6.2.4 AI system verification and validation
ISO/IEC 27001
- A.5.7 Threat intelligence
- A.8.8 Management of technical vulnerabilities
- A.8.29 Security testing in development and acceptance
SOC 2
- CC4.1
- CC7.1
Key takeaways
- Attack your own AI on a tier-based schedule and after major changes.
- Partial reuse from ISO 27001: security testing, vulnerability management and threat intelligence (A.8.29, A.8.8, A.5.7) give the process; AI attack techniques are new.
- Keep reports; RBI FREE-AI expects them available to supervisors.