What it means
- Once live, keep watching whether the AI still works as tested. Models degrade as customers, markets and data change.
- Track: accuracy or error rates, input data drift, output distribution shifts, fairness metrics, hallucination or complaint rates for GenAI, override rates by human reviewers, latency and availability.
- Set alert thresholds and a response: investigate, retrain, fall back to manual or rules, or switch off.
- Include periodic sample checks of AI decisions by humans to catch silent degradation.
Why it matters
- RBI FREE-AI expects monitoring for drift, degradation, bias and unexplained behaviour, with defined fallbacks, and sample human checks as part of AI BCP.
- A model approved last year may be quietly wrong today; without monitoring nobody notices until complaints arrive.
- ICMR, NITI Aayog and Tamil Nadu guidance all call for ongoing monitoring or periodic re-scoring.
Checklist
- Define monitoring metrics and thresholds per system, based on the acceptance criteria from SC-AI-015 Pre-deployment testing and validation.
- Set up dashboards and automated alerts routed to the system owner.
- Monitor input drift and output drift, not just accuracy (labels often arrive late).
- Track human override rates and complaint volumes linked to AI.
- Run monthly or quarterly sample reviews of AI decisions by a human.
- Write a playbook for each alert: who investigates, when to retrain, when to fall back or switch off.
- Report monitoring results to the AI committee at a set frequency; re-score self-learning systems at least every six months.
Evidence examples
- Monitoring dashboard screenshot for the fraud model with drift and accuracy thresholds.
- Alert history and linked investigation tickets.
- Monthly sample review log of 50 AI decisions checked by a credit officer.
- Quarterly model performance report to the AI committee.
- Fallback playbook and a record of it being used.
Common pitfalls
- Monitoring only uptime and latency, not decision quality.
- Alerts going to a shared inbox nobody reads.
- No threshold for "switch off", so a bad model runs for weeks.
- Ignoring vendor model changes that show up as sudden output shifts.
Satisfies
- VoluntaryISO/IEC 42001
- BindingEU AI Act
- SupervisoryPRA SS1/23
- VoluntaryNIST AI RMF and GenAI Profile
- BindingTexas TRAIGA
- BindingKorea AI Basic Act
- SupervisoryOSFI E-23
- VoluntaryRBI FREE-AI
Also relevant to
- ICMR AI in Healthcare Guidelinesno numbered obligations; mapped in its practical notes (ongoing monitoring)
- NITI Aayog Responsible AI for Allno numbered obligations; mapped in its practical notes (monitor through the lifecycle)
- Tamil Nadu Safe and Ethical AI Policyno numbered obligations; mapped in its practical notes (six-monthly re-scoring of self-learning systems)
Standards crosswalk
Clause references only. Check the standard itself for the exact wording.
ISO/IEC 42001
- 9.1 Monitoring, measurement, analysis and evaluation
- A.6.2.6 AI system operation and monitoring
- A.6.2.8 AI system recording of event logs
ISO/IEC 27001
- A.8.6 Capacity management
- A.8.15 Logging
- A.8.16 Monitoring activities
SOC 2
- CC4.1
- CC7.2
- PI1.4
Key takeaways
- Watch decision quality, drift and fairness in production, with thresholds and a playbook.
- Partial reuse from ISO 27001: logging and monitoring tooling and processes (A.8.15, A.8.16) carry over; model performance and drift metrics are new.
- Add human sample checks to catch silent degradation.