Skip to main content

Skill Guide

AI/ML Model Risk Management (SR 11-7 / SS1/23)

AI/ML Model Risk Management (MRM) is the structured framework for identifying, measuring, monitoring, and controlling the risks arising from the development, validation, implementation, and use of artificial intelligence and machine learning models within an enterprise, guided by regulatory standards like the Federal Reserve's SR 11-7 and the UK's SS1/23.

This skill ensures AI/ML systems are robust, fair, and compliant, directly protecting the organization from significant financial loss, regulatory penalty, and reputational damage. It enables the safe scaling of AI/ML initiatives by providing the governance necessary to meet board-level and supervisory expectations.
1 Careers
1 Categories
9.1 Avg Demand
15% Avg AI Risk

How to Learn AI/ML Model Risk Management (SR 11-7 / SS1/23)

Focus on: 1) Understanding the core regulatory pillars: SR 11-7's three lines of defense (LoD) and SS1/23's five principles. 2) Learning the model lifecycle (development, validation, implementation, monitoring) and key risk types (conceptual soundness, data, performance, implementation, ethical). 3) Mastering foundational terms: Model Inventory, Validation Report, Challenger Model, Conceptual Soundness Review.
Move to practice by: 1) Conducting end-to-end model risk assessments for specific use cases (e.g., credit scoring, fraud detection). 2) Drafting model validation reports that quantify weaknesses and propose actionable remediation. 3) Avoid common mistakes like conflating model validation with software testing or focusing solely on performance metrics while ignoring fairness and explainability.
Mastery involves: 1) Designing enterprise-wide MRM frameworks that align with business strategy and evolving regulations (e.g., EU AI Act). 2) Governing complex, opaque models (e.g., LLMs) through advanced techniques like SHAP/LIME, counterfactual analysis, and robust stress-testing. 3) Mentoring teams and influencing model owners on risk-based capital allocation and board-level risk appetite statements.

Practice Projects

Beginner
Project

Model Risk Assessment for a Hypothetical Loan Default Model

Scenario

You are given documentation for a simple logistic regression model used to predict mortgage defaults. The model uses income, debt-to-income ratio, and credit score.

How to Execute
1. Review the model documentation and identify the intended use, data sources, and key assumptions. 2. Perform a basic conceptual soundness review: assess variable selection, data quality, and target definition. 3. Check model performance using provided metrics (AUC-ROC, Gini) against a test dataset. 4. Draft a 2-page risk assessment memo highlighting one key strength and one key limitation (e.g., 'Relies on static features, may not capture recent economic shocks').
Intermediate
Case Study/Exercise

Challenging a Production Model: A Credit Decisioning Scenario

Scenario

A bank's live credit approval model is showing a 15% performance degradation on a recent vintage of loans. Business leadership is concerned. You are a validator tasked with investigating.

How to Execute
1. Acquire the latest production data and define the performance degradation precisely (e.g., lift in Gini coefficient, shift in score distribution). 2. Conduct a stability analysis (Population Stability Index - PSI) to detect data drift. 3. Build a challenger model (e.g., LightGBM) on the new data to benchmark performance. 4. Document findings in a validation report recommending specific actions (e.g., recalibration, full redevelopment, enhanced monitoring) with quantified risk impact.
Advanced
Case Study/Exercise

Governing a Fairness-Intensive Model: Hiring Algorithm Under Regulatory Scrutiny

Scenario

Your firm is deploying an ML model to screen job applicants. A regulator has flagged potential disparate impact on protected classes. The model is a complex ensemble method.

How to Execute
1. Define the fairness criteria relevant to the jurisdiction (e.g., demographic parity, equal opportunity) and assess the model against them using disparate impact ratios. 2. Implement explainability tools (SHAP) to audit decision drivers and identify if protected attributes are acting as proxies. 3. Develop a remediation plan that may include re-weighting training data, applying fairness constraints during training, or using adversarial debiasing techniques. 4. Present a risk control plan to the board, including ongoing fairness monitoring and a rollback protocol.

Tools & Frameworks

Regulatory & Governance Frameworks

SR 11-7 (Fed)SS1/23 (PRA)NIST AI RMFEU AI Act (High-Risk)ISO/IEC 42001

Apply SR 11-7/SS1/23 as the core governance blueprint for any MRM program. Use NIST RMF for a structured, lifecycle-based risk approach. Reference the EU AI Act for specific technical requirements (transparency, logging) for high-risk systems. ISO 42001 provides a certifiable management system standard.

Technical Risk Assessment Tools

SHAP/LIME (Explainability)Fairness Indicators / AI Fairness 360 (Fairness)Evidently AI / NannyML (Monitoring)Great Expectations (Data Validation)Open Source Libraries: Alibi Detect (Drift)

Use SHAP/LIME for model interpretability audits. Employ fairness toolkits to quantify and mitigate bias. Use monitoring platforms to track performance and data drift in production. Integrate data validation libraries into the ML pipeline to enforce data quality as a first line of defense.

Process & Documentation

Model Inventory System (e.g., in a GRC platform)Validation Report TemplateModel Risk Assessment (MRA) ChecklistChallenger Model Comparison Framework

Maintain a central inventory for oversight. Standardize validation reporting to ensure consistent communication of risk. Use checklists to ensure all risk facets (conceptual, data, performance, ethical) are assessed. Structure challenger model analysis to objectively evaluate incumbent models.

Interview Questions

Answer Strategy

Structure your answer around the SR 11-7 pillars: Conceptual Soundness, Ongoing Monitoring, and Outcomes Analysis. Mention data, performance, and compliance. Sample: 'I would start with a conceptual soundness review, examining the model's theory, data integrity, and variable selection. I'd then assess ongoing monitoring for data and concept drift using PSI and population stability metrics. Finally, I'd conduct outcomes analysis, comparing predicted vs. actual performance and performing fairness testing for disparate impact, ensuring all findings are documented for the second line.'

Answer Strategy

Tests ability to balance risk with business needs and apply proportionality. Use the concept of 'risk-based validation'. Sample: 'I would not block deployment but would advocate for a risk-based approach. I'd implement enhanced controls: a robust monitoring framework for performance and drift, a parallel running period with a simpler, interpretable model as a challenger, and mandatory explainability analysis (e.g., SHAP) to identify key drivers. I would document these interim controls in the model approval, with a firm commitment to a full validation within a defined timeframe.'

Careers That Require AI/ML Model Risk Management (SR 11-7 / SS1/23)

1 career found