Skill Guide

Threat modeling for generative AI architectures and RAG systems

A systematic process to identify, evaluate, and prioritize security threats specific to AI systems that generate content and use retrieval-augmented generation (RAG) to ground responses in external data.

It directly protects against data poisoning, model inversion, and system compromise in high-value AI applications. Proactive threat modeling reduces breach risk and compliance costs while maintaining user trust in AI-driven products.

1 Careers

1 Categories

9.1 Avg Demand

15% Avg AI Risk

How to Learn Threat modeling for generative AI architectures and RAG systems

Focus on foundational AI security concepts: 1) Master the OWASP Top 10 for LLM Applications. 2) Understand the core components of a RAG pipeline (retriever, vector DB, generator) and their inherent risks. 3) Learn the DREAD or STRIDE threat classification models.

Transition to practice by conducting threat modeling sessions on specific architectures. Analyze common vulnerabilities like prompt injection via retrieved context, embedding model poisoning, and insecure API endpoints. A key mistake is focusing only on the LLM, ignoring the entire data supply chain.

Master the skill by architecting security controls into the CI/CD pipeline for AI systems. Develop risk quantification models for AI-specific threats and lead cross-functional mitigation workshops. Align threat models with business risk tolerance and regulatory frameworks like the EU AI Act.

Practice Projects

Beginner

Project

Threat Model a Simple Chatbot with RAG

Scenario

You are given a Python script using LangChain that connects a basic chatbot to a PDF knowledge base via a vector store. The goal is to produce a threat model document.

How to Execute

1) Diagram the data flow: user input -> retriever -> vector DB -> context -> LLM -> response. 2) For each component, apply the STRIDE model (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege). 3) Prioritize threats by likelihood and impact (e.g., malicious document poisoning the vector store is high impact). 4) Propose a mitigation for each high-priority threat (e.g., input validation, chunk sanitization).

Intermediate

Case Study/Exercise

Red Team Exercise on a Knowledge Base

Scenario

A healthcare company uses a RAG system to answer clinician queries from internal research papers. An adversarial actor has planted a subtly misleading but plausible document in the source corpus.

How to Execute

1) Simulate the attack: Craft queries designed to retrieve the poisoned document. 2) Evaluate the LLM's response for factual accuracy and potential harm. 3) Analyze the system's failure points: no provenance tracking, lack of document freshness scoring, weak embedding robustness. 4) Draft a remediation plan including document provenance checks and retrieval result re-ranking.

Advanced

Project

Enterprise RAG Architecture Security Review

Scenario

Lead the threat modeling for a financial services firm's customer-facing RAG assistant that queries multiple internal and licensed external data sources in real-time.

How to Execute

1) Map the complex data supply chain, including third-party API integrations and access control layers. 2) Conduct a workshop using the PASTA (Process for Attack Simulation and Threat Analysis) methodology to align technical threats with business objectives. 3) Design a layered defense: input sanitization, output filtering, real-time anomaly detection on retrieval patterns, and strict API gateway policies. 4) Develop a security testing playbook for the QA team, including fuzzing the retriever and testing for data exfiltration.

Tools & Frameworks

Mental Models & Methodologies

STRIDEDREADPASTAOWASP Top 10 for LLM

STRIDE/DREAD for component-level threat identification and prioritization. PASTA for business-aligned risk analysis. The OWASP list provides a consensus baseline of specific vulnerabilities for LLM applications.

Technical Tooling & Platforms

LangChainGuardrails AIRagasPromptfoo

LangChain's ecosystem includes tools for testing. Guardrails AI and Ragas are used to define output validation rules and evaluate RAG pipeline safety/quality. Promptfoo is an open-source tool for red-teaming prompts and measuring attack success rates.

Interview Questions

Answer Strategy

Use a structured framework like STRIDE to break down the system. Focus on the unique interaction between user-controlled data and the retrieval/generation pipeline. Sample answer: 'I would apply STRIDE to the data ingestion, storage, retrieval, and generation layers. My top three concerns are: 1) Tampering of the source documents to manipulate financial summaries, a data poisoning attack. 2) Information Disclosure via the LLM revealing private document content not meant for the user. 3) Denial of Service through maliciously crafted documents that crash the ingestion or embedding process.'

Answer Strategy

The interviewer is testing for proactive, deep analytical thinking beyond checklist security. The answer should demonstrate an understanding of AI-specific attack surfaces. Sample answer: 'In a system using a semantic search retriever, I identified a risk where an attacker could craft queries to systematically exfiltrate the entire embedding model's representation of the knowledge base through repeated, careful probing. This went beyond data leakage to model intellectual property theft. The mitigation involved adding query rate limiting and implementing differential privacy techniques on the retrieval results.'