Course Outline
Module 1 — Understanding How AI Applications Fail
Lab: None — Architecture walkthrough & discussion
Developing a builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from the developer’s perspective
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Points where untrusted data enters (and re-enters) the model
- Trust boundaries owned versus inherited by the developer
- Why AI attacks are semantic rather than syntactic
- Mapping the OWASP LLM Top 10 to your codebase
Key insight: Any point where untrusted text reaches the model, or model output reaches your code, represents a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The "SQL injection moment" for AI — acknowledging that while you can't fully escape it, you must manage it.
Topics:
- Differences between direct and indirect prompt injection
- Hidden instructions within documents, web pages, or tool output
- Jailbreaks and role-confusion tactics
- The importance of separating instructions from data
- Defensive prompt design (using delimiters, structure, and minimal authority)
- Why prevention is partial — designing for containment instead
Hands-on activities:
- Attack your own chatbot
- Bypass a naive filter
- Restructure the prompt to shrink the potential blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
Addressing the bug class developers often underestimate most.
Topics:
- Model output treated as untrusted input to the rest of the application
- Insecure output handling (LLM09): downstream XSS, SSRF, command/SQL injection risks
- Avoiding eval/exec/render on raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web/UI contexts
Hands-on activities:
- Identify and fix an insecure-output-handling vulnerability
- Enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
Tackling one of the most significant new attack surfaces — and recognizing it as your responsibility to secure.
Topics:
- Threats involving vector databases and retrieval mechanisms
- Ingestion sanitization techniques
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions within retrieved content (indirect injection)
- Data exfiltration via retrieval channels
Hands-on activity: Poison a RAG pipeline with a malicious document, then add ingestion sanitization and retrieval scoping to defend against it.
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
The transition point where a software bug becomes an actionable exploit.
Topics:
- Excessive agency (LLM10) and tool abuse risks
- Implementing least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop processes
- Sandboxing tool execution environments
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining behaviors
Hands-on activities:
- Lock down an over-permissioned agent
- Add an allowlist + approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost Management
Lab: Lab 05 — 05-Secrets-and-Cost
Addressing the operational mistakes that have the fastest and most severe impact.
Topics:
- API key and secret management (never store in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval systems
- Denial-of-wallet attacks: unbounded token/cost consumption risks
- Implementing rate limits, token budgets, and timeouts
- Logging practices that avoid leaking secrets or PII
Hands-on activities:
- Remove secrets from the prompt/code path
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Evaluating buy vs. build decisions for input/output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: detecting injection, PII, and topic violations
- Output guardrails: validation, filtering, and grounding checks
- When to use a guardrail versus your own deterministic checks
- Layering guardrails with controls from previous modules
- Performance impacts, false positives, and failure modes
Hands-on activities:
- Add an input/output guardrail layer to an AI feature
- Measure detection rates and missed incidents
Module 8 — Red-Teaming Your Own Application
Lab: Lab 07 — 07-Red-Teaming
Shipping your product as if an attacker has already found a way in.
Topics:
- Building abuse/test suites for AI features
- Automated prompt-injection and jailbreak testing
- Regression-testing guardrails and policies
- Integrating AI security checks into CI pipelines
- Model and dependency supply chain security (provenance, pinning)
- A pre-ship security checklist for AI features
Hands-on activities:
- Write automated red-team tests for an AI feature
- Wire them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — Scoring exercise (utilizing the Capstone application)
Converting your work into a repeatable security score.
Topics:
- The AI Security Hexagon: six critical questions instead of "is it secure?"
- Six scored categories: Data, Prompt, Agent, Supply Chain, Detection, Governance
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Using scoring before and after hardening as a metric for improvement
Hands-on activities:
- Score the Capstone application on the 100-point scale
- Identify the single change that most significantly raises the score
Key insight: The three highest-weighted categories map directly to the trust boundaries a developer owns — meaning the score measures exactly what this course has taught.
Capstone Project
Students systematically harden a deliberately vulnerable AI application from end-to-end.
The starter application contains:
- An injectable prompt structure
- Insecure output handling mechanisms
- An unscoped RAG pipeline
- An over-permissioned agent configuration
- Secrets embedded in the prompt path
- No cost limits implemented
Students apply course concepts to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Move secrets out of scope and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened application plus a short OWASP LLM Top 10 self-assessment.
Module to Lab Map
Labs run in sequential order, following the module sequence. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion, and Module 9 is a scoring exercise; therefore, neither has its own dedicated lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone application.
Requirements
- Skill level: Intermediate.
- Students should be comfortable with: building and consuming REST APIs, scripting languages (labs utilize Python), basic application authentication, Git, and the Command Line Interface (CLI).
- No machine-learning background is required—this is an application-security course for developers who build with LLMs, not those who train them.
Target Audience
- Software and backend engineers building LLM-integrated features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Tech leads and senior engineers owning AI feature development
Testimonials (2)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us
