Struttura del corso
Module 1 — Understanding How AI Applications Fail
Lab: None — Architecture walkthrough & discussion
Developing a builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from the developer’s perspective
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Points where untrusted data enters (and re-enters) the model
- Trust boundaries owned versus inherited by the developer
- Why AI attacks are semantic rather than syntactic
- Mapping the OWASP LLM Top 10 to your codebase
Key insight: Any point where untrusted text reaches the model, or model output reaches your code, represents a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The "SQL injection moment" for AI — acknowledging that while you can't fully escape it, you must manage it.
Topics:
- Differences between direct and indirect prompt injection
- Hidden instructions within documents, web pages, or tool output
- Jailbreaks and role-confusion tactics
- The importance of separating instructions from data
- Defensive prompt design (using delimiters, structure, and minimal authority)
- Why prevention is partial — designing for containment instead
Hands-on activities:
- Attack your own chatbot
- Bypass a naive filter
- Restructure the prompt to shrink the potential blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
Addressing the bug class developers often underestimate most.
Topics:
- Model output treated as untrusted input to the rest of the application
- Insecure output handling (LLM09): downstream XSS, SSRF, command/SQL injection risks
- Avoiding eval/exec/render on raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web/UI contexts
Hands-on activities:
- Identify and fix an insecure-output-handling vulnerability
- Enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
Tackling one of the most significant new attack surfaces — and recognizing it as your responsibility to secure.
Topics:
- Threats involving vector databases and retrieval mechanisms
- Ingestion sanitization techniques
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions within retrieved content (indirect injection)
- Data exfiltration via retrieval channels
Hands-on activity: Poison a RAG pipeline with a malicious document, then add ingestion sanitization and retrieval scoping to defend against it.
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
The transition point where a software bug becomes an actionable exploit.
Topics:
- Excessive agency (LLM10) and tool abuse risks
- Implementing least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop processes
- Sandboxing tool execution environments
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining behaviors
Hands-on activities:
- Lock down an over-permissioned agent
- Add an allowlist + approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost Management
Lab: Lab 05 — 05-Secrets-and-Cost
Addressing the operational mistakes that have the fastest and most severe impact.
Topics:
- API key and secret management (never store in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval systems
- Denial-of-wallet attacks: unbounded token/cost consumption risks
- Implementing rate limits, token budgets, and timeouts
- Logging practices that avoid leaking secrets or PII
Hands-on activities:
- Remove secrets from the prompt/code path
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Evaluating buy vs. build decisions for input/output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: detecting injection, PII, and topic violations
- Output guardrails: validation, filtering, and grounding checks
- When to use a guardrail versus your own deterministic checks
- Layering guardrails with controls from previous modules
- Performance impacts, false positives, and failure modes
Hands-on activities:
- Add an input/output guardrail layer to an AI feature
- Measure detection rates and missed incidents
Module 8 — Red-Teaming Your Own Application
Lab: Lab 07 — 07-Red-Teaming
Shipping your product as if an attacker has already found a way in.
Topics:
- Building abuse/test suites for AI features
- Automated prompt-injection and jailbreak testing
- Regression-testing guardrails and policies
- Integrating AI security checks into CI pipelines
- Model and dependency supply chain security (provenance, pinning)
- A pre-ship security checklist for AI features
Hands-on activities:
- Write automated red-team tests for an AI feature
- Wire them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — Scoring exercise (utilizing the Capstone application)
Converting your work into a repeatable security score.
Topics:
- The AI Security Hexagon: six critical questions instead of "is it secure?"
- Six scored categories: Data, Prompt, Agent, Supply Chain, Detection, Governance
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Using scoring before and after hardening as a metric for improvement
Hands-on activities:
- Score the Capstone application on the 100-point scale
- Identify the single change that most significantly raises the score
Key insight: The three highest-weighted categories map directly to the trust boundaries a developer owns — meaning the score measures exactly what this course has taught.
Capstone Project
Students systematically harden a deliberately vulnerable AI application from end-to-end.
The starter application contains:
- An injectable prompt structure
- Insecure output handling mechanisms
- An unscoped RAG pipeline
- An over-permissioned agent configuration
- Secrets embedded in the prompt path
- No cost limits implemented
Students apply course concepts to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Move secrets out of scope and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened application plus a short OWASP LLM Top 10 self-assessment.
Module to Lab Map
Labs run in sequential order, following the module sequence. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion, and Module 9 is a scoring exercise; therefore, neither has its own dedicated lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone application.
Requisiti
- Skill level: Intermediate.
- Students should be comfortable with: building and consuming REST APIs, scripting languages (labs utilize Python), basic application authentication, Git, and the Command Line Interface (CLI).
- No machine-learning background is required—this is an application-security course for developers who build with LLMs, not those who train them.
Target Audience
- Software and backend engineers building LLM-integrated features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Tech leads and senior engineers owning AI feature development
Recensioni (2)
Ho apprezzato molto aver appreso riguardo agli attacchi di intelligenza artificiale e agli strumenti disponibili per iniziare a praticare e utilizzare attivamente nei test di sicurezza. Ho acquisito molte conoscenze che non avevo all'inizio e il corso ha corrisposto alle mie aspettative. La parte che ho preferito della formazione è stata Comet Browser: mi ha sorpreso ciò che era in grado di fare. Sicuramente valuterò di approfondire l'argomento. Nel complesso è stato un corso ottimo e ho apprezzato moltissimo l'apprendimento del Top 10 OWASP GenAI.
Patrick Collins - Optum
Corso - OWASP GenAI Security
Traduzione automatica
La conoscenza professionale e il modo in cui l'ha presentata a noi
Miroslav Nachev - PUBLIC COURSE
Corso - Cybersecurity in AI Systems
Traduzione automatica
