Loading lesson…
AI Systems Engineering: Run, Operate & Evaluate / Operating & Evaluating
Defending LLM systems: injection, agency, disclosure.
Reviewed by Yuvaraj
Connecting a language model to real tools, real data, and real users creates real attack surface. This lesson names the main risk classes for LLM-powered systems and, importantly, how to defend against each. It is deliberately defensive: the goal is to build systems that hold up, not to hand anyone a recipe for breaking them.
Framing
Security here means designing for the worst-case input. We describe what can go wrong at the level a defender needs to build controls, and spend most of the space on mitigations. No step-by-step exploits.
A classical program separates code from the data it processes. An LLM does not: everything is text in one context window, so content that looks like an instruction can be treated like one, whoever wrote it. Almost every LLM security risk descends from this single fact. Internalize it and the rest follows.
What it is: untrusted text (a web page the model browses, a document it summarizes, a user message) contains instructions that hijack the model's behavior, overriding what the developer intended. Indirect injection is the dangerous variant: the malicious text arrives through data the model consumes, not from the user typing it.
Defenses (layered, no single one is sufficient):
Answer from memory before revealing, retrieval practice is what builds durable recall.
Which is the most reliable defense against prompt injection?
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
What it is: the system can do more than it safely should, too many tools, too broad permissions, too much autonomy, so a single bad decision (from injection or plain model error) causes real damage.
Defenses: scope each tool narrowly; prefer read-only access where possible; gate write and destructive operations behind explicit authorization and human-in-the-loop approval; log every action so it can be audited and reversed.
What it is: the system reveals data it should not, secrets in a system prompt, another user's data pulled into a shared context, or private information the model memorized during training.
Defenses: never place secrets (API keys, credentials) in prompts, keep them server-side and out of any model-visible text; enforce per-user data isolation in retrieval so one user's query can never surface another's documents; filter outputs for sensitive patterns; minimize what enters the context in the first place.
What it is: a tool the agent calls is itself dangerous, an unsandboxed code executor, a database tool that permits arbitrary queries, or a component you depend on (a model, a plugin, a package) is compromised or untrustworthy.
Defenses: sandbox any code execution with strict isolation (never run model-generated code on your application server unrestricted); constrain tools to safe, parameterized operations rather than open-ended ones; vet and pin the provenance of models and dependencies.
The defensive mindset
Assume any text entering the context may be adversarial, and assume the model may be talked into misusing any capability it has. Then put your real security boundary in deterministic code around the model, authorization, validation, sandboxing, human approval, not in the prompt. Prompts guide behavior; they do not enforce it.
What it is: corrupted data introduced during training or fine-tuning, or, for RAG, into the knowledge base, skews the system's behavior. RAG poisoning is the accessible version: plant misleading content where retrieval will find it.
Defenses: control and review what enters training sets and knowledge bases; track data provenance; monitor for anomalous inputs and outputs; treat your retrieval corpus as a security boundary, not a dumping ground.
Fast-moving, but principles endure
Specific attack techniques and the tooling around them evolve quickly, and no checklist stays complete. The durable defenses do not change: least privilege, isolate untrusted input, validate consequential actions outside the model, sandbox execution, keep humans in the loop for high-stakes steps, and log everything. Consult current authoritative guidance for the latest specifics.