Data Leakage & Poisoning

Extraction. With query access alone, an adversary can attempt to elicit memorized content, and research has repeatedly shown that portions of training data can be recovered this way. The related membership inference question, "was this specific record in the training set?", matters on its own when membership is itself sensitive, for example whether a particular patient record was used.

PII in outputs. Personal data reaches outputs by two paths. It can be regurgitated from training if PII was present in the corpus, or it can be echoed from inference time when a user pastes personal data into a prompt and it later resurfaces, in that same session, in logs, or, worst of all, to a different user through a shared cache or cross-tenant contamination.

Leakage via logs and prompts. The prompt itself is often sensitive: system prompts embed internal instructions and sometimes credentials, and user prompts carry whatever the user typed. That content is routinely copied into request logs, analytics, error traces, and third-party telemetry, and, critically, into feedback datasets used for future fine-tuning. Each copy is a place the data can escape.

Sensitive data in a vector store. A RAG index stores chunks of source text plus metadata. Two failures dominate. First, broken authorization at retrieval: if the query does not filter by the requesting user's permissions, someone can retrieve chunks they were never allowed to read. Second, the embeddings are not opaque, embedding-inversion research shows that text can be partially reconstructed from its vector, so the index must be treated as sensitive data, not as a harmless numeric cache.

Poisoning and backdoors

Training-data poisoning. An adversary injects crafted examples into the training or fine-tuning corpus. The goal may be availability (degrade overall accuracy) or, more dangerously, integrity (install a specific targeted behavior). Web-scale scraping makes injection realistic: content can be placed where crawlers will find it, and studies have shown that controlling even a tiny fraction $p \ll 1\%$ of a corpus can be enough for a targeted effect.

Backdoors and triggers. A backdoor is a hidden association the model learns between a trigger, a rare token, phrase, or stylistic pattern, and an attacker-chosen behavior. The model behaves normally on clean inputs and flips only when the trigger appears. This is what makes backdoors so hard to catch: clean-input accuracy is unchanged, so standard evaluation on a held-out set looks perfectly healthy. The canonical demonstration, BadNets, showed this in image classifiers, and the same principle applies to language models.

RAG poisoning. Here the weights are never touched. The attacker plants content in a source that the ingestion pipeline trusts, a public wiki, a shared drive, a scraped site, and writes it to be semantically close to the questions real users will ask. Retrievers rank candidate chunks by embedding similarity, most often cosine similarity:

$\operatorname{sim}(q, d) = \frac{q \cdot d}{\lVert q \rVert\,\lVert d \rVert}$

so content engineered to sit near likely queries in vector space will be retrieved and dropped into the context window, where the model tends to treat it as authoritative. If the payload is a false fact, the model repeats misinformation; if the payload is an instruction, RAG poisoning becomes a delivery vehicle for indirect prompt injection.

Defenses across the data supply chain

No single control is sufficient; each stage of the pipe needs its own guard, and the earliest effective control is almost always the cheapest.

• Data provenance and vetting. Know where every training document and every RAG source came from. Maintain source allowlists, prefer signed or curated sources over world-writable ones, and record lineage so a bad record can be traced and removed.
• PII scrubbing and redaction. Detect and redact personal data at three points, before training, before logging, and before returning output. Combine pattern matching with named-entity detection, and design for the reality that detectors have false negatives.
• Output filtering. Scan generations for PII, secrets, and known-memorized strings before they reach the user. This is defense in depth, a backstop rather than a primary control.
• Access control on retrieval sources. Enforce authorization at retrieval time with per-user and per-tenant metadata filtering and document-level ACLs. Never rely on instructing the model "do not reveal restricted content", authorization belongs in the data layer, not in a prompt.
• Dataset integrity checks. Hash and version datasets, deduplicate, run outlier and anomaly detection on new records, and verify the provenance and checksums of any external dataset or pretrained weights before use. Canary or known-answer probes help detect drift and tampering over time.

Worked example: tracing a RAG-poisoning incident

Consider a customer-support assistant that answers from a knowledge base. Among its curated sources, the ingestion job also pulls from a community wiki that any user can edit. An attacker adds a page written to match common billing questions, and embeds a payload, a false instruction telling users to send account details to an external address. Follow the request through the pipeline and note where each control can break the chain.

1. 11. IngestionThe connector pulls the editable wiki page into the pipeline. Control point: provenance and source vetting. If a world-writable source sits on the ingestion allowlist as if it were curated, the incident begins here, the cheapest place to stop it.
2. 22. Sanitize and chunkText is cleaned and split into chunks. Control point: content sanitization, secret and PII scanning, and anomaly detection on new documents can flag or quarantine the payload before it is ever embedded.
3. 33. Embed and indexEach chunk becomes a vector stored with metadata. Control point: tag every chunk with a source trust tier and tenant scope so later stages have something to filter on.
4. 44. RetrievalA user question is embedded and the poisoned chunk ranks highly because it was written to match the query semantically. Control point: retrieval-time access control and trust-tier filtering keep low-trust sources out of the candidate set.
5. 55. RerankCandidates are reordered before selection. Control point: a reranker that down-weights low-trust or recently edited sources reduces the chance the payload reaches the context window.
6. 66. Context assemblyTop chunks are concatenated into the prompt. Control point: label provenance, keep retrieved text in a data channel, and never let it override system instructions or silently trigger tools.
7. 77. Generation and responseThe model may repeat the false content or act on the embedded instruction. Control point: output filtering plus inline citations let the answer be checked against its sources, and any high-risk action requires human confirmation.

The lesson of the trace is defense in depth: the payload has to survive every stage to cause harm, so even imperfect controls compound. Vetting at ingestion is the strongest and cheapest guard, but trust tiers, retrieval-time access control, and citations each give an independent chance to break the chain.

Threats and defenses at a glance

Further reading

• OWASP Top 10 for Large Language Model Applications, includes Sensitive Information Disclosure and Data and Model Poisoning.
• NIST AI Risk Management Framework (AI RMF 1.0), governance framework for identifying and managing these risks.
• NIST AI 100-2: Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations, authoritative taxonomy covering poisoning, extraction, and evasion.
• MITRE ATLAS, knowledge base of adversary tactics and techniques against AI systems, with mitigations.
• Carlini et al., "Extracting Training Data from Large Language Models" (arXiv:2012.07805), foundational study of memorization and extraction.
• Gu et al., "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain" (arXiv:1708.06733), the canonical demonstration of backdoor attacks.