Prompt injection

Prompt injection is an attack where instructions hidden inside content a model reads cause it to behave as the attacker intends. Because a model cannot reliably separate instructions written by the user from text embedded in retrieved documents, any channel that feeds the index becomes a potential attack surface.

It is underestimated because it does not resemble a traditional exploit. There is no malformed input and no crash. A document in your own knowledge base simply contains text addressed to the model rather than to a reader.

There is no single fix. The workable mitigations are layered: constrain what the system can do rather than only what it can read, enforce authorization at retrieval so injected instructions cannot widen access, filter output as well as input, and keep a human in the loop for consequential actions.