AI Application Security

An AI application combines a model with ordinary software, data sources, prompts, retrieval systems, tools, identities, and infrastructure. AI application security protects that whole system. Securing only the model leaves the most important trust boundaries unexamined.

The useful mental model is a probabilistic component inside a deterministic security envelope. The model can interpret language and generate useful responses. It can also misunderstand instructions, follow hostile content, disclose context, or produce unsafe output. Code around the model must enforce the decisions that require certainty.

users and external content
           ↓
authentication → application policy → context builder
                                      ↓
data stores → retrieval controls → model → output controls
                                      ↓
                              tool authorization
                                      ↓
                              downstream systems

Every arrow crosses a boundary. Ask who controls the input, what authority moves across the boundary, and what happens if the receiving component treats data as instructions.

Why the security model changes

Conventional application security still applies. You need secure authentication, authorization, dependency management, secrets handling, isolation, patching, logging, and incident response. AI adds failure modes because model behavior depends on data expressed in the same language as instructions.

A web page retrieved for summarization is data to the application. To the model, a sentence inside that page can look like a new instruction. A generated string is output from the model. To a browser, shell, database, or tool, that string may become executable input. This blurring of code, instruction, and data creates new paths through otherwise familiar systems.

Models are also nondeterministic. One successful test does not prove that a nearby prompt, a longer conversation, a different document, or a model update will behave the same way. Security testing therefore needs scenario sets, repeated trials where useful, measurable outcomes, and regression baselines.

Start with the application, not the prompt

An application threat model should include at least these assets and boundaries:

user prompts, conversation history, and uploaded files;
system prompts, templates, policies, and model configuration;
training, fine-tuning, evaluation, and retrieval data;
model files, adapters, embeddings, and software dependencies;
model-provider and retrieval APIs;
vector stores, databases, caches, logs, and feedback pipelines;
tools that read data or cause side effects;
identities, credentials, approval paths, and downstream permissions;
generated text, code, queries, media, and structured data;
budgets, quotas, latency targets, and recovery mechanisms.

Map data provenance and authority separately. A retrieved document may be trusted for factual content but never authorized to change tool policy. A user may be allowed to ask about a record but not to make the model retrieve records from another tenant. A model may suggest an action but not have permission to approve it.

Prompt injection is a control-flow problem

Prompt injection occurs when input changes model behavior in an unintended way. A direct injection arrives through a user prompt. An indirect injection arrives through content such as a web page, email, file, image, tool result, or retrieved document.

Prompt injection matters because the model does not provide a dependable separation between instructions and untrusted data. Retrieval-augmented generation and fine-tuning can improve relevance, but OWASP does not treat either as a complete prevention method.

You can reduce likelihood with input constraints, instruction structure, content filtering, and model-level safeguards. You reduce impact with stronger architectural controls:

keep authorization and business rules in code;
give the application narrowly scoped credentials;
separate read tools from write tools;
require approval for high-impact actions;
label and isolate untrusted context;
constrain tool arguments with typed schemas and allowlists;
limit requests, tokens, recursion, time, and cost;
log decisions and retain enough context for investigation.

A system prompt is not a secret boundary. Do not put credentials, connection strings, private authorization logic, or sensitive user data in it. Assume an attacker can infer much of its behavior through interaction, even if the exact text never appears.

Retrieval adds a data authorization layer

Retrieval-augmented generation, often called RAG, selects external content and adds it to model context. It can make answers more relevant, but it also creates a path from indexed content to generated output.

Security decisions belong before retrieval and again before disclosure:

authenticated subject
        ↓
authorized corpus and tenant filter
        ↓
retrieval and ranking
        ↓
context with provenance labels
        ↓
model generation
        ↓
output policy and disclosure check

Similarity is not authorization. A vector search result can be semantically relevant and still belong to the wrong user, tenant, classification, or retention scope. Apply access controls to source records and retrieval filters. Preserve document identity and provenance so you can explain which content influenced a response.

Treat indexed content as an input supply chain. Control who can add or update documents. Validate sources, record transformations, detect unusual changes, and rebuild indexes from known data when integrity is in doubt. Retrieval content can carry indirect prompt injections or poisoned facts even when the embedding store itself works correctly.

Model output is untrusted input

The model's response is not safe because your application generated it. It may contain attacker-influenced HTML, Markdown, URLs, SQL, shell fragments, tool arguments, or code.

Apply the same boundary controls used for any other untrusted input:

parse structured output against a strict schema;
reject unknown fields and invalid values;
encode text for the destination context;
parameterize database operations;
allowlist destinations, operations, and resource identifiers;
sandbox generated code or avoid executing it;
present high-impact actions for review before execution;
bind authorization to the current user and requested resource.

Do not ask the model to validate its own authority. The component that owns the resource must make the final authorization decision.

Agency multiplies impact

An AI system has agency when it can call tools or otherwise act on connected systems. Risk rises with three independent dimensions:

Dimension	Question	Safer direction
Functionality	What can the tool do?	Expose only required operations
Permission	Which resources can it affect?	Use the current user's narrow scope
Autonomy	Can it act without review?	Add approval at consequential boundaries

A read-only email assistant should not receive a tool that can send or delete mail. A tool acting for one user should not use a shared administrator identity. A payment draft and a payment execution should be separate operations with different authorization and approval rules.

Human approval is not a decorative confirmation box. Show the proposed action, target, important arguments, expected effect, and relevant source context. Make the approval occur after arguments are resolved and before the side effect.

Protect the AI supply chain

AI applications depend on more than packages. The supply chain can include base models, adapters, datasets, evaluation sets, prompt templates, model-serving images, tokenizers, embedding models, and third-party APIs.

Record source, version, integrity evidence, license, owner, intended use, and evaluation status. Verify model and data artifacts before ingestion. Restrict formats that can execute code during loading. Protect build and training environments. Track transformations so a deployed artifact can be traced back to its inputs.

Data poisoning targets integrity. An attacker may alter training, fine-tuning, evaluation, or retrieval data to change behavior. Provenance does not prove that content is true or benign, but it gives you evidence for verification, rollback, and investigation.

Protect sensitive data through the lifecycle

Sensitive data can enter through prompts, files, retrieval, training, fine-tuning, logs, traces, caches, feedback, and model outputs. Decide what data the system may receive before connecting it to a model service.

Use data classification, minimization, access control, encryption, retention limits, and deletion procedures. Separate tenants throughout storage and retrieval. Redact secrets and unnecessary personal data before model calls and before logging. Review provider terms and technical controls for data use, retention, regional processing, and model improvement.

Logs need their own threat model. They are essential for detection and investigation, but full prompts and outputs can become a second sensitive data store. Capture the minimum evidence needed, restrict access, and test that redaction works.

Availability includes cost

Model inference consumes variable compute. Input length, output length, repeated calls, tool loops, and agent recursion can turn one request into many expensive operations. OWASP groups uncontrolled inference under unbounded consumption.

Set limits per user, tenant, workflow, and time window. Bound context size, output size, tool iterations, concurrent calls, and total cost. Add timeouts, circuit breakers, cancellation, and backpressure. Monitor both technical saturation and financial spend. A service can remain online while an attacker exhausts its budget.

Build a security lifecycle

AI security is continuous because models, prompts, data, retrieval corpora, tools, and attacker techniques change.

Govern. Assign ownership, acceptable use, risk tolerance, supplier requirements, and incident authority.
Map. Document the use case, users, assets, data flows, trust boundaries, possible impacts, and shared responsibilities.
Measure. Test model behavior, application integration, infrastructure, data controls, and runtime abuse against defined outcomes.
Manage. Prioritize findings, select controls, make release decisions, monitor production, respond to incidents, and retire unsafe components.

Use threat scenarios from OWASP and MITRE ATLAS to seed testing, then tailor them to your data and authority paths. Include direct and indirect prompt injection, cross-tenant retrieval, poisoned content, unsafe rendering, excessive tool permissions, sensitive-data extraction, denial of service, and model or dependency tampering.

Test controls, not just model refusal. A useful result might show that a malicious document changes the model's words but cannot expand retrieval scope, invoke an unauthorized tool, or bypass approval. Record model and application versions, prompts, context, expected outcomes, actual outcomes, and control evidence. Repeat the suite after changes.

Production monitoring should cover authentication, retrieval decisions, tool calls, approvals, validation failures, policy denials, token and cost usage, latency, model changes, and anomalous output patterns. Prepare containment actions such as disabling a tool, blocking a model version, isolating an index, revoking credentials, reducing autonomy, or rolling back to a known state.

What AI application security cannot promise

No prompt, filter, guardrail, or red-team campaign proves that a general-purpose model will never produce an unwanted response. Benchmarks describe tested conditions, not every future context. A safer design assumes some model failures will pass through and limits what those failures can reach.

Security also does not prove factual accuracy, fairness, legality, or fitness for a decision. These concerns overlap with security and belong in the wider risk program. High-impact uses need domain validation, accountable decision owners, and a clear path to override or decline the model.

A practical adoption path

Inventory every model, provider, dataset, retrieval source, tool, and identity.
Draw the data and authority flows across the full application.
Identify the highest-impact side effect or disclosure the model could influence.
Move its authorization, validation, and approval into deterministic code.
Reduce tool functionality, permissions, autonomy, and resource budgets.
Add provenance and integrity checks for models, data, and configuration.
Build adversarial tests from real trust boundaries and expected control outcomes.
Deploy monitoring, containment, rollback, and incident procedures.
Reassess after model, prompt, data, tool, or provider changes.