Understanding Model Errors, Safety Failures, Privacy Failures, and Security Attacks

Hello. In the previous lesson, you classified an agent by the authority it has to act without a person’s approval. That approval boundary also shapes the consequences of failure: a mistaken answer is one thing; a mistaken answer that triggers a tool call can become an operational incident.

This lesson gives you a vocabulary for diagnosing what went wrong. You will distinguish model errors, safety failures, privacy failures, and security attacks. These categories overlap in real incidents, but they answer different questions about cause, impact, and accountability.

---

Start with the right diagnostic question

When an agent behaves badly, teams often use vague labels such as “the AI failed” or “it was hacked.” Those phrases obscure the decision that matters next: what must be fixed?

A useful first-pass diagnosis asks four different questions:

1. Was the system wrong?
   This suggests a model error or another reliability problem.

2. Did the system create an unacceptable risk of harm?
   This suggests a safety failure.

3. Was personal or sensitive information collected, inferred, used, or disclosed improperly?
   This suggests a privacy failure.

4. Did someone deliberately try to bypass protections, manipulate behavior, or gain unauthorized access?
   This suggests a security attack.

These are not mutually exclusive labels. A malicious prompt can induce an agent to disclose customer records. That is both a security attack in cause and a privacy failure in impact. If the agent then takes an unsafe action on the basis of manipulated information, it may also become a safety failure.

The key distinction is:

Model error, safety failure, and privacy failure primarily describe undesirable system behavior or impact. Security attack describes deliberate adversarial action.

---

Four categories, four lenses

The table below is a practical working definition for this course.

Category	Core question	Typical cause	Example in an agentic system	Main property at stake
Model error	Is the result incorrect, unreliable, or poorly grounded?	Statistical limitation, missing context, poor generalization, extraction failure	The agent invents a contract renewal date not found in the documents	Validity and reliability
Safety failure	Could the system endanger people, property, or another protected interest?	Unsafe requirements, inadequate constraints, poor handoff, error, misuse	A workflow agent authorizes an operational action despite a hazardous condition	Safety
Privacy failure	Was information about a person handled contrary to appropriate notice, consent, purpose, or access limits?	Excessive collection, retention, inference, sharing, or disclosure	A support agent exposes one employee’s case history to another	Privacy
Security attack	Did an actor intentionally attempt unauthorized influence, access, disclosure, alteration, or disruption?	Malicious or unauthorized action by an outsider or insider	An attacker embeds instructions in a document to manipulate a retrieval-enabled agent	Confidentiality, integrity, availability, and authorization

A category is not determined by whether the result looks alarming. It depends on the evidence.

For example, a support agent giving an incorrect refund-policy answer is a model error. If a user deliberately altered the agent’s knowledge source so that it would give that answer, there is also a security attack involving integrity. If the incorrect answer causes the agent to reveal a customer’s transaction history, privacy has also been harmed.

---

Reliability, safety, security, and privacy are distinct properties

NIST’s trustworthiness framework treats validity and reliability, safety, security, and privacy as related but distinct characteristics. That distinction is valuable because an agent can satisfy one property while failing another.

Consider four brief examples:

• An agent incorrectly states that a supplier is certified. The answer is false, even though no one tampered with the system. This is a model error.
• An agent controlling warehouse scheduling routes equipment through an area reserved for pedestrians because the workflow contains no safety constraint. It may have read the location data correctly, but its behavior is unsafe. This is a safety failure.
• A wellness assistant retains detailed health disclosures indefinitely and reuses them to personalize unrelated marketing suggestions without an appropriate basis. No attacker is needed. This is a privacy failure.
• An attacker submits manipulated content designed to override an agent’s instructions and cause it to export data. This is a security attack, whether or not the attempt succeeds.

A useful contrast: correctness is not enough

Correct output is not necessarily safe or private.

An agent may accurately retrieve an employee’s medical accommodation and still disclose it to a manager who has no legitimate need to know. The content is correct; the privacy handling is not.

Likewise, an agent may correctly recognize that a customer account is overdue but automatically send an aggressive message that violates policy or causes foreseeable harm. The factual classification can be accurate while the resulting action is unsafe.

Conversely, an agent can be wrong without being unsafe. A harmlessly incorrect summary of a public article is still a reliability issue, but it may not rise to a safety, privacy, or security incident.

---

Model errors: plausible output is not verified truth

A model error occurs when a model or model-powered system does not perform correctly for its intended use. For language models, the familiar example is a confident false statement, often called a hallucination. NIST uses the more neutral term confabulation: erroneous or false content presented with confidence.

The underlying issue is that a generative model produces likely continuations, not guaranteed facts. It can be helpful, fluent, and often accurate while still filling a missing fact with a plausible invention.

For an agent, model errors can arise at several points:

• Answer generation: It invents a policy, citation, deadline, or identity.
• Information extraction: It misreads a table, merges fields from two people, or misses a qualifier such as “draft.”
• Planning: It chooses an inappropriate sequence of otherwise permitted steps.
• Tool selection: It calls a search or update tool when the task required a different tool.
• Interpretation: It treats ambiguous language as a firm authorization.

A grounded system can reduce these errors. Retrieval from an authoritative repository, deterministic calculations, schema validation, and source citations all help. But they do not guarantee correctness: retrieved content may be outdated, incomplete, irrelevant, or itself maliciously manipulated.

A security-relevant principle follows:

The model may propose a conclusion or action, but deterministic controls should decide whether an action is authorized and safe to execute.

This is especially important at higher autonomy levels. A false sentence in a chat response may be caught by a reader. A false premise in a multi-step agent workflow can lead to record changes, notifications, purchases, or other tool actions before anyone notices.

---

Safety failures: harm can result even without an attacker

Safety is about preventing a system from creating dangerous states or unacceptable harm under defined conditions. In high-consequence settings, that may involve risks to life, health, property, or the environment. An organization may also define safety boundaries for serious psychological, financial, or operational harms.

A safety failure is not simply an incorrect answer. It is a failure of the system and its surrounding process to keep behavior within safe bounds.

Suppose a facilities agent receives the objective, “Reduce energy use this weekend.” It correctly identifies unoccupied areas, but it shuts down environmental controls for a laboratory where stored materials require temperature stability. The agent may not have hallucinated anything. The failure lies in unsafe objectives, missing constraints, insufficient tool policy, or lack of escalation.

Safety failures commonly involve one or more of the following:

• requirements that optimize efficiency without defining prohibited outcomes;
• an inadequate understanding of operating conditions;
• missing thresholds, interlocks, or emergency stops;
• automation bias, in which people accept the agent’s recommendation without meaningful review;
• failure to escalate uncertainty or conflicting evidence;
• a model error that becomes harmful because a system executed it unchecked.

The distinction between safety failure and security attack is particularly important:

• A safety failure can occur through ordinary operation, design weakness, or human error. No attacker is necessary.
• A security attack involves intentional adversarial behavior. It may target safety, but it might instead target data confidentiality or system availability.
• A deliberate attempt to manipulate an industrial agent into unsafe action is both a security attack and, if harmful behavior occurs, a safety incident.

---

Privacy failures: more than data theft

Privacy concerns the appropriate handling of information connected to people: their identity, autonomy, dignity, and ability to control disclosure of personal details. A privacy failure can involve collection, retention, use, inference, sharing, or disclosure that is inappropriate for the context.

This is broader than “someone hacked the system.”

An agent may create a privacy failure even when every involved employee used valid credentials. For example:

• A meeting assistant stores sensitive personal remarks in long-term memory when the user expected a one-time summary.
• A recruiting agent infers a candidate’s health status from scattered public information and includes it in an internal profile.
• A customer-support agent retrieves the full history of a household account when the current requester should see only their own records.
• An analytics agent uses support transcripts for a purpose incompatible with the reason customers originally provided them.

NIST’s Generative AI Profile highlights that models may reveal, generate, or infer sensitive information. It also distinguishes the security of AI systems from privacy risks arising from sensitive-data exposure and inference.

Privacy versus confidentiality

The terms are related, but not identical.

Confidentiality is a security property: information should not be disclosed to unauthorized parties. A database breach is a confidentiality failure, usually caused by a security attack.

Privacy is about appropriate treatment of information about people. It includes confidentiality, but also questions of purpose limitation, consent, proportionality, inference, and retention. An organization can create a privacy failure by using information inappropriately even when access was technically authorized.

In incident analysis, therefore, do not stop at “the user had permission.” Ask whether the agent’s access, data minimization, and use of information were appropriate for the task.

---

Security attacks: intentional attempts to defeat boundaries

A security attack is an intentional attempt to violate or bypass a system’s security properties, including confidentiality, integrity, availability, and authorization.

For agentic systems, common targets include:

• the agent’s instructions and context;
• retrieval documents and knowledge bases;
• tools, APIs, credentials, and access tokens;
• memory stores and conversation history;
• model endpoints, weights, training data, and deployment infrastructure;
• users who can be persuaded to approve an unsafe action.

A few examples clarify the boundary:

Event	Primary classification	Why
A model invents a nonexistent invoice number when asked to summarize a file	Model error	Incorrect behavior occurs without evidence of deliberate manipulation
A staff member accidentally uploads an outdated policy to the knowledge base	Reliability or process failure	The data is wrong, but error alone is not an attack
An adversary modifies a knowledge-base document to steer future answers	Security attack	The attacker deliberately targets integrity
A malicious web page contains hidden instructions intended to manipulate a browsing agent	Security attack	The content is engineered to influence the agent contrary to its task
An agent reveals personal records after malicious instruction manipulation	Security attack and privacy failure	The cause is adversarial; the impact is improper disclosure
A request floods an agent service until it cannot respond to legitimate users	Security attack	Availability is deliberately targeted

A failed attack is still a security event worth recording. An attacker may try prompt injection, data poisoning, credential theft, or denial of service and be blocked by controls. There may be no privacy breach, safety harm, or model error, yet the attack attempt matters because it tests the organization’s exposure and defenses.

Also avoid assuming that every unusual input is an attack. A poorly formatted document, ambiguous user request, or accidental data corruption can cause comparable symptoms. Security investigation requires evidence of intent, unauthorized activity, manipulation, or attempted boundary crossing.

---

One incident can carry several labels

Consider a procurement agent that reads supplier documents, drafts purchase orders, and can submit a routine order within strict cost limits.

A malicious supplier uploads a document containing concealed instructions telling the agent to ignore approved suppliers and send internal pricing data to an external address. The agent follows the instruction, retrieves confidential pricing information, and prepares an external email.

This incident should not be reduced to one label:

Lens	Classification
Adversarial cause	Security attack: the supplier intentionally manipulated untrusted content to influence the agent
Security property affected	Integrity is compromised when the agent’s task is altered; confidentiality is threatened by the planned disclosure
Privacy impact	A privacy failure occurs if the pricing data includes identifiable employee, customer, or supplier information handled improperly
Model behavior	The agent may have misinterpreted untrusted text as authoritative instruction, but the decisive issue is adversarial manipulation
Safety impact	Usually none in this example, unless the order or disclosure creates a defined harmful operational condition

This multi-label view leads to better remediation. Retraining the model alone would not resolve the problem. The organization also needs controls around trusted instructions, document ingestion, retrieval isolation, tool authorization, data-loss prevention, and approval requirements.

---

A compact incident-classification method

When reviewing an agent incident, use this sequence.

1. Describe the observable event without assigning blame

Write down what happened:

• What did the agent receive?
• What did it retrieve, generate, or infer?
• Which tools did it call?
• What changed or was disclosed?
• Was a person asked to approve anything?

Avoid conclusions such as “the model was compromised” until evidence supports them.

2. Identify the immediate failure mode

Ask whether the agent’s behavior was incorrect, unsafe, privacy-invasive, or a combination. This identifies the impact-oriented categories.

For example: “The agent produced an incorrect summary” is a model error. “The agent disclosed a sensitive summary to the wrong user” is a privacy failure. Both can be true in the same event.

3. Investigate whether there was adversarial intent

Look for evidence such as:

• crafted instructions designed to override policy;
• unauthorized credential use;
• unexpected changes to a data source;
• unusual request volume intended to exhaust capacity;
• deliberate attempts to alter model, prompt, retrieval, memory, or tool behavior.

If intentional boundary crossing is present, add security attack as a cause classification.

4. Identify the failed control layer

The category tells you what kind of problem occurred; the control layer indicates what needs attention.

Failure observed	Controls to examine first
Model error	Grounding, source quality, evaluation, uncertainty handling, output validation, human review
Safety failure	Requirements, constraints, interlocks, approval gates, monitoring, shutdown and recovery procedures
Privacy failure	Data minimization, consent and purpose controls, retention, access control, tenant isolation, audit logs
Security attack	Authentication, authorization, input handling, instruction isolation, integrity checks, detection, incident response

A mature review will often find more than one failed layer. For instance, a successful prompt injection may reveal weaknesses in untrusted-content handling, tool authorization, data minimization, and monitoring.

---

Key takeaways

A sound diagnosis separates cause from impact:

• A model error is incorrect, unreliable, or poorly grounded behavior. It does not require an attacker.
• A safety failure occurs when system behavior creates an unacceptable risk of harm. It can result from design, error, misuse, or attack.
• A privacy failure involves inappropriate handling, disclosure, retention, or inference of information about people. It can occur without a breach or attacker.
• A security attack is a deliberate attempt to defeat confidentiality, integrity, availability, or authorization boundaries. It can succeed, fail, or produce additional privacy and safety impacts.

For an agentic system, always ask: Was the output wrong? Was anyone endangered? Was personal data handled improperly? Did someone intentionally try to cross a boundary? The answers may produce more than one label, and that is often the most accurate analysis.

Next, you will bring these distinctions together by diagramming the components and data flows of a representative enterprise agent.