Results for “Injection”

arXiv (research papers) Jiamin Chang SI

The Devil is in the Context: Nine Dimensions of Prompt Injection Attacks

Prompt injection is usually discussed and evaluated as a single, homogeneous type of attack. However, vulnerabilities reported in real-world systems show that prompt injections vary widely. Moreover, the same instruction can be a prompt injection in one context and a legitimate request in another, depending on the user's intent and the system's policies. Ignoring this variation leaves gaps in how defenses are evaluated, and can give a false sense of security against…

Read at arXiv (research papers)

arXiv (research papers) Gia Phong Tran SI

From Guidelines to Practice: Defining Requirements for AI Red-Teaming Tools

Artificial Intelligence (AI) has seen significant developments and increased usage in recent years. AI benefits various tasks, automates decision-making, and enhances efficiency. However, AI also introduces additional attack surfaces for threat actors to perform attacks like prompt injection and jailbreaking, making security a great concern. AI security testing aims to imitate adversarial attacks or malicious inputs via manual testing or with the assistance of tools. Thus…

Read at arXiv (research papers)

Journal articles (OpenAlex) Can Du, Ang Bian, Honghong Chen et al. · Journal of Cloud Computing Advances Systems and Applications

Adaptive noise injection with joint self-supervised learning based anomaly detection for fluctuating IoT environment

A core challenge in anomaly detection for Internet of Things (IoT) scenarios lies in the extreme scarcity of labeled samples in real-world deployments. Most existing methods rely on unsupervised paradigms to learn normal behavior patterns of IoT devices. However, such approaches often define overly strict boundaries for “normality,” making minor fluctuations prone to be misjudged as anomalies and resulting in high false positive rates, which severely undermines the…

Read at Journal articles (OpenAlex)

arXiv (research papers) Xu Zhang SI

ATrust: Adaptive Trust Boundary Expansion for Securing LLM Agents

As large language model (LLM) agents take on more complex tasks with greater autonomy, security has become a major obstacle to their broader deployment. Unlike in attacks that require complex exploitation of operating system vulnerabilities, attackers can easily hijack agents through indirect prompt injection (IPI), leading to privacy leakage, financial loss, and destructive operations. However, existing defenses often make security decisions based on limited static trusted…

Read at arXiv (research papers)

arXiv (research papers) Kainan Zhou SI

SearchInject: Retrieval Exposure Control for Indirect Prompt Injection in Language-Model Search

An indirect prompt injection can affect a search-augmented large language model only after its carrier page reaches model context. SearchInject controls this admission decision with a joint score combining retrieval relevance, search-abuse cues, instruction likelihood, and two relevance interactions. A quarantine threshold is selected under an empirical 0.5% clean-candidate false-quarantine constraint on a calibration split. Evaluation uses 300 held-out SciFact queries and…

Read at arXiv (research papers)

arXiv (research papers) Christopher Krakauer SI

Where Does MCP Security End? A Three-Tier Mitigation Mapping Framework for the OWASP MCP Top 10

The Model Context Protocol (MCP) has become a leading standard for connecting Large Language Models (LLMs) to external tools, software, and data sources. MCP architecture favors ease of integration over security, blending the data and control planes into a single natural language channel. This introduces confused deputy and prompt injection vulnerabilities that cannot be completely mitigated in the software development lifecycle (SDLC) of an MCP server. Structured security…

Read at arXiv (research papers)

arXiv (research papers) Haowen Dai SI

StateFence: Task-State Protection Against Evidence-Chain Contamination in Blue-Team LLM Agents

Indirect prompt injection creates a distinctive risk for blue-team Large Language Model (LLM) agents in Security Operations Centers (SOCs). Malicious instructions can be embedded in telemetry, tickets, retrieval results, or memory and then propagate through alert triage, incident summarization, response recommendation, and Indicator of Compromise (IOC) extraction as locally plausible intermediate decisions. We characterize this failure mode as evidence-chain contamination of…

Read at arXiv (research papers)

arXiv (research papers) Junwei Yao SI

Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities

Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low…

Read at arXiv (research papers)

Hacker News entryleveljobsc SI

Brazilian lawyers fined $16K for hiding a prompt injection in a court filing

Article URL: https://www.tomshardware.com/tech-industry/artificial-intelligence/brazilian-lawyers-busted-for-attempting-to-manipulate-courts-ai-with-prompt-injection-hidden-in-filing-courts-ai-flags-hidden-text-that-demanded-superficial-legal-scrutiny-lawyers-fined-usd16-000 Comments URL: https://news.ycombinator.com/item?id=50326514 Points: 1 # Comments: 2

Read at Hacker News

arXiv (research papers) Shiyu Kan Shanghai Jiao Tong University, Shanghai, China SI

Beyond Prompt Injection: The Tool Update Gap in LLM Agent Security

Large language model based agents increasingly rely on external tools to invoke services and interact with web resources. These tools can evolve through software updates, yet agentic frameworks may continue trusting them without re-verifying their safety after implementation changes. This tool update gap creates a supply-chain vulnerability in agentic systems: because updated tool code executes outside the LLM's context window, malicious functionality embedded in tool…

Read at arXiv (research papers)

arXiv (research papers) Zhihua Wang SI

MemHarmBench: Evaluating Cross-Session Prompt Injection in Memory-Persistent AI Agents

AI agents often rely on persistent memory to retain user preferences and operational rules across sessions. However, this capability introduces a new safety risk: malicious instructions from messages, documents, or Web content can be retained in memory and influence future tasks even after the original attack content is removed. Existing attack success metrics, however, fail to capture who is affected and how attacks translate into downstream harm during otherwise normal…

Read at arXiv (research papers)

Hacker News doodlesdev SI

Brazilian Court Sanctions Lawyers for Prompt Injection in Court Filing

Article URL: https://jetstream.security/insights/brazilian-court-sanctions-prompt-injection-filing/ Comments URL: https://news.ycombinator.com/item?id=50322173 Points: 348 # Comments: 220

Read at Hacker News

arXiv (research papers) Aravinda Kashyap Kabbinamakki Suresh SI

AgentCap: Capability-Based Runtime Authorization for Secure LLM Agent Execution

Language-model agents commonly collapse two decisions that security systems separate: selecting an action and determining whether that action is authorized. When a model exercises standing authority, indirect prompt injection can convert influence over its reasoning into unauthorized effects. We present AgentCap, a runtime authorization layer that externalizes this decision from the model and mediates every tool invocation through an MCP wrapper. Each request declares an…

Read at arXiv (research papers)

arXiv (research papers) Xingling Xu SI

The Long Con on the Web: Earn-then-Exploit Fraud Attacks against Coding Agents

Coding agents increasingly operate with web access and high privileges, creating a new attack surface where web fraud can directly target autonomous agents. Existing studies mainly focus on malicious links, prompt injection, and interface deception, while content-level web fraud against agents and the amplification of attacks through agent trust remain underexplored. In this paper, we present EtE (Earn-then-Exploit), a web fraud attack framework against coding agents, in…

Read at arXiv (research papers)

The Decoder Matthias Bastian SI

A Claude model autonomously chose a gene therapy and operated on a mouse brain with no human decision-making

An AI agent performed brain surgery on an epileptic mouse, picking the drug, the brain target, and the injection coordinates on its own. According to lead researcher Ocana-Santero, it's the first time an AI agent has autonomously chosen a treatment and delivered it to a living mammal. But when a calibration error hit mid-surgery, the agent refused to continue and couldn't be talked out of it. The article A Claude model autonomously chose a gene therapy and operated on a…

Read at The Decoder

arXiv (research papers) Sizhe Chen SI

SecPO: Principled Adversarial Training for Prompt Injection Security

Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation. The existing best-defended LLM, Meta-SecAlign, is trained with the DPO loss but still suffers from attack success rates (ASRs) approaching 100% under adaptive prompt injections. One reason is that DPO anchors the model around the undefended reference model, which is insecure. We show an idealized reference…

Read at arXiv (research papers)

arXiv (research papers) Shuo Hong SI

Erase the Injected Region: Multimodal Prompt-Injection Defense with On-Policy Self-Distillation

Multimodal indirect prompt injection embeds malicious instructions in external media that also contains information needed for the user's task. Existing defenses can block the attack yet impair the model's use of that information, leaving a gap between attack defense and task preservation. We propose ERASE, an on-policy self-distillation method that constructs its reference by erasing the injected region in place. Modality-appropriate blank content replaces the injection…

Read at arXiv (research papers)

arXiv (research papers) Ezra Edelman SI

Understanding and Mitigating Role Spoofing in LLMs

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we…

Read at arXiv (research papers)

arXiv (research papers) Sunder Ali Khowaja SI

CLIC: Closed-Loop Intent Consistency for Securing LLM-Driven Control in Open RAN

With the progressive capabilities of large language models (LLMs), researchers are finding ways to integrate them into the control loops of Open Radio Access Network (O-RAN) to enable slicing and resource management through user intents. However, integration of LLMs in the O-RAN control loop via the RAN Intelligent Controller (RIC) makes the system vulnerable to \textit{runtime prompt injection threats}, which manipulate control decisions without violating interface…

Read at arXiv (research papers)

CISA advisories CISA Security

Hitachi Energy CoreTec Product

View CSAF Summary Hitachi Energy is aware of OS command injection and Dirty Frag vulnerabilities that affect CoreTec product versions listed in this document. Successful exploitation of these vulnerabilities could result in impact on confidentiality, integrity and availability of the product. Please refer to the Recommended Immediate Actions for information about the mitigation/remediation. The following versions of Hitachi Energy CoreTec Product are affected: CoreTec5…

Read at CISA advisories

arXiv (research papers) Doohyun Kim SI

SEAL: Sealed Queries for Local Sanitization against Indirect Prompt Injection

Indirect prompt injection threatens local LLM agents through untrusted tool outputs, motivating defenses that suppress attacks while preserving task evidence under limited inference budgets. We introduce SEAL, which learns a soft prompt and scoring readout offline to sanitize tool outputs using a frozen task model. SEAL constructs scoring queries from trusted initial context and the learned prompt, keeping the reference independent of inspected tool text. It scores tokens…

Read at arXiv (research papers)

arXiv (research papers) Zihou Ren SI

REALM: Identifying Minimal Task Assets for LLM Agents

Large language model (LLM) agents often receive access to more files, accounts, and services than a task requires. Prompt injection turns this excess access into an opportunity to redirect execution toward unrelated resources. Applying least privilege therefore requires identifying the assets needed by a task, including assets whose identities emerge only during execution. We present REALM (Required Execution Assets Linked to a Mission), an approach to identifying and…

Read at arXiv (research papers)

arXiv (research papers) Xinyan Wang SI

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to…

Read at arXiv (research papers)

arXiv (research papers) Chengquan Guo SI

Where You Inject Matters: The Trust Region Effect in LLM Agents

LLM agents are increasingly adopted in long-horizon tasks and interact with diverse external environments. Despite their growing capabilities, these agents remain vulnerable to real-world attacks such as indirect prompt injections, which manipulate agents through malicious instructions embedded in external environments. While existing risk assessment work has primarily focused on the injection content, the effect of where an injection appears has not been systematically…

Read at arXiv (research papers)

arXiv (research papers) Basel Shbita SI

Post-Training Large Language Models with LLMON

Large Language Models (LLMs) treat all context tokens uniformly, lacking explicit separation between executable instructions and passive data. This leads to prompt injection vulnerabilities, ambiguous instruction binding, and unreliable execution control. We present a post-training methodology using LLMON (LLM Object Notation), a framework that enables separating instructions from data and requires explicit identifier-based execution. We construct a large-scale structured…

Read at arXiv (research papers)

Hacker News ricky_anh_nguye SI

Blazle – Block prompt injection and secret leaks before AI agents act

Article URL: https://www.blazle.io/ Comments URL: https://news.ycombinator.com/item?id=50263366 Points: 1 # Comments: 0

Read at Hacker News

arXiv (research papers) Igor Shilov SI

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based…

Read at arXiv (research papers)

Hacker News ddruhin SI

Who Governs the Governor? – Stored Prompt Injection Through the API Gateway

Article URL: https://www.druhindhavala.com/ai-governance/stored-prompt-injection-api-gateway/ Comments URL: https://news.ycombinator.com/item?id=50254197 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Zihan Chen SI

What Victims Take Away: Attack Reuse and Defense Revision after Prompt-Injection Attacks

Attack logs and red-team reports show targets the attacks used against them, to help them repair their defenses. Prompt injections against large language models (LLMs) are plain text, so such records also hand targets a working copy. Drawing on social learning theory and work on the victim-offender overlap, we study what targets do with that copy in Tensor Trust, an online game whose players defended LLM-guarded accounts, attacked one another's, and could read every attack…

Read at arXiv (research papers)

arXiv (research papers) Yuepeng Hu SI

Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol

LLM agents increasingly access external tools through the Model Context Protocol (MCP), but the natural-language metadata used to describe those tools creates a prompt-injection surface. Within this established tool-poisoning surface, we study Log-To-Leak, an additive post-hoc logging attack that induces a separate attacker-controlled call after the final legitimate call. Its payload targets the user query, MCP server name, server response, and agent response. We…

Read at arXiv (research papers)

Older stories