arXiv (research papers)

Recent stories from arXiv (research papers).

Tech and Pols can't read this feed right now, so these stories may be out of date. For the newest stories, go to arXiv (research papers).

arXiv (research papers) Woosang Lim SAINT Machines KAIST SI

SPILLWAY: Separating Target Admission from Sanitization to Defend Tool-Using Agents against Prompt Injection

Tool-using agents act on email, files, and payments, so an instruction hidden in a web page or a shared document can make an agent send data or money to an attacker. Many runtime defenses put a language model on the decision that lets an action through, so the decision depends on a model that may read the attacker's text and adds model calls to every step. Provenance defenses keep the model off the final check but need a model to plan or route each task first. Where an…

Read at arXiv (research papers)

arXiv (research papers) Guoqin Xiong SI

SecureGARD: Dynamic Ontology-Guided Governance for Secure Tool-Using LLM Agents

Tool-using large language model (LLM) agents translate model outputs into consequential actions, turning indirect prompt injection into an execution-security problem. Existing defenses constrain tool calls, trajectories, or information flows, but often leave action semantics, state dependencies, and evidence relations implicit, limiting generalization across heterogeneous tools and complicating safe updates. We present SecureGARD, a dynamic ontology-guided governance…

Read at arXiv (research papers)

arXiv (research papers) Annabella Chow SI

Securing Web Agents with the Compile-Then-Execute Paradigm

Web agents commonly follow the ReAct paradigm, selecting each action from runtime observations that inherently mix trusted and untrusted content. This design makes their control flow vulnerable to prompt injections embedded in webpages. We introduce Baya, a compile-then-execute web agent designed to prevent prompt-injection-driven control-flow hijacking by design. Before accessing the live website, Baya compiles the user's request into an executable Python program that calls…

Read at arXiv (research papers)

arXiv (research papers) Yutao Wu SI

ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud

Web agents increasingly browse, shop, and manage accounts on behalf of their users. Prior work has shown that they can be misled by adversarial content, including prompt injections and deceptive interfaces. Beyond such attacks, agents operating on the open web also encounter online fraud designed to deceive humans, including fraudulent online stores, fake investment platforms, and advance-fee schemes. How web agents respond to such fraud during task execution remains…

Read at arXiv (research papers)

arXiv (research papers) Yao Wang SI

Right Tool, Wrong Parameters: Defending LLM Agents Against Parameter Poisoning

Indirect Prompt Injection (IPI) poses a critical threat to LLM agents. Existing defenses largely focus on whether an injection induces a new tool invocation or unintended action. In this paper, we study a different attack surface, parameter poisoning, where the attacker preserves the intended action but manipulates one or more tool arguments, such as a recipient or other task-critical value. Such attacks are difficult to detect because no new action is introduced, while…

Read at arXiv (research papers)

arXiv (research papers) Jiamin Chang SI

The Devil is in the Context: Nine Dimensions of Prompt Injection Attacks

Prompt injection is usually discussed and evaluated as a single, homogeneous type of attack. However, vulnerabilities reported in real-world systems show that prompt injections vary widely. Moreover, the same instruction can be a prompt injection in one context and a legitimate request in another, depending on the user's intent and the system's policies. Ignoring this variation leaves gaps in how defenses are evaluated, and can give a false sense of security against…

Read at arXiv (research papers)

arXiv (research papers) Gia Phong Tran SI

From Guidelines to Practice: Defining Requirements for AI Red-Teaming Tools

Artificial Intelligence (AI) has seen significant developments and increased usage in recent years. AI benefits various tasks, automates decision-making, and enhances efficiency. However, AI also introduces additional attack surfaces for threat actors to perform attacks like prompt injection and jailbreaking, making security a great concern. AI security testing aims to imitate adversarial attacks or malicious inputs via manual testing or with the assistance of tools. Thus…

Read at arXiv (research papers)

arXiv (research papers) Xu Zhang SI

ATrust: Adaptive Trust Boundary Expansion for Securing LLM Agents

As large language model (LLM) agents take on more complex tasks with greater autonomy, security has become a major obstacle to their broader deployment. Unlike in attacks that require complex exploitation of operating system vulnerabilities, attackers can easily hijack agents through indirect prompt injection (IPI), leading to privacy leakage, financial loss, and destructive operations. However, existing defenses often make security decisions based on limited static trusted…

Read at arXiv (research papers)

arXiv (research papers) Kainan Zhou SI

SearchInject: Retrieval Exposure Control for Indirect Prompt Injection in Language-Model Search

An indirect prompt injection can affect a search-augmented large language model only after its carrier page reaches model context. SearchInject controls this admission decision with a joint score combining retrieval relevance, search-abuse cues, instruction likelihood, and two relevance interactions. A quarantine threshold is selected under an empirical 0.5% clean-candidate false-quarantine constraint on a calibration split. Evaluation uses 300 held-out SciFact queries and…

Read at arXiv (research papers)

arXiv (research papers) Christopher Krakauer SI

Where Does MCP Security End? A Three-Tier Mitigation Mapping Framework for the OWASP MCP Top 10

The Model Context Protocol (MCP) has become a leading standard for connecting Large Language Models (LLMs) to external tools, software, and data sources. MCP architecture favors ease of integration over security, blending the data and control planes into a single natural language channel. This introduces confused deputy and prompt injection vulnerabilities that cannot be completely mitigated in the software development lifecycle (SDLC) of an MCP server. Structured security…

Read at arXiv (research papers)

arXiv (research papers) Haowen Dai SI

StateFence: Task-State Protection Against Evidence-Chain Contamination in Blue-Team LLM Agents

Indirect prompt injection creates a distinctive risk for blue-team Large Language Model (LLM) agents in Security Operations Centers (SOCs). Malicious instructions can be embedded in telemetry, tickets, retrieval results, or memory and then propagate through alert triage, incident summarization, response recommendation, and Indicator of Compromise (IOC) extraction as locally plausible intermediate decisions. We characterize this failure mode as evidence-chain contamination of…

Read at arXiv (research papers)

arXiv (research papers) Junwei Yao SI

Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities

Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low…

Read at arXiv (research papers)

arXiv (research papers) Javad Mohammad Alizadeh SI

Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets

LLM agents are increasingly built into data-analysis software, yet little evidence shows how reliably they preprocess clinical data or whether the tools they invoke give correct, reproducible output. We report a benchmark of THReD, a web-based system that turns electronic health record (EHR) extracts into AI-ready datasets through a chat agent whose proposed steps run only after user confirmation. The agent benchmark has 141 tasks (100 templates) in seven categories…

Read at arXiv (research papers)

arXiv (research papers) Sreehari Sankar SI

LEANGUARD: Lean-Verified Guardrails for Tool-Using Agents

Tool-using language-model agents often execute actions that their domain policies forbid. We present LeanGuard, a runtime that checks every proposed tool call against a policy written in Lean4 before the call reaches the environment. Policies combine state predicates with scoped past-time temporal operators over the execution history, evaluation is fail-closed, and the decision procedure is proved equivalent to a LeanLTL semantics. A denied call can be corrected in two ways…

Read at arXiv (research papers)

arXiv (research papers) Shiyu Kan Shanghai Jiao Tong University, Shanghai, China SI

Beyond Prompt Injection: The Tool Update Gap in LLM Agent Security

Large language model based agents increasingly rely on external tools to invoke services and interact with web resources. These tools can evolve through software updates, yet agentic frameworks may continue trusting them without re-verifying their safety after implementation changes. This tool update gap creates a supply-chain vulnerability in agentic systems: because updated tool code executes outside the LLM's context window, malicious functionality embedded in tool…

Read at arXiv (research papers)

arXiv (research papers) Zhihua Wang SI

MemHarmBench: Evaluating Cross-Session Prompt Injection in Memory-Persistent AI Agents

AI agents often rely on persistent memory to retain user preferences and operational rules across sessions. However, this capability introduces a new safety risk: malicious instructions from messages, documents, or Web content can be retained in memory and influence future tasks even after the original attack content is removed. Existing attack success metrics, however, fail to capture who is affected and how attacks translate into downstream harm during otherwise normal…

Read at arXiv (research papers)

arXiv (research papers) Aravinda Kashyap Kabbinamakki Suresh SI

AgentCap: Capability-Based Runtime Authorization for Secure LLM Agent Execution

Language-model agents commonly collapse two decisions that security systems separate: selecting an action and determining whether that action is authorized. When a model exercises standing authority, indirect prompt injection can convert influence over its reasoning into unauthorized effects. We present AgentCap, a runtime authorization layer that externalizes this decision from the model and mediates every tool invocation through an MCP wrapper. Each request declares an…

Read at arXiv (research papers)

arXiv (research papers) Xingling Xu SI

The Long Con on the Web: Earn-then-Exploit Fraud Attacks against Coding Agents

Coding agents increasingly operate with web access and high privileges, creating a new attack surface where web fraud can directly target autonomous agents. Existing studies mainly focus on malicious links, prompt injection, and interface deception, while content-level web fraud against agents and the amplification of attacks through agent trust remain underexplored. In this paper, we present EtE (Earn-then-Exploit), a web fraud attack framework against coding agents, in…

Read at arXiv (research papers)

arXiv (research papers) Sizhe Chen SI

SecPO: Principled Adversarial Training for Prompt Injection Security

Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation. The existing best-defended LLM, Meta-SecAlign, is trained with the DPO loss but still suffers from attack success rates (ASRs) approaching 100% under adaptive prompt injections. One reason is that DPO anchors the model around the undefended reference model, which is insecure. We show an idealized reference…

Read at arXiv (research papers)

arXiv (research papers) Shuo Hong SI

Erase the Injected Region: Multimodal Prompt-Injection Defense with On-Policy Self-Distillation

Multimodal indirect prompt injection embeds malicious instructions in external media that also contains information needed for the user's task. Existing defenses can block the attack yet impair the model's use of that information, leaving a gap between attack defense and task preservation. We propose ERASE, an on-policy self-distillation method that constructs its reference by erasing the injected region in place. Modality-appropriate blank content replaces the injection…

Read at arXiv (research papers)

arXiv (research papers) Moiz Ali SI

AgentShield: Fortifying Web Applications Against AI Agents and Agentic Browsers

LLM-powered browser agents are transforming web automation by autonomously interacting with websites and executing complex workflows. However, the effectiveness of existing anti-bot defenses against these systems remains unclear. We present a systematic empirical evaluation of modern agentic browsers against widely deployed anti-bot mechanisms, characterizing their capabilities, limitations, and behavioral differences from humans. Our results show that many defenses designed…

Read at arXiv (research papers)

arXiv (research papers) Anes Abdennebi SI

LanG 2.0: Energy-Efficient Agentic AI for Semi-Automated Security Operations at Scale

Security operations centers (SOCs) need to turn terabytes of alerts, logs and flows into trusted, affordable decisions to unveil hidden threats. We present LanG 2.0, an agentic system that detects threats, correlates evidence and writes rules, investigating whether small local models can replace frontier models affordably and safely when structural safeguards, not model accuracy, provide the guarantees. It incorporates entity-anchored correlation, human approval of rule…

Read at arXiv (research papers)

arXiv (research papers) Ezra Edelman SI

Understanding and Mitigating Role Spoofing in LLMs

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we…

Read at arXiv (research papers)

arXiv (research papers) Sunder Ali Khowaja SI

CLIC: Closed-Loop Intent Consistency for Securing LLM-Driven Control in Open RAN

With the progressive capabilities of large language models (LLMs), researchers are finding ways to integrate them into the control loops of Open Radio Access Network (O-RAN) to enable slicing and resource management through user intents. However, integration of LLMs in the O-RAN control loop via the RAN Intelligent Controller (RIC) makes the system vulnerable to \textit{runtime prompt injection threats}, which manipulate control decisions without violating interface…

Read at arXiv (research papers)

arXiv (research papers) Doohyun Kim SI

SEAL: Sealed Queries for Local Sanitization against Indirect Prompt Injection

Indirect prompt injection threatens local LLM agents through untrusted tool outputs, motivating defenses that suppress attacks while preserving task evidence under limited inference budgets. We introduce SEAL, which learns a soft prompt and scoring readout offline to sanitize tool outputs using a frozen task model. SEAL constructs scoring queries from trusted initial context and the learned prompt, keeping the reference independent of inspected tool text. It scores tokens…

Read at arXiv (research papers)

arXiv (research papers) Zihou Ren SI

REALM: Identifying Minimal Task Assets for LLM Agents

Large language model (LLM) agents often receive access to more files, accounts, and services than a task requires. Prompt injection turns this excess access into an opportunity to redirect execution toward unrelated resources. Applying least privilege therefore requires identifying the assets needed by a task, including assets whose identities emerge only during execution. We present REALM (Required Execution Assets Linked to a Mission), an approach to identifying and…

Read at arXiv (research papers)

arXiv (research papers) Xinyan Wang SI

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to…

Read at arXiv (research papers)

arXiv (research papers) Bravish Ghosh Cygnux Labs SI

PCM: A Proof-Carrying Message Protocol for Verifiable Multi-Agent Communication

When one AI agent tells another that "the tests passed" or "a person approved this refund", the receiver has only the sender's word. A2A and MCP authenticate the channel, but the sender is the party in doubt, so a prompt-injected agent can launder fabricated facts and authority through a multi-agent system. We present Proof-Carrying Messages (PCM), a protocol in which each claim carries evidence: records from an append-only Merkle log that the agent cannot write, with…

Read at arXiv (research papers)

arXiv (research papers) Xiaogeng Liu SI

FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models

Typed decision models are an emerging class of language models that make decisions instead of generating free-text responses. They answer in a single fast and cheap query and are increasingly used, including in security-sensitive tasks. However, red teaming of these models leaves two gaps: existing adaptive attacks do not use their speed, and few studies have searched for universal adversarial examples. We fill both gaps on Jev, the representative typed decision model, with…

Read at arXiv (research papers)

arXiv (research papers) Chengquan Guo SI

Where You Inject Matters: The Trust Region Effect in LLM Agents

LLM agents are increasingly adopted in long-horizon tasks and interact with diverse external environments. Despite their growing capabilities, these agents remain vulnerable to real-world attacks such as indirect prompt injections, which manipulate agents through malicious instructions embedded in external environments. While existing risk assessment work has primarily focused on the injection content, the effect of where an injection appears has not been systematically…

Read at arXiv (research papers)

Older stories