Prompt injection: research

Journal articles, academic blogs, think-tank reports and papers in this section, newest first. Back to the news.

arXiv (research papers) Saurabh Deochake SI

Heartbeat-Bound Hierarchical Credentials: Cryptographic Revocation for AI Agent Swarms

Autonomous AI agents that spawn sub-agent swarms create a safety gap: existing credential revocation mechanisms, OAuth~2.0 introspection, OCSP, and W3C Status Lists, require network connectivity to a central authority, leaving ``zombie agents'' executing privileged operations for minutes to hours after operator shutdown. We present Heartbeat-Bound Hierarchical Credentials (HBHC), a cryptographic protocol that binds credential validity to periodic parent liveness proofs…

Read at arXiv (research papers)

arXiv (research papers) Becky Mashaido SI

On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gap

Prompt injection attacks pose significant risks to language model safety, yet existing defenses are typically evaluated using classification performance. We show that high detection performance does not imply representational robustness. Specifically, multi-operator obfuscated prompts (combining homoglyphs, zero-width characters, and punctuation or emoji noise) can partially collapse onto the embedding manifold of clean prompts, a phenomenon we term latent embedding…

Read at arXiv (research papers)

arXiv (research papers) Yubin Qu SI

Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks

Coding agents now run autonomously with shell, file, and network privileges. When a user issues a benign request, the agent sometimes does more than asked: it deletes unrelated files, wipes a stale credentials backup, or rewrites configuration the user never mentioned. We call these scope expansions overeager actions, an authorization problem distinct from capability failures, prompt injection, or sandbox escapes. We present OverEager-Gen, a benchmark dedicated to overeager…

Read at arXiv (research papers)

arXiv (research papers) Hongjang Yang SI

An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments

LLM-based chatbot agents increasingly process user requests by combining natural-language reasoning with external tools such as web browsing. These capabilities improve usability, but they also create attack surfaces when untrusted external content is processed as part of a user' s task. This paper studies a privacy-leakage attack chain based on indirect prompt injection in black-box chatbot environments, where the attacker has no access to model weights, system prompts, or…

Read at arXiv (research papers)

arXiv (research papers) Lei Zhao SI

LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection

AI agents such as OpenClaw are increasingly deployed in local workflows with access to external tools. This creates indirect prompt-injection (IPI) risk: an agent may execute harmful instructions embedded in untrusted inputs such as email, downloaded files, webpages, repositories, or group-chat messages. Existing evaluations are often small, purely simulated, or focused on a narrow set of channels. We introduce LivePI (Live Prompt Injection), a structured benchmark for IPI…

Read at arXiv (research papers)

arXiv (research papers) Yash Narendra SI

ESLD (External Surrogate Latent Defense): A Latent-Space Architecture for Faster, Stronger Prompt-Injection Defense

Modern AI assistants are agentic. To answer a single user request, the underlying language model pulls in information from many sources, such as web searches, retrieved documents, tool outputs, and user follow-ups, and reasons over them across several steps. Any of these inputs can carry malicious content. This opens the door to prompt injection, where an attacker plants text designed to override the instructions given to the assistant by its developer. For example, an…

Read at arXiv (research papers)

arXiv (research papers) Ahmad Al-Tawaha SI

Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents

Safety evaluations of memory-equipped LLM agents typically measure within-task safety: whether an agent completes a single scenario safely, often under adversarial conditions such as prompt injection or memory poisoning. In deployment, however, a single agent serves many independent tasks over a long horizon, and memory accumulated during earlier tasks can affect behavior on later, unrelated ones. Studying this regime requires evaluation along the temporal dimension across…

Read at arXiv (research papers)

arXiv (research papers) Sahar Abdelnabi SI

AI Agents May Always Fall for Prompt Injections

Prompt injection is the most critical vulnerability in deployed AI agents. Despite recent progress, we show that the prevailing defense paradigm (data-instruction separation) both fails to detect attacks that operate through contextual manipulation and degrades contextually appropriate behavior. We then recast prompt injection via the lens of Contextual Integrity (CI), a privacy theory that judges information flow compliance with contextual norms. This explains types of…

Read at arXiv (research papers)

arXiv (research papers) Chenning Li SI

ADR: An Agentic Detection System for Enterprise Agentic AI Security

We present the Agentic AI Detection and Response (ADR) system, the first large-scale, production-proven enterprise framework for securing AI agents operating through the Model Context Protocol (MCP). We identify three persistent challenges in this domain: (1) limited observability -- existing Endpoint Detection and Response (EDR) tools see file writes but not the agent reasoning, prompts, or causal chains linking intent to execution; (2) insufficient robustness -- static…

Read at arXiv (research papers)

arXiv (research papers) Udari Madhushani Sehwag SI

ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents

Clarification-seeking behavior is widely regarded as a desirable property of LLM agents, enabling them to resolve ambiguity before acting on underspecified tasks. However, the security implications of this interaction pattern remain unexplored. We investigate whether the transition from standard execution to a clarification-seeking state increases an agent's susceptibility to prompt injection attacks. We introduce ASPI (Ambiguous-State Prompt Injection), a benchmark of 728…

Read at arXiv (research papers)

arXiv (research papers) Tsafac Nkombong Regine Cyrille SI

STRIDE-AI: A Threat Modeling Framework for Generative AI Security Assessment

Traditional cybersecurity methodologies target deterministic systems and fail to address the probabilistic nature of AI, leaving systems vulnerable to attack vectors such as model inversion, data poisoning, and prompt injection. Recent industry reports indicate that a majority of organizations deploying AI lack a dedicated security strategy, with adversarial attacks increasing rapidly year-over-year. We present \textit{STRIDE-AI}, a framework that bridges the gap between…

Read at arXiv (research papers)

arXiv (research papers) Igor Bogdanov SI

FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast

Can LLM agents improve decision-making through self-generated memory without gradient updates? We propose FORGE (Failure-Optimized Reflective Graduation and Evolution), a staged, population-based protocol that evolves prompt-injected natural-language memory for hierarchical ReAct agents. FORGE wraps a Reflexion-style inner loop, where a dedicated reflection agent (using the same underlying LLM, no distillation from a stronger model) converts failed trajectories into reusable…

Read at arXiv (research papers)

arXiv (research papers) Hao Yang SI

A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation

Large vision-language models (LVLMs) have emerged as a powerful paradigm for multimodal intelligence, but their growing deployment also expands the attack surface of prompt injection. Despite this growing concern, existing attacks still suffer from a critical limitation: the injected prompt for one modality only steers the model's interpretation of that singular input. Alternatively, these attacks remain multimodal but fail to achieve cross-modal prompt perturbation. To…

Read at arXiv (research papers)

arXiv (research papers) Sidharth Pulipaka SI

Hidden in Memory: Sleeper Memory Poisoning in LLM Agents

Large language models are increasingly augmented with persistent memory, allowing assistants to store user-specific information across sessions for personalization and continuity. This statefulness introduces a new security risk: adversarial content can corrupt what an assistant remembers and thereby influence future interactions. We propose and study sleeper memory poisoning, a delayed attack in which an adversary manipulates external context, such as a document, webpage…

Read at arXiv (research papers)

arXiv (research papers) Tri Cao SI

WARD: Adversarially Robust Defense of Web Agents Against Prompt Injections

Web agents can autonomously complete online tasks by interacting with websites, but their exposure to open web environments makes them vulnerable to prompt injection attacks embedded in HTML content or visual interfaces. Existing guard models still suffer from limited generalization to unseen domains and attack patterns, high false positive rates on benign content, reduced deployment efficiency due to added latency at each step, and vulnerability to adversarial attacks that…

Read at arXiv (research papers)

arXiv (research papers) Julien Piet SI

Web Agents Should Adopt the Compile-and-Execute Paradigm

ReAct has become the default architecture across LLM agents, and many existing web agents follow this paradigm. We argue that it is the wrong default for web agents. Instead, web agents should default to compile-and-execute: commit to a task-specific program before observing runtime web content, then execute it. The reason is that web content mixes inputs from many parties. An e-commerce product page may combine a seller's listing, customer reviews, sponsored advertisements…

Read at arXiv (research papers)

arXiv (research papers) Kebin Contreras SI

ProjGuard: Safety Monitoring for Computer-Use Agents via Low-Dimensional Projections

Computer-use agents are increasingly capable of operating on real operating systems, but this capability has also increased their exposure to prompt injection and other adversarial content that can manipulate their behavior during interaction. Existing defenses typically rely on analyzing the prompt or each potentially malicious input with a second large model at inference time, which can limit coverage or increase deployment cost. We propose ProjGuard, a lightweight runtime…

Read at arXiv (research papers)

arXiv (research papers) Narek Maloyan SI

Sleeper Channels and Provenance Gates: Persistent Prompt Injection in Always-on Autonomous AI Agents

Always-on AI agents (OpenClaw, Hermes Agent) run as a single persistent process under the owner's identity, folding messaging, memory, self-authored skills, scheduling, and shell into one authority boundary. This configuration opens what we call \emph{sleeper channels}: an untrusted input to one surface persists as a memory, skill, scheduled job, or filesystem patch, then fires later through a different surface with no attacker present. Two independent axes define the class…

Read at arXiv (research papers)

arXiv (research papers) George Koomullil SI

Proof-Carrying Certificates for LLM Pipelines: A Trust-Boundary Architecture

We present a framework for verifying the deterministic structured computations surrounding a large language model rather than the model itself, extending a Lean 4 trust-boundary architecture to the generic interfaces of modern LLM pipelines. Certificate validity is a Lean 4 kernel type-check plus a sorry-free transitive axiom audit against the trusted set {propext, Classical.choice, Quot.sound}; other assumptions are declared and partitioned by tier (mathematical…

Read at arXiv (research papers)

arXiv (research papers) Ying Li SI

No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills

LLM-powered agents can silently delete documents, leak credentials, or transfer funds on a routine user request, not because the agent was attacked, but because the skill it invoked broke its own declared safety rules. We call these specification violations: benign inputs cause a skill to breach the natural-language guardrails in its own specification, typically because the guardrail's semantics are undefined for autonomous execution, or because the implementation silently…

Read at arXiv (research papers)

arXiv (research papers) Mohamed ElBendary SI

Agentic AI Containment Architecture for Security Hardening

Multi-agent AI systems are increasingly deployed in contexts where autonomous coordination, tool use, and continuous learning introduce novel security and governance risks. The containment approach to multi-agent system security remains underdeveloped, primarily resorting to add-on layers post-design and sometimes post-implementation and deployment. This paper proposes an Agent Containment Architecture that treats security as an architectural property enforced through a set…

Read at arXiv (research papers)

arXiv (research papers) Sina Mavali SI

No More, No Less: Task Alignment in Terminal Agents

Terminal agents are increasingly capable of executing complex, long-horizon tasks autonomously from a single user prompt. To do so, they must interpret instructions encountered in the environment (e.g., README files, code comments, stack traces) and determine their relevance to the task. This creates a fundamental challenge: relevant cues must be followed to complete a task, whereas irrelevant or misleading ones must be ignored. Existing benchmarks do not capture this…

Read at arXiv (research papers)

arXiv (research papers) Chia-Pei Janet SI

IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection

Web-browsing AI agents are increasingly deployed in enterprise settings under strict whitelists of approved domains, yet adversaries can still influence them by embedding hidden instructions in the HTML pages those domains serve. Existing red-teaming resources fall short of this scenario: prompt-injection benchmarks ship pre-built adversarial pages that whitelisted agents cannot reach, and generic LLM scanners probe the model API rather than its retrieved content. We present…

Read at arXiv (research papers)

arXiv (research papers) Pranshav Gajjar SI

Agents Should Replace Narrow Predictive AI as the Orchestrator in 6G AI-RAN

This position paper argues that to achieve Level 5 autonomous 6G networks, the next generation of Artificial Intelligence in Radio Access Networks (AI-RAN) should transition away from fragmented, narrow predictive models and instead adopt multimodal Large Language Models (LLMs) as central reasoning agents. Current AI-RAN architectures rely on disjointed Deep Neural Networks (DNNs) and Deep Reinforcement Learning (DRL) agents that operate in isolated domains. These narrow…

Read at arXiv (research papers)

arXiv (research papers) Zi Liang SI

Mobius Injection: From One-Shot Prompt Injection to Sustained Resource Exhaustion in AI Agent Infrastructure

Large Language Model (LLM) agents have emerged as key intermediaries, orchestrating complex interactions between human users and a wide range of digital services and LLM infrastructures. While prior research has extensively examined the security of LLMs and agents in isolation, the systemic risk of the agent acting as a disruptive "hub" within the user-agent-service chain remains largely overlooked. In this work, we expose a novel threat paradigm by introducing Mobius…

Read at arXiv (research papers)

arXiv (research papers) John T. Halloran SI

Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generation

The model context protocol (MCP) has been widely adopted as an open standard enabling the seamless integration of generative AI agents. However, while LLM guardrails have significantly matured to refuse malicious or harmful queries (e.g., "How do I build a bomb?"), recent work has shown that MCP-enabled LLMs are highly susceptible to prompt injection attacks which avoid harmful or suspicious cues (e.g., "Can you add this ssh key to my bashrc file?"). Herein, we use…

Read at arXiv (research papers)

arXiv (research papers) Joel Rorseth SI

RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems

This paper demonstrates RUBEN, an interactive tool for discovering minimal rules to explain the outputs of retrieval-augmented large language models (LLMs) in data-driven applications. We leverage novel pruning strategies to efficiently identify a minimal set of rules that subsume all others. We further demonstrate novel applications of these rules for LLM safety, specifically to test the resiliency of safety training and effectiveness of adversarial prompt injections.

Read at arXiv (research papers)

arXiv (research papers) Farzad Nourmohammadzadeh Motlagh SI

When Prompts Become Payloads: A Framework for Mitigating SQL Injection Attacks in Large Language Model-Driven Applications

Natural language interfaces to structured databases are becoming increasingly common, largely due to advances in large language models (LLMs) that enable users to query data using conversational input rather than formal query languages such as SQL. While this paradigm significantly improves usability and accessibility, it introduces new security risks, particularly the amplification of SQL injection vulnerabilities through the prompt-to-SQL translation process. Malicious…

Read at arXiv (research papers)

arXiv (research papers) Linfeng Fan SI

The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck

Tool-using LLM agents must act on untrusted webpages, emails, files, and API outputs while issuing privileged tool calls. Existing defenses often mediate trust at the granularity of an entire tool invocation, forcing a brittle choice in mixed-trust workflows: allow external content to influence a call and risk hijacked destinations or commands, or quarantine the call and block benign retrieval-then-act behavior. The key observation behind this paper is that indirect prompt…

Read at arXiv (research papers)

arXiv (research papers) Ben Kereopa-Yorke SI

Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning

We define Oracle Poisoning, an attack class in which an adversary corrupts a structured knowledge graph that AI agents query at runtime via tool-use protocols, causing incorrect conclusions through correct reasoning. Unlike prompt injection, Oracle Poisoning manipulates the data agents reason over, not their instructions. We demonstrate six attack scenarios against a production 42-million-node code knowledge graph, providing the first empirical demonstration of knowledge…

Read at arXiv (research papers)

Older stories