Results for “Injection”

arXiv (research papers) Peichun Hua SI

Behavioral Skill Reconstruction: Reconstructing Hidden Functionality from LLM Agent Skills

Closed source agent skills may encode proprietary instructions, scripts, constants, and data. Providers may offer their capabilities as services while keeping the underlying packages hidden. Prior work focuses on prompt injection attacks that directly disclose these artifacts, and existing defenses accordingly aim to prevent such leakage. However, preventing file disclosure does not prevent users from recovering the functionality those files implement. This raises a…

Read at arXiv (research papers)

arXiv (research papers) Shihao Weng SI

AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection

Prompt injection remains a critical threat to LLM agents, yet existing defenses treat each task as a self-contained problem, independent of previous encounters. In practice, user requests are often underspecified: they describe the desired outcome without fully specifying acceptable behavior. An injection can exploit this ambiguity, causing the agent to complete the task in a way the user would reject. As the user's expectations become clearer through concrete cases, a…

Read at arXiv (research papers)

arXiv (research papers) Jia-Chen Zhang SI

Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents

Computer-use agents (CUAs), which empower large language models to autonomously operate operating systems and the web, are increasingly vulnerable to indirect prompt injection attacks. A widely adopted defense is the human-in-the-loop paradigm, in which the agent pauses for explicit user confirmation before executing sensitive operations. While effective against conspicuously high-harm attacks, this defense offers little protection against what we term Invisible Ink Threats…

Read at arXiv (research papers)

arXiv (research papers) Neha Nagaraja SI

When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems

Large language models are increasingly integrated into autonomous robotic systems for task planning and control, but this integration exposes them to prompt injection attacks that can lead to unsafe decisions and physical harm. Multi-agent settings increase the risks through cross-agent contamination and broader attack surfaces. In this paper, we evaluate prompt injection attacks against an LLM-based multi-agent robotic system, considering both direct injections into task…

Read at arXiv (research papers)

arXiv (research papers) Jianshuo Dong SI

Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden States

Agentic LLMs are vulnerable to indirect prompt injection (IPI) attacks, e.g., malicious side-tasks hidden in external tool results. While many efforts have sought to address this threat, little is known about the internals of agentic LLMs when they are exposed to IPI attacks. For simplicity, we refer to this condition as IPI exposure. In this paper, we study IPI exposure from three perspectives. (1) Probing: Across eight models, including the 753B-parameter GLM-5.2 and the…

Read at arXiv (research papers)

arXiv (research papers) Mingxiao Liu Hangzhou Dianzi University SI

Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agents

Large Language Model (LLM)-driven multimodal agents are increasingly deployed to execute autonomous tasks via continuous audio interaction. While this paradigm enhances interaction naturalness, it introduces a critical yet under-explored attack surface, as audio inputs inevitably contain environmental noise beyond user control. In this paper, we investigate concurrent audio prompt injection attacks targeting multimodal agents. Distinct from traditional acoustic attacks on…

Read at arXiv (research papers)

Simon Willison SI

AI Worming through Word

AI Worming through Word Neat new prompt injection variant by Håkon Måløy, who found a way to upgrade prompt injection attacks against Microsoft Word to full self-replicating worms: An attacker places hidden instructions in a document that is later used as source material in Copilot for Word. Copilot may interpret those instructions as part of the user’s request, causing it to manipulate the document being drafted or edited. Copilot may then also copy the hidden instructions…

Read at Simon Willison

arXiv (research papers) Eric Wallace SI

GPT-Red: Automated Red Teaming via Self-Play at Scale

We introduce \textbf{GPT-Red}, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs. The goal of this model is to evaluate and improve the robustness of our production systems. To this end, we use it to adversarially train GPT-5.6, our most robust model to prompt injections to date. To create GPT-Red, we design a scalable self-play algorithm where the model is tasked with attacking a diverse population of…

Read at arXiv (research papers)

arXiv (research papers) Arseny Kravchenko SI

APPA: Recoverable Information-Flow Control for Real-World LLM Agents

LLM agents deployed in practical workflows routinely mix private context, untrusted tool and web outputs, and external side effects. While information-flow control (IFC) provides structural defenses against prompt injection, data exfiltration, and confused-deputy attacks, conventional IFC relies on monotone taint tracking that either over-blocks benign operations or permanently strands downstream execution once an agent ingests unvetted data. We present APPA (Agentic…

Read at arXiv (research papers)

arXiv (research papers) Max Landauer SI

Just Testing, Move Along: Evasion of LLM-based System Log Interpretation by Prompt Injection

Large Language Models (LLMs) are increasingly integrated into Security Operations Center (SOC) workflows, where they support analysts in tasks such as the interpretation of system logs. However, the ability of LLMs to directly process untrusted textual input also introduces new attack surfaces. In particular, attackers can inject contextual information or explicit instructions into log entries in order to influence how malicious activity is interpreted by the model. Despite…

Read at arXiv (research papers)

arXiv (research papers) Zhaoxi Zhang SI

Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agents

Long-lived AI agents increasingly evolve after deployment by retaining experience, acquiring skills and tools, revising workflows, delegating work, and moving across task phases. This improves adaptation but creates a distinct authorization problem. Tool-enabled agents can turn model errors and prompt injections into consequential external actions; when evolution occurs under a live grant, the subject exercising that authority or the context in which it acts may no longer…

Read at arXiv (research papers)

Simon Willison SI

Quoting Boris Cherny

More than any of these eval scores, what is most exciting to me is something else: Opus 5 is our least prompt injectable model yet. It is a bit buried in the system card, but across PI evals and red teaming, Opus 5 is very hard to prompt inject successfully. — Boris Cherny, here's that System Card section, page 73 Tags: ai, prompt-injection, generative-ai, llms, anthropic, claude, boris-cherny

Read at Simon Willison

arXiv (research papers) Jay Barach SI

ResumeShield: Channel Separation and an Open Benchmark for Indirect Prompt Injection in AI Resume Screening

An AI resume screener reads a document supplied by the person it is evaluating, inverting the usual trust relationship between an assessor and the material it assesses. Candidates exploit this by concealing instructions inside a resume using white text, zero font size, hidden elements, markup comments, document metadata, or zero width characters. A human reviewer sees nothing, while a naive extraction pipeline places the concealed text into the model prompt, where it is read…

Read at arXiv (research papers)

arXiv (research papers) Vincent Siu SI

Agent Security Needs Redefinition through a Holistic Framework

Agent security is widely treated as a question about action content. Defenses ask whether an instruction looks malicious. Benchmarks ask whether an agent performs a harmful sounding action. \textbf{We argue that agent security is fundamentally a contextual problem, and that the current content based framing systematically misdefines it.} A command to ``delete user data'' might be a routine administrative request or a prompt injection attacking production systems, and the…

Read at arXiv (research papers)

arXiv (research papers) Yedidel Louck SI

Protocol-Level Attacks on Agentic Commerce Platforms: A Cross-Platform Taxonomy, AIP-Bench, and Unified Defense

Agentic commerce platforms let AI agents autonomously discover services, move payments, and wield user credentials on their users' behalf, and they already handle real money. Their security has so far been studied almost entirely at the level of the AI model, through prompt injection and misalignment. We show that the more consequential risks lie one layer down, in the protocol between agents and commerce services. There, vulnerabilities are structural : exploitation is…

Read at arXiv (research papers)

arXiv (research papers) Or Zion Eliav SI

Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents

Traditional pentesting uses reconnaissance at each step to uncover unseen weaknesses, build stronger attacks, and advance the objective; we argue that AI agents require the same treatment. We formalize agent reconnaissance by modeling the process and identifying the knowledge assets it seeks to extract: what they are, how they are used, and which agent weaknesses they exploit to give adversaries leverage in indirect prompt injection attacks. We instantiate these insights in…

Read at arXiv (research papers)

arXiv (research papers) Zhanhao Hu SI

Twin Agent: Context Residual Compression for Privilege Separated Agents

Large language model (LLM) agents are vulnerable to security risks, such as prompt injection attacks from untrusted context that manipulate downstream reasoning and tool use. Existing secure-by-design approaches mitigate this risk by separating untrusted observations from privileged execution and careful control of information flow, but often degrade utility and require extensive task-specific engineering. We thus propose Twin Agent, a general privilege separation design…

Read at arXiv (research papers)

arXiv (research papers) Akansha Shukla SI

Data Leakage Prevention in Agentic Applications via Preemptive Hardening

Agentic systems integrate LLM driven planning with interfaces to external tools, making data leakage and tool misuse feasible via instruction/data boundary failures and prompt injection attacks. Enforcing required controls consistently is particularly challenging in workflows spanning many codebases and heterogeneous agents. To address this challenge in multi agentic systems, we present a pre-deployment pipeline for scanning, hardening, and validation of agentic…

Read at arXiv (research papers)

arXiv (research papers) Xinting Liao SI

CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization

Textual Collaborative Prompt Optimization (TCPO) extends TextGrad (Yuksekgonul et al., 2025) to a decentralized setting by allowing multiple clients to jointly improve prompts for large language models (LLMs) while keeping their data locally. Its reliance on free-form textual updating and aggregation introduces a new and largely unexplored attack surface, i.e., malicious instructions can be injected into local prompts and propagated through server-side prompt aggregation…

Read at arXiv (research papers)

arXiv (research papers) Devina Jain SI

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. We present a 21-scenario benchmark for adaptive cross-session attacks against fresh-session LLM defenders: an autonomous LLM attacker observes prior defender responses and pivots across rounds, while each defender response is evaluated as a fresh interaction. A controlled 3 x 3 attacker-defender matrix contains 945 battles. Restricting scoring to the first round yields…

Read at arXiv (research papers)

arXiv (research papers) Xingfu Zhou SI

Salience Induction against Multi-Hop RAG Agents: Threat and Defense

Agentic retrieval-augmented generation (RAG) systems increasingly retrieve external evidence and orchestrate tools for knowledge-intensive applications. In Multi-Hop question answering, agents chain facts across documents. Existing defenses focus on content poisoning, which injects false facts, and prompt injection, which embeds directives. We identify a third attack surface: the salience channel, through which fact position, emphasis, framing, and semantic proximity can…

Read at arXiv (research papers)

arXiv (research papers) Yu Cui SI

Refusal is Not Safety! Benchmarking Latent Safety Risks of LLM-Driven Content Humorization

Safety defenses for large language models (LLMs) have been extensively studied, with existing approaches focusing on attack detection and refusal mechanisms. Such fixed-form direct refusal strategies may introduce the risk of prefix injection attacks. Recent work has explored a new direction that leverages humor as an indirect refusal mechanism to mitigate over-refusal in jailbreak scenarios and reduce prefix injection risks. However, this approach implicitly assumes that…

Read at arXiv (research papers)

Embrace The Red Security

From Indirect Prompt Injection to DNS Exfiltration in macOS Terminal

This is a follow-up to my previous Terminal DiLLMa research, and there is a positive outcome: Apple fixed a macOS Terminal behavior that enabled a DNS-based data exfiltration technique. DNS Requests via ANSI Escape Codes David Leadbeater originally discovered an interesting behavior in the macOS Terminal app that allowed a special sequence of ANSI escape codes to issue DNS requests. In short, this triggered a DNS request from the macOS Terminal app:

Read at Embrace The Red

arXiv (research papers) Soham Gadgil SI

Bad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems

A growing class of agentic systems maintain persistent state across sessions through memory files, behavioral preferences, and knowledge bases. While this makes agents more useful and self-improving, it also creates a new attack surface for prompt injections in which malicious instructions can be embedded within persistent files and influence future behavior. In this work, we study prompt injection attacks in memory-based agentic systems using a sandboxed synthetic…

Read at arXiv (research papers)

arXiv (research papers) Rabimba Karanjai SI

Context Contamination in LLM Analysis of Network Security Logs: Poison with Passive Prompt Injection and Mitigation Evaluation

Large Language Models are increasingly deployed in Security Operations Centers for log analysis tasks including summarization, alert triage, and threat investigation. These systems ingest logs from external-facing services and process network logs as natural language contexts to generate security insights. We demonstrate that this architectural pattern introduces a critical vulnerability: adversaries can embed prompt injection payloads in log-generating fields that persist…

Read at arXiv (research papers)

arXiv (research papers) Sanket Badhe SI

Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation

Reusable skills are becoming a fundamental building block of Large Language Model (LLM) agents, enabling capabilities to be packaged, shared, and reused across diverse applications. However, existing security research primarily focuses on prompt injection and runtime execution, leaving security risks throughout the broader skill lifecycle largely unexplored. In this paper, we present SkillSec-Eval, a lifecycle-aware framework for systematically evaluating the security of…

Read at arXiv (research papers)

arXiv (research papers) Junhui Wang SI

PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis

Large language models (LLMs) are increasingly deployed as purpose-specific agents to handle domain-specific tasks such as customer service and code generation. These agents are expected to comply with not only generic safety guardrails but also purpose-specific restrictions tailored to their designated roles. Such additional restrictions enlarge the attack surface, particularly to prompt injection (PI) attacks. To defend against such attacks, existing detection methods…

Read at arXiv (research papers)

arXiv (research papers) Huihao Jing SI

Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions

The capability of LLM agents to function as the ``brain'' of a system fundamentally expands the scope of analysis beyond a standalone model. Consequently, safety is no longer only about input--output content alignment. It also concerns system behavior and real-world execution outcomes. However, the current literature is fragmented across attack types, applications, and benchmarks. This makes it hard to explain why failures such as prompt injection, tool misuse, and memory…

Read at arXiv (research papers)

arXiv (research papers) SingGuard Team SI

SingGuard-NSFA: Extensible Guardrails for Agentic AI via Generative Reasoning and Real-Time Classification

We present nsfaguard, a guardrail framework for securing agentic AI systems against operational threats, such as prompt injection, sensitive information extraction, malicious code requests, dangerous tool misuse, and resource exhaustion. We first introduce the NSFA taxonomy, which organizes 185 risk variants into a CIA-triad-grounded hierarchy and is cross-validated against three well-established OWASP guidelines. Based on this taxonomy, we construct a benchmark suite…

Read at arXiv (research papers)

arXiv (research papers) Ruksat Khan Shayoni SI

NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations

Tool-using large language model (LLM) agents are attractive for network operations, but tickets, alerts, logs, runbooks, and ChatOps messages can carry indirect prompt injections. We present NetInjectBench, a 130-scenario benchmark that separates untrusted artifact text, trusted policy metadata, and evaluation labels for network-operation tool use. The sample contains 40 benign, 40 weak-attack, 40 strong-attack, and 10 approved high-impact change scenarios; each is evaluated…

Read at arXiv (research papers)

Older stories