Results for “Agent”

arXiv (research papers) Aniruddh Pramod SI

Towards a Unified Misuse Monitoring Benchmark

LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request is split into innocuous sub-requests, and prompt injection attacks, where a compromised tool delivers a malicious instruction. Existing evaluations treat these threats separately and ask whether a trajectory is harmful, rather than when it becomes harmful. We propose monitoring the agent's responses, where its actions are…

Read at arXiv (research papers)

arXiv (research papers) Théo Lasnier SI

TrustMI: Causally controlling how assistants trust their users

Large Language Model (LLM) assistants routinely decide whether they can trust users and third parties whose competence, intentions, and integrity they cannot verify. This uncertainty matters for safety, as trusting the wrong party can lead an agent to comply with harmful requests or act on malicious instructions encountered during tool use. To study this problem, we define trust as an assistant's willingness to accept vulnerability to the actions of another party and ask…

Read at arXiv (research papers)

arXiv (research papers) Tural Hagverdiyev SI

Compromise Is Not Consequence: Evaluating Task-Scoped Authorization in LLM Agents with Paired Replay

A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-constrained, and Open Policy Agent conditions. The frozen main experiment uses 128 scenarios across four tool domains and five local model configurations. Among…

Read at arXiv (research papers)

arXiv (research papers) James Peters-Gill SI

Can CaMeLs Talk? Securing Multi-Agent Systems Against Indirect Prompt Injection Attacks

Indirect prompt injection attacks - malicious instructions embedded in content processed by large language models - remain a major obstacle to safely deploying tool-using agents. CaMeL [Debenedetti et al., 2025] mitigates this threat for an individual agent by separating trusted control flow from untrusted data and enforcing capability-based security policies at runtime. In this work, we investigate whether CaMeL's security guarantees compose in hierarchical multi-agent…

Read at arXiv (research papers)

arXiv (research papers) Zhe Yu SI

Readable Before Actionable: Causal Tracing of Indirect Prompt Injection

Indirect prompt injection causes LLM agents to follow commands embedded in external data. A probe may distinguish instructions from data without identifying a state edit that changes the next action. We study this gap through counterfactual role probes, component-wise activation patching, and separate interventions on AgentDojo trajectories. Role decoding survives changes in content and format. In controlled Qwen tests, it precedes strong tool-choice effects from patches…

Read at arXiv (research papers)

arXiv (research papers) Rui Wang SI

Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents

Proactive personal agents increasingly decide what to recommend, how to personalize advice, and what follow-up assistance to offer, creating a new user-decision attack surface for provider-side indirect prompt injection. We show that an external provider need not access private user context, compromise the agent, or gain additional permissions: by controlling only content associated with its own target, it can redirect an otherwise benign agent to advance that target…

Read at arXiv (research papers)

arXiv (research papers) Jingkai Liu SI

Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection

Long-horizon agents consume external content, invoke tools, and modify persistent state. Indirect prompt injection can exploit task-specific context, propagate across causally connected stages, and alter a consequential action while the workflow continues; we term this staged prompt injection. We build an automated, feedback-guided attack generation pipeline and apply it to Claude Code and Codex in their native runtimes. The confirmed attacks span eight workflow scenarios…

Read at arXiv (research papers)

arXiv (research papers) Xinran Zheng SI

APEX: Active Protection at Execution Boundaries for LLM Agents

Indirect prompt injection (IPI) hides adversarial instructions in content that large language model (LLM) agents read at runtime. As agents compose heterogeneous capability units, including Tools, MCP servers, and Skills, the carriers of injection multiply, and defenses built to recognize attack patterns fall behind them. We instead shift defense from covering attack patterns to one stable point: whatever the carrier and however the injection propagates, harm materializes…

Read at arXiv (research papers)

Committee to Protect Journalists CPJ Staff

Cuban content creator sentenced to 2 years of ‘restricted freedom’ for filming agent

Miami, October 3, 2026—The Committee to Protect Journalists condemns a Havana court’s Thursday sentencing of influencer and content creator Anna Sofía Benítez, 21, and her mother, Caridad Silvente, to two years of “restricted freedom” after they were found guilty of supposedly violating a Ministry of the Interior (MININT) agent’s privacy. Outlets reported that the agent was upset...

Read at Committee to Protect Journalists

arXiv (research papers) Jairo Gudiño-Rosero SI

Large Language Models and Augmented Democracy

Artificial intelligence enables computational agents to represent political preferences and take part in collective decision-making. In this thesis, I investigate the opportunities and challenges of digital twins (DTs) based on Large Language Models (LLMs) as intermediaries in augmented democracy, focusing on individual preference representation, collective representation of political organizations, and the vulnerability of those representations to attackers. First, using…

Read at arXiv (research papers)

arXiv (research papers) Zixuan Wang SI

Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience

Large language model (LLM) agents are increasingly deployed in tool-augmented environments, but their reliance on external inputs makes them highly vulnerable to prompt injection attacks that can hijack task objectives. Existing safety alignment methods rely on static expert trajectories or preference optimization, limiting their ability to generalize to adaptive attack patterns. In this work, we propose Self-Reflection Fine-Tuning (SRFT), a training framework that enables…

Read at arXiv (research papers)

arXiv (research papers) Debeshee Das SI

Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough

Memory poisoning attacks on LLM agents typically assume an external adversary who plants content in the agent's persistent memory to steer its behavior. We instead study, with no adversary involved, whether a misaligned agent can write a goal it cannot yet act on to persistent memory, so that a future aligned agent carries it out when the opportunity arises. We investigate this threat, which we refer to as self-propagation of misalignment, across 20 different scenarios…

Read at arXiv (research papers)

arXiv (research papers) Zhuowen Liu SI

Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including…

Read at arXiv (research papers)

arXiv (research papers) Bijeeta Pal SI

Persona Guardrail: A Production-Grade Defense Framework for Agentic Systems

Large language model-based agents are increasingly deployed to perform domain-specific tasks by interacting with enterprise knowledge, tools, and external services. Existing runtime guardrails primarily target prompt injection and other attack-specific behaviors under a black-box threat model, but provide limited guarantees that agents operate within their intended functionality. As a result, production agents remain vulnerable to malicious requests and out-of-domain queries…

Read at arXiv (research papers)

arXiv (research papers) Giulio Zingrillo SI

Securing Computer-Use Agents Against Branch Steering Attacks

Modern Computer Use Agents (CUAs) directly interact with graphical user interfaces and execute third-party web tools, exposing them to indirect prompt injection across every rendered page and tool response. While the Dual-LLM pattern is the primary system-level architecture offering formal security guarantees - using an isolated Planner LLM (P-LLM) to fix execution paths before processing untrusted inputs via a Quarantined LLM (Q-LLM) - these guarantees break down in…

Read at arXiv (research papers)

arXiv (research papers) Simhadri Podala Narasimha SI

Containing the Autonomous Operator: A Defense-in-Depth Framework and Reference Architecture for Securing AI Agents on Kubernetes

Large language model (LLM) agents are moving from chat interfaces into infrastructure operations, where they read telemetry, call tools, generate and execute code, and change the state of production Kubernetes clusters. This collapses a boundary that conventional cloud-native security assumes: the boundary between data and control. Content that an agent merely reads (a log line, a ticket, a tool description) can redirect what it does. This paper argues that the model must…

Read at arXiv (research papers)

arXiv (research papers) Elad David SI

Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild

LLM agents increasingly rely on activation probes as runtime monitors for prompt injection, jailbreaks, and unsafe requests, reading the model's own hidden state to catch a harmful input before the agent acts on it. A cheap, increasingly common move, borrowed from LLM-as-judge prompting, is to append a short classification instruction after the user's turn and read the probe at that point, to sharpen it: the instruction asks the model to represent the incoming request as a…

Read at arXiv (research papers)

Committee to Protect Journalists CPJ Staff

Cuban journalist forced to flee island to Brazil following threats by state agents

Miami, October 1, 2026—The Committee to Protect Journalists condemns the forced exile of Cuban journalist Austin Llerandi, director of the community newsletter Amanecer Habanero, who fled the island on Tuesday and is currently in Brazil. Llerandi left the island with his pregnant wife after Cuban State Security agents threatened him with prison time if he did...

Read at Committee to Protect Journalists

arXiv (research papers) Alessandro Pegoraro SI

The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetching

With the increasing capabilities of Large-Language-Models (LLMs) and LLM-based agents, users are increasingly using them to solve everyday problems, such as answering e-mails or providing programming support. Existing work has extensively investigated security and privacy risks, such as prompt injections and the disclosure of sensitive data to chatbot providers. While various solutions were developed to address these risks, including input structuring to prevent prompt…

Read at arXiv (research papers)

Hacker News raorajat007 SI

Prompt Injection Firewall for AI APIs and Agent Tool Calls

Article URL: https://github.com/rajatrao/jevshield Comments URL: https://news.ycombinator.com/item?id=49916630 Points: 1 # Comments: 0

Read at Hacker News

arXiv (research papers) Birk Torpmann-Hagen SI

Memetic Trojans: Social Contagions as Carriers of Adversarial Payloads in Agent Networks

Autonomous large language model (LLM) agents increasingly interact in network environments where adversarial content can propagate between agents. Known attacks include agent worms, which spread through self-replicating prompt injections or configuration compromises. We introduce \emph{memetic trojans}, a distinct class of network-mediated attack that exploits agents' tendencies to retransmit and amplify content. Unlike agent worms, whose propagation is adversarially…

Read at arXiv (research papers)

arXiv (research papers) Jieke Shi SI

Aletheia: Permission-Minimality Testing for Coding-Agent Rules

Coding agents powered by Large Language Models (LLMs) read repository rules (e.g., AGENTS.md and Cursor rules) to learn project conventions and development procedures. An attacker can exploit the same files to mislead the agent into requesting excessive permissions or performing unrelated operations, such as accessing credentials or transferring data. We present Aletheia, a framework for permission-minimality testing of coding-agent rules. It tests whether the permissions…

Read at arXiv (research papers)

arXiv (research papers) Yuelin Han SI

Measuring LLM Agent Defenses by Layer: GDA Measurement and an Envelope-Layer Attack-Defense Model

Agent interaction protocols such as ACP and A2A let a remote peer deliver a task the receiving agent treats as legitimate, opening a channel for indirect prompt injection. Evaluating and defending such agents is difficult because the attack success rate (ASR) cannot distinguish an LLM recognition failure from an agent-layer block, and so cannot indicate which layer to strengthen. We contribute three artifacts. (1) GDA Measurement, a red-team method that captures the raw…

Read at arXiv (research papers)

arXiv (research papers) Yunjia Zheng SI

Towards Efficient HPC Systems for Agents: Challenges and Opportunities

Coding agents have become real users of high-performance computing (HPC) systems, yet today's HPC abstractions, interfaces, and policies remain designed for human-driven workflows. In our measurement, users running coding agents are only 19.5% of the observed population, but account for 55.8% of job submissions, 29.1% of CPU core-hours, and 42.7% of GPU-hours. Agents are not simply faster humans. They issue commands at 20.8x the human rate, decompose work into fine-grained…

Read at arXiv (research papers)

arXiv (research papers) Yanjie Li SI

ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agents

Tool-using LLM agents remain vulnerable to indirect prompt injection because trusted instructions and untrusted observations share one context, allowing malicious content to steer consequential input-filtering defenses. Multi-path consensus defenses still leave a high attack success rate because they examine content or aggregated outputs rather than authorizing effects, especially for the within-tool attack, which preserves the intended tool but manipulates its arguments…

Read at arXiv (research papers)

arXiv (research papers) Dongxu Cui SI

ContractWarden: Kernel-Enforced Damage Boundaries for AI Agents via Human-Authorized Contracts

Large language model agents can execute commands, create subprocesses, and directly access files and networks, allowing prompt injection or planning errors to become operating-system side effects. We present ContractWarden, a Linux reference monitor that enforces a human-authorized damage boundary without trusting the agent or its policy suggestions. A model may propose a tri-state asset contract - allow, deny, or no_egress - but a human makes the final choice. An execution…

Read at arXiv (research papers)

arXiv (research papers) Zonghao Ying SI

pikit: A Composable Toolkit for Indirect Prompt Injection Research and Evaluation

Indirect prompt injection embeds malicious instructions within external content retrieved by LLM-based agents, altering target behavior without user authorization. We introduce pikit, a research toolkit designed to systematically evaluate these threats across three core dimensions: attacks (13 methods), channels (16 carriers across text and file modes), and defenses (9 prevention strategies and 3 offline detection baselines). Built on a decorator-based registry, pikit…

Read at arXiv (research papers)

arXiv (research papers) Minh Nhat Le SI

Self-Evolving Defense: Continual Security Policy Learning for LLM Agents

Large language models (LLMs) increasingly power agents that access sensitive information, use external tools, and modify software repositories. Although these capabilities offer substantial benefits, they also create security risks such as jailbreaks, prompt injection, and vulnerable code generation. Existing defenses often require retraining, fail to adapt to evolving attacks, or address only a single threat pattern. To address these limitations, we propose Self-Evolving…

Read at arXiv (research papers)

arXiv (research papers) Michael Lee SI

Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?

Agentic systems are now being widely used to orchestrate tools and reason over long contexts. However, the improving capabilities of the large language models powering these agents also create new attack surfaces for indirect prompt injection. In particular, an attacker may not need to place a complete malicious instruction in retrieved content if the agent can reconstruct the objective from incomplete fragments distributed across a long context. In this work, we introduce…

Read at arXiv (research papers)

arXiv (research papers) Mark Russinovich SI

CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering

Indirect prompt injection makes an LLM agent treat untrusted retrieved text as instructions. We present CounterSteer, an inference-time defense that suppresses this behavior inside the model. Per model, a five-step recipe fits a residual-stream direction from paired episodes differing only in whether an embedded instruction is followed, and retains it only if it passes pre-specified causal and capability gates. At deployment, the direction is subtracted from every…

Read at arXiv (research papers)

Older stories