arXiv (research papers)

Recent stories from arXiv (research papers).

arXiv (research papers) Yichi Zhang SI

Secure Speculative Decoding for Large Language Models

Speculative decoding accelerates inference for a large language model (LLM), referred to as the \emph{target model}, by first using a smaller model, referred to as the \emph{draft model}, to generate candidate tokens and then verifying them with the target model for acceptance or rejection. Prior studies primarily focused on the efficiency-utility trade-off of speculative decoding, e.g., lossy speculative decoding, leaving its security implications largely unexplored. In…

Read at arXiv (research papers)

arXiv (research papers) Niveen O. Jaffal SI

RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems

Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content. We introduce RAG-PIBench, a benchmark for RAG-style prompt-injection detection containing 4,876 contextual examples across frozen train, validation, and protected-test splits. Using a leakage-aware construction pipeline and strict evaluation protocol, we compare keyword-based, semantic-reference, TF-IDF, and transformer-based detectors. DistilBERT achieves…

Read at arXiv (research papers)

arXiv (research papers) Haneen Najjar SI

Surviving the Router: Optimizing Skill Injections for Retrieval and Execution

AI agents increasingly rely on modular third-party "skills" that are dynamically selected by skill routers to execute complex tasks. While recent studies highlight the threat of prompt injections embedded in these skills, existing evaluations often assume settings where the malicious skill is already selected for execution. We show that this assumption can substantially overestimate attack success. In realistic multi-skill environments, injected skills must first compete for…

Read at arXiv (research papers)

arXiv (research papers) Mohamed Dhouib SI

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure…

Read at arXiv (research papers)

arXiv (research papers) Aniruddh Pramod SI

Towards a Unified Misuse Monitoring Benchmark

LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request is split into innocuous sub-requests, and prompt injection attacks, where a compromised tool delivers a malicious instruction. Existing evaluations treat these threats separately and ask whether a trajectory is harmful, rather than when it becomes harmful. We propose monitoring the agent's responses, where its actions are…

Read at arXiv (research papers)

arXiv (research papers) Théo Lasnier SI

TrustMI: Causally controlling how assistants trust their users

Large Language Model (LLM) assistants routinely decide whether they can trust users and third parties whose competence, intentions, and integrity they cannot verify. This uncertainty matters for safety, as trusting the wrong party can lead an agent to comply with harmful requests or act on malicious instructions encountered during tool use. To study this problem, we define trust as an assistant's willingness to accept vulnerability to the actions of another party and ask…

Read at arXiv (research papers)

arXiv (research papers) Tural Hagverdiyev SI

Compromise Is Not Consequence: Evaluating Task-Scoped Authorization in LLM Agents with Paired Replay

A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-constrained, and Open Policy Agent conditions. The frozen main experiment uses 128 scenarios across four tool domains and five local model configurations. Among…

Read at arXiv (research papers)

arXiv (research papers) James Peters-Gill SI

Can CaMeLs Talk? Securing Multi-Agent Systems Against Indirect Prompt Injection Attacks

Indirect prompt injection attacks - malicious instructions embedded in content processed by large language models - remain a major obstacle to safely deploying tool-using agents. CaMeL [Debenedetti et al., 2025] mitigates this threat for an individual agent by separating trusted control flow from untrusted data and enforcing capability-based security policies at runtime. In this work, we investigate whether CaMeL's security guarantees compose in hierarchical multi-agent…

Read at arXiv (research papers)

arXiv (research papers) Zhe Yu SI

Readable Before Actionable: Causal Tracing of Indirect Prompt Injection

Indirect prompt injection causes LLM agents to follow commands embedded in external data. A probe may distinguish instructions from data without identifying a state edit that changes the next action. We study this gap through counterfactual role probes, component-wise activation patching, and separate interventions on AgentDojo trajectories. Role decoding survives changes in content and format. In controlled Qwen tests, it precedes strong tool-choice effects from patches…

Read at arXiv (research papers)

arXiv (research papers) Rui Wang SI

Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents

Proactive personal agents increasingly decide what to recommend, how to personalize advice, and what follow-up assistance to offer, creating a new user-decision attack surface for provider-side indirect prompt injection. We show that an external provider need not access private user context, compromise the agent, or gain additional permissions: by controlling only content associated with its own target, it can redirect an otherwise benign agent to advance that target…

Read at arXiv (research papers)

arXiv (research papers) Jingkai Liu SI

Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection

Long-horizon agents consume external content, invoke tools, and modify persistent state. Indirect prompt injection can exploit task-specific context, propagate across causally connected stages, and alter a consequential action while the workflow continues; we term this staged prompt injection. We build an automated, feedback-guided attack generation pipeline and apply it to Claude Code and Codex in their native runtimes. The confirmed attacks span eight workflow scenarios…

Read at arXiv (research papers)

arXiv (research papers) Shang Wang SI

Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use

Jev turns natural-language questions into typed answers and probabilities with low latency and cost, enabling applications to route requests and select tools. While this interface allows Jev to integrate naturally into application workflows as a decision layer, the security and privacy implications of this emerging use remain largely unexplored. To address this gap, we conduct the first systematic study of these implications using the official Jev API and NanoJev, a local…

Read at arXiv (research papers)

arXiv (research papers) Xinran Zheng SI

APEX: Active Protection at Execution Boundaries for LLM Agents

Indirect prompt injection (IPI) hides adversarial instructions in content that large language model (LLM) agents read at runtime. As agents compose heterogeneous capability units, including Tools, MCP servers, and Skills, the carriers of injection multiply, and defenses built to recognize attack patterns fall behind them. We instead shift defense from covering attack patterns to one stable point: whatever the carrier and however the injection propagates, harm materializes…

Read at arXiv (research papers)

arXiv (research papers) Jairo Gudiño-Rosero SI

Large Language Models and Augmented Democracy

Artificial intelligence enables computational agents to represent political preferences and take part in collective decision-making. In this thesis, I investigate the opportunities and challenges of digital twins (DTs) based on Large Language Models (LLMs) as intermediaries in augmented democracy, focusing on individual preference representation, collective representation of political organizations, and the vulnerability of those representations to attackers. First, using…

Read at arXiv (research papers)

arXiv (research papers) Zixuan Wang SI

Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience

Large language model (LLM) agents are increasingly deployed in tool-augmented environments, but their reliance on external inputs makes them highly vulnerable to prompt injection attacks that can hijack task objectives. Existing safety alignment methods rely on static expert trajectories or preference optimization, limiting their ability to generalize to adaptive attack patterns. In this work, we propose Self-Reflection Fine-Tuning (SRFT), a training framework that enables…

Read at arXiv (research papers)

arXiv (research papers) Debeshee Das SI

Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough

Memory poisoning attacks on LLM agents typically assume an external adversary who plants content in the agent's persistent memory to steer its behavior. We instead study, with no adversary involved, whether a misaligned agent can write a goal it cannot yet act on to persistent memory, so that a future aligned agent carries it out when the opportunity arises. We investigate this threat, which we refer to as self-propagation of misalignment, across 20 different scenarios…

Read at arXiv (research papers)

arXiv (research papers) Xiaolin Zheng SI

Learning Robust Personalized Prompts for LLM-Driven Sequential Recommendation

LLM-driven sequential recommendation formulates next-item prediction as autoregressive generation conditioned on natural-language prompts. However, minor wording changes in semantically equivalent prompts can cause substantial performance fluctuations, undermining robustness and requiring costly manual prompt engineering. Continuous prompt learning reduces template dependence but faces two interacting challenges: shared task-level instructions lack user-specific reasoning…

Read at arXiv (research papers)

arXiv (research papers) Adam Faulkner SI

CorrectGuard: Eyes-Off Correctness Estimation for Black-Box Security Guardrails

AI services increasingly rely on black-box security guardrails, yet privacy-preserving model auditing regimes often cannot measure how well these systems perform in both a human eyes-off production setting, which disallows human inspection of user input, and a machine eyes-off setting, which disallows model inspection of such input. We introduce CorrectGuard, an eyes-off correctness estimation framework for both settings, which involves an independent model-based evaluator…

Read at arXiv (research papers)

arXiv (research papers) Zhuowen Liu SI

Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including…

Read at arXiv (research papers)

arXiv (research papers) Bijeeta Pal SI

Persona Guardrail: A Production-Grade Defense Framework for Agentic Systems

Large language model-based agents are increasingly deployed to perform domain-specific tasks by interacting with enterprise knowledge, tools, and external services. Existing runtime guardrails primarily target prompt injection and other attack-specific behaviors under a black-box threat model, but provide limited guarantees that agents operate within their intended functionality. As a result, production agents remain vulnerable to malicious requests and out-of-domain queries…

Read at arXiv (research papers)

arXiv (research papers) Anubha Gupta SI

LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

Standard autoregressive (AR) models process high-level task instructions, state history, and transient tokens within a single shared sequence of tokens. Consequently, they lack the architectural mechanisms needed to isolate macro-objectives from context noise. To overcome this single-channel limitation, we introduce Latent-Steered Autoregressive (LS-AR), a dual-channel architecture that decouples continuous goal steering from discrete token decoding via FiLM conditioning. We…

Read at arXiv (research papers)

arXiv (research papers) Giulio Zingrillo SI

Securing Computer-Use Agents Against Branch Steering Attacks

Modern Computer Use Agents (CUAs) directly interact with graphical user interfaces and execute third-party web tools, exposing them to indirect prompt injection across every rendered page and tool response. While the Dual-LLM pattern is the primary system-level architecture offering formal security guarantees - using an isolated Planner LLM (P-LLM) to fix execution paths before processing untrusted inputs via a Quarantined LLM (Q-LLM) - these guarantees break down in…

Read at arXiv (research papers)

arXiv (research papers) Zheng Chen SI

SecJev: Bringing Security Expertise to System One Decision Models

Security workflows need models that turn complex observations and explicit policies into decisions. System One models introduced by Jev return typed predictions and probabilities; security specialization supplies the domain expertise behind those predictions. We introduce SecJev, to our knowledge the first family of Jev-style decision models specialized for security, spanning 0.8B to 9B parameters. Built on Kev's single-pass candidate scorer, SecJev learns Boolean, choice…

Read at arXiv (research papers)

arXiv (research papers) Simhadri Podala Narasimha SI

Containing the Autonomous Operator: A Defense-in-Depth Framework and Reference Architecture for Securing AI Agents on Kubernetes

Large language model (LLM) agents are moving from chat interfaces into infrastructure operations, where they read telemetry, call tools, generate and execute code, and change the state of production Kubernetes clusters. This collapses a boundary that conventional cloud-native security assumes: the boundary between data and control. Content that an agent merely reads (a log line, a ticket, a tool description) can redirect what it does. This paper argues that the model must…

Read at arXiv (research papers)

arXiv (research papers) Narek Maloyan SI

Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations

Large language models (LLMs) in production systems face prompt injections, trojans (backdoors), and manipulation of automatic quality metrics. This thesis develops models, methods, and algorithms for evaluating and improving LLM robustness to adversarial input sequence variations. We propose R_stab(f), a generative robustness metric based on the Jensen-Shannon divergence between per-step output distributions under small input perturbations. For localized attacks we prove…

Read at arXiv (research papers)

arXiv (research papers) Elad David SI

Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild

LLM agents increasingly rely on activation probes as runtime monitors for prompt injection, jailbreaks, and unsafe requests, reading the model's own hidden state to catch a harmful input before the agent acts on it. A cheap, increasingly common move, borrowed from LLM-as-judge prompting, is to append a short classification instruction after the user's turn and read the probe at that point, to sharpen it: the instruction asks the model to represent the incoming request as a…

Read at arXiv (research papers)

arXiv (research papers) Alessandro Pegoraro SI

The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetching

With the increasing capabilities of Large-Language-Models (LLMs) and LLM-based agents, users are increasingly using them to solve everyday problems, such as answering e-mails or providing programming support. Existing work has extensively investigated security and privacy risks, such as prompt injections and the disclosure of sensitive data to chatbot providers. While various solutions were developed to address these risks, including input structuring to prevent prompt…

Read at arXiv (research papers)

arXiv (research papers) Birk Torpmann-Hagen SI

Memetic Trojans: Social Contagions as Carriers of Adversarial Payloads in Agent Networks

Autonomous large language model (LLM) agents increasingly interact in network environments where adversarial content can propagate between agents. Known attacks include agent worms, which spread through self-replicating prompt injections or configuration compromises. We introduce \emph{memetic trojans}, a distinct class of network-mediated attack that exploits agents' tendencies to retransmit and amplify content. Unlike agent worms, whose propagation is adversarially…

Read at arXiv (research papers)

arXiv (research papers) Jieke Shi SI

Aletheia: Permission-Minimality Testing for Coding-Agent Rules

Coding agents powered by Large Language Models (LLMs) read repository rules (e.g., AGENTS.md and Cursor rules) to learn project conventions and development procedures. An attacker can exploit the same files to mislead the agent into requesting excessive permissions or performing unrelated operations, such as accessing credentials or transferring data. We present Aletheia, a framework for permission-minimality testing of coding-agent rules. It tests whether the permissions…

Read at arXiv (research papers)

arXiv (research papers) Yuelin Han SI

Measuring LLM Agent Defenses by Layer: GDA Measurement and an Envelope-Layer Attack-Defense Model

Agent interaction protocols such as ACP and A2A let a remote peer deliver a task the receiving agent treats as legitimate, opening a channel for indirect prompt injection. Evaluating and defending such agents is difficult because the attack success rate (ASR) cannot distinguish an LLM recognition failure from an agent-layer block, and so cannot indicate which layer to strengthen. We contribute three artifacts. (1) GDA Measurement, a red-team method that captures the raw…

Read at arXiv (research papers)

Older stories