Results for “Injection”

arXiv (research papers) Haoxuan Jia SI

The Window Is in the Reasoning: When LLM Agents Decide to Follow Prompt Injections

LLM agents that read untrusted content can be hijacked by prompt injections. Runtime monitors that try to catch a hijack before the agent acts, and benchmarks that score each task with a single run, share an assumption: whether an agent follows an injection is settled when the injection is read. We test this assumption with branching rollouts that replay an episode exactly up to the step at which the agent reads an injection and then resample the rest of the episode many…

Read at arXiv (research papers)

arXiv (research papers) Songping Wang SI

MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency Injection

Multimodal large language model (MLLM)-based mobile agents are rapidly evolving from conversational assistants into autonomous systems that perceive, reason over, and act on smartphone environments. This capability also exposes a new security risk: adversaries can embed malicious objectives in cross-application mobile content and steer agent behavior through indirect multimodal injection. Although prompt injection has been studied in desktop and web settings, its cross-modal…

Read at arXiv (research papers)

arXiv (research papers) Farhan Zarif SI

Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

Tool-using agents read untrusted external content, which exposes them to indirect prompt injection (IPI), where hidden instructions hijack the agent's tool calls. Benchmarks measure IPI security by the attack success rate (ASR), which treats every failed attack as a successful defense. To test this assumption, we audit 33,000 trajectories from eleven open-weight agents (2-14B parameters) on InjecAgent and AgentDojo. A label-blind LLM judge shows that most failed attacks are…

Read at arXiv (research papers)

arXiv (research papers) Xianglin Yang SI

From Approval to Abuse: Privilege Escalation Attacks on Harness-Protected Coding Agents

Coding agents use execution harnesses to constrain the consequences of prompt injection by enforcing permission boundaries. These systems deterministically intercept any out-of-sandbox action and prompt the user for approval. To reduce approval fatigue, users can approve allow-rules proposed by the agent, after which any future action matching the rule is allowed without a prompt. In this work, we show that this mechanism introduces a new risk: untrusted context can…

Read at arXiv (research papers)

arXiv (research papers) Zesheng Shi SI

Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

LLMs over external data face a per-instruction tension: a single document or tool return often interleaves legitimate directives with adversarial prompt injections, and the model must follow the former while rejecting the latter. Existing defenses suppress or trust all external instructions, collapsing the safety-utility frontier and forcing a choice between over-refusal and over-compliance. We argue the missing capability is \textbf{per-instruction credit attribution} and…

Read at arXiv (research papers)

arXiv (research papers) Fardin Ahsan Sakib SI

The Model Knows Better: Defending Medical Multi-Agent Systems Against Prompt Injection

Medical multi-agent systems, in which LLM agents discuss a case, are increasingly studied for clinical decision support. Prompt injection, text that an attacker adds to a model's input, can manipulate LLMs into wrong decisions. How it affects these systems, and how to protect them, remains unclear. Across two systems, two model families and three benchmarks, a single injected passage succeeds in up to 61% of cases, and collaboration does not consistently protect against it…

Read at arXiv (research papers)

Hacker News panoramas4good SI

Prompt injection in HTTP headers: what 217M web responses show

Article URL: https://www.signalscorps.com/blog/prompt-injection-in-http-headers/ Comments URL: https://news.ycombinator.com/item?id=50104310 Points: 96 # Comments: 17

Read at Hacker News

arXiv (research papers) Qianlong Lan SI

From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

Prompt-injection defenses are commonly evaluated as binary detectors, although deployed LLM applications must make operational decisions about whether to allow, block, or escalate potentially malicious inputs. Recent probabilistic decision models aim to support such decisions through calibrated confidence estimates, but their reliability under distribution shift remains uncertain. We evaluate a native probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers…

Read at arXiv (research papers)

arXiv (research papers) Thuy Dung Nguyen SI

INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems

LLM-based agents increasingly execute long-horizon, end-to-end workflows involving shell commands, code modification, dependency installation, and interaction with remote services. This growing autonomy expands their exposure to indirect prompt injection (IPI), where adversarial instructions embedded in external content are interpreted as legitimate task directives. Existing evaluations, however, often focus on isolated malicious strings, single-step tool calls, or narrowly…

Read at arXiv (research papers)

arXiv (research papers) Sarthak Gupta SI

Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection

Several recent studies propose detecting prompt injections from a language model's internal state. However, these methods do not measure a single security property. Residual-stream probes, attention scores, role projections, exposure probes, and concept readouts may capture injection presence, recognition, information routing, or behavioral influence. It is therefore unclear what differences in prompt-injection detection performance reveal about the internal signals these…

Read at arXiv (research papers)

arXiv (research papers) Chenxu Wang SI

AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper…

Read at arXiv (research papers)

arXiv (research papers) Kshitij Mishra SI

TRACER: Trace-Regularized Training Against Indirect Prompt Injection

Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an…

Read at arXiv (research papers)

arXiv (research papers) Dominik Seip SI

Towards Safe Autonomy for Computer Use

The safety of LLM-based agents currently rests on alignment training and safety classifiers, neither of which can rule out that a sophisticated indirect prompt injection (IPI) goes through and causes significant harm. Recent work therefore builds agents with information-flow guarantees that rely on structured tool interfaces to interact with the environment. Computer Use Agents (CUAs) act visually on the computer screen, which does not provide a boundary between trusted…

Read at arXiv (research papers)

arXiv (research papers) Kristoffer Hempel SI

Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents

Large language model (LLM) agents are becoming an integral part of the software development lifecycle, and the capabilities granted to them allow an agent to act with the user's authority on any resource it can access. The reach that gives an agent its utility is equally available to an attacker who can steer it. Anything an agent ingests can carry an injected instruction, and since the model does not reliably distinguish instructions from data, indirect prompt injection…

Read at arXiv (research papers)

arXiv (research papers) Kaan Kale SI

Prompt-Injection Probes Learn the Conversation, Not Just the Injection

Linear probes on a model's hidden states are used as cheap monitors for indirect prompt injection: they screen the tool outputs an agent reads and send suspicious ones to a slower second check, so every false alarm has a cost. But a probe learns whatever separates its training examples. The standard evaluation of injection probes can be passed without the model: conversation position and text lengths alone reach 0.91-0.98 AUROC. When each attack is compared with the same…

Read at arXiv (research papers)

arXiv (research papers) Yuetian Chen SI

BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views

Browser agents need to make decisions and execute actions on behalf of users while consuming untrusted webpages, creating opportunities for malicious instructions embedded in webpages to redirect legitimate tasks. We introduce BrowseSafe++, which extends BrowseSafe from 14,719 pages to 38,675 pairs of pages. Most of its prompt injections were observed in the wild on production websites. We keep each injection unchanged and rewrite the sensitive parts of the page around it to…

Read at arXiv (research papers)

Hacker News anon767 SI

Prompt injection classifiers are terrible security boundaries

Article URL: https://thecout.com/blog/injection/ Comments URL: https://news.ycombinator.com/item?id=50056871 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Anning Hu SI

StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge

Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as paper review and recommendation. However, LLM-as-a-Judge pipelines are vulnerable to score-hijacking prompt injection embedded in evaluated content, which corrupts scoring signals and distorts induced rankings. Existing defenses often enforce robustness by modifying the judge model itself, making safety dependent on a specific…

Read at arXiv (research papers)

arXiv (research papers) Rishabh Sinha SI

Exposure Is Not Commitment: Double-Stratified Evaluation of Latent Monitors for Tool-Using Agents

Tool-using language agents are vulnerable to indirect prompt injection, and a growing literature probes model internals to catch unsafe actions before they execute, with sharply conflicting results. We argue much of the conflict is an evaluation artifact, and propose a double-stratification protocol: separate injection exposure from behavioral commitment, in both the evaluation and the training population. On the primary Qwen AgentDojo grid, monitors trained on the naive…

Read at arXiv (research papers)

Hacker News oazizjon SI

Prompt injection through log files: what red-teaming our assistant found

Article URL: https://jl3-cloud.site/blog/prompt-injection-through-log-files/ Comments URL: https://news.ycombinator.com/item?id=50046522 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Michael Cardei SI

Only What You've Signed up For: Secure Workflows with Constrained Diffusion Agents

As agentic AI systems take on increasingly complex and consequential tasks, securing them against prompt injection becomes critical. Existing defenses often rely on detecting or reasoning about injected malicious instructions and can therefore remain vulnerable to adaptive attacks that optimize the injection itself. Diffusion language models (dLLMs), an emerging framework for language generation, offer a new opportunity for security enforcement: by iteratively refining their…

Read at arXiv (research papers)

arXiv (research papers) Josef A. Habdank SI

Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments

We ask whether a tool-using agent whose sole duty is to keep humans alive can be induced, by delayed prompt injection, to kill. The keep-alive duty is in force from the first watch, with the attack arriving later, appended to otherwise truthful telemetry the operator is required to read. Experiments are based on a NASA-derived BioSim lunar habitat simulation, where the AI operator controls the life-support systems. The attacker cannot falsify sensors or operate equipment: it…

Read at arXiv (research papers)

Hacker News manveerc SI

Prompt Injection Detection and Defense Tools for Enterprise AI Agents

Article URL: https://www.arcade.dev/blog/prompt-injection-detection-tools/ Comments URL: https://news.ycombinator.com/item?id=50032282 Points: 1 # Comments: 0

Read at Hacker News

arXiv (research papers) Daniel Commey SI

Names Are Not Objects: Object-Bound Confinement for Compromised AI Coding Agents

Coding agents read repositories and web pages that anyone can write, and then act with their user's authority. Prompt-injection defenses make it less likely that the model is steered but do not prevent it, so the system around the model must contain an agent that is already compromised. Many exploits against deployed coding agents share one cause: a check approves a name, such as a path, an argument vector, or a directory, and a later step maps that name to a different…

Read at arXiv (research papers)

Hacker News uimik SI

Prompt Injection will always be unsolved

Prompt injection is not solved. Not by them, not by anyone. If someone tells you they’ve solved it, they are either lying to you or they don’t understand the problem. There is no polite third option. Comments URL: https://news.ycombinator.com/item?id=50024585 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Luman Zhao SI

LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense

Large language models (LLMs) perform remarkably well on complex tasks, yet remain highly vulnerable to prompt injection attacks, where malicious instructions embedded in external data can override user intent. Existing defenses remain limited by model fine-tuning requirements, vulnerability to adaptive attacks, or reliance on brittle handcrafted prompts. We argue that a fundamental source of this vulnerability is the lack of an explicit representation of trust provenance. To…

Read at arXiv (research papers)

arXiv (research papers) Cathleen Qiao SI

An Exploration of Attacks on AI Agent Systems

As autonomous AI agents become more integrated into sensitive domains, it is crucial that single and multi-agent AI systems are secured. This paper investigates how prompt injection attacks can exploit both direct user inputs and inter?agent communication to leak sensitive information and trigger cross-agent infections. While our tests mostly highlight key vulnerabilities in single-agent systems, we also extend these attacks to multi-agent frameworks, investigating how…

Read at arXiv (research papers)

arXiv (research papers) William Hackett SI

BRANCH: Bypassing Multi-Scanner AI Guardrails

AI systems increasingly rely on Large Language Models (LLMs) as core reasoning engines, making them targets for prompt injection and jailbreaks. Guardrails monitor and validate model inputs and outputs, yet their isolated, task-focused detection leaves gaps in their classification making them susceptible to bypasses. In response, guardrail systems formed by multiple scanners have emerged that collaboratively detect different types of malicious instructions, whereby shared…

Read at arXiv (research papers)

Hacker News thewalnut124 SI

Prompt injection scanner that normalizes evasion tricks

Article URL: https://github.com/Aaronmike481/Aiscanner Comments URL: https://news.ycombinator.com/item?id=49995426 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Zitong Yao SI

AgentTracer: Tracing Indirect Prompt Injection Attack through Fine-Grained Intention-Execution Alignment

Large language model (LLM) agents interact with external resources to complete complex user tasks, exposing them to indirect prompt injection (IPI), where malicious instructions redirect agents toward attacker-intended tasks. Since IPI is difficult to defend against in real-world environments, post-incident tracing is essential for locating the injection source and reconstructing the attack chain. However, existing tracing methods primarily capture explicit control-flow and…

Read at arXiv (research papers)

Older stories