Results for “Injection”

arXiv (research papers) Minh Nhat Le SI

Self-Evolving Defense: Continual Security Policy Learning for LLM Agents

Large language models (LLMs) increasingly power agents that access sensitive information, use external tools, and modify software repositories. Although these capabilities offer substantial benefits, they also create security risks such as jailbreaks, prompt injection, and vulnerable code generation. Existing defenses often require retraining, fail to adapt to evolving attacks, or address only a single threat pattern. To address these limitations, we propose Self-Evolving…

Read at arXiv (research papers)

arXiv (research papers) Michael Lee SI

Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?

Agentic systems are now being widely used to orchestrate tools and reason over long contexts. However, the improving capabilities of the large language models powering these agents also create new attack surfaces for indirect prompt injection. In particular, an attacker may not need to place a complete malicious instruction in retrieved content if the agent can reconstruct the objective from incomplete fragments distributed across a long context. In this work, we introduce…

Read at arXiv (research papers)

arXiv (research papers) Mark Russinovich SI

CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering

Indirect prompt injection makes an LLM agent treat untrusted retrieved text as instructions. We present CounterSteer, an inference-time defense that suppresses this behavior inside the model. Per model, a five-step recipe fits a residual-stream direction from paired episodes differing only in whether an embedded instruction is followed, and retains it only if it passes pre-specified causal and capability gates. At deployment, the direction is subtracted from every…

Read at arXiv (research papers)

arXiv (research papers) Jie Zhang SI

Render Before Reading: Visual Rendering as a Prompt Injection Defense

Large language models are vulnerable to prompt injection attacks, where third-party adversarial content can hijack the model's behavior. In this paper, we study the role played by the adversarial data's input modality, and identify a systematic asymmetry: multimodal LLMs are more likely to follow adversarial instruction when they appear as text than when the same instruction is delivered through a non-textual channel (e.g., as an image). We hypothesize that this modality gap…

Read at arXiv (research papers)

arXiv (research papers) Yan Zhan SI

Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection

Prompt injection against LLM agents becomes much stronger when the injected instruction is wrapped in the model's own chat template. A forged template marker such as can reach the model either as a single reserved control token or as a sequence of ordinary subword tokens. The two decode to exactly the same text, and because tokenization runs on the server, the defender rather than the attacker decides which one the model receives. We use this to measure how much of the…

Read at arXiv (research papers)

arXiv (research papers) Rohit Saxena SI

Nudgeability: Reasoning Models Follow Confidence Signals Without Tracking Their Own Competence

Reasoning language models that can call tools must decide during inference whether to answer unaided or delegate. Any self-reflection mechanism for this must answer three questions: where the reflective signal comes from (verbal reports, output distributions, hidden states, a separate predictor), how it is presented to the model (numerical prediction, confidence token, prompt injection), and whether it changes the model's subsequent action. We isolate the third question. At…

Read at arXiv (research papers)

arXiv (research papers) Xiao Yang SI

CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents

Large language model (LLM)-based agents increasingly rely on external tools and content, exposing them to indirect prompt injection (IPI). This threat has motivated a wide range of defenses, among which training-based defenses are often regarded as most reliable. However, existing training-based defenses are typically optimized on a static distribution of explicit injections. They learn surface-form cues rather than the boundary between serving the user and obeying an…

Read at arXiv (research papers)

arXiv (research papers) Anmol Pandey SI

Certified Multi-Source Integrity for Structured Agent Actions

LLM agents increasingly take privileged, often irreversible structured actions, such as paying an invoice. They assemble each action from action-critical fields in documents and tool outputs that an adversary can corrupt, and indirect prompt injection can drive the model itself to extract attacker-chosen values. Current defenses gate on a source's trust label or certify free-text answer quality. None certifies the integrity of a coupled, policy-bound structured action under…

Read at arXiv (research papers)

arXiv (research papers) Chenlong Yin SI

Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning

Prompt injection is a leading security risk for LLMs and LLM-based applications such as agents. State-of-the-art red-teaming methods for prompt injection leverage reinforcement learning (RL) to train an attacker LLM to generate effective injected prompts. However, when targeting frontier LLMs such as GPT-6-Luna, a major challenge is the cold-start problem: every attack attempt by the attacker LLM fails and thus receives zero reward, providing no signal for learning. In this…

Read at arXiv (research papers)

Hacker News YehielAmor SI

Show HN: A deterministic tool-call gateway vs. prompt-injection classifiers

Article URL: https://github.com/Yehielamor/provenance-gate/blob/main/posts/01-provenance-vs-detection.en.md Comments URL: https://news.ycombinator.com/item?id=49866732 Points: 1 # Comments: 0

Read at Hacker News

arXiv (research papers) Patrick Kenney SI

API Secrets Should Never Become Tokens in the LLM's Vocabulary: A Threat Analysis of API Credential Handling in LLM Agent Systems and an Empirical Evaluation of a Vault-Mediated Execution Boundary

Tool-using large language model (LLM) agents turn credential hygiene from a storage problem into an execution-security problem. A key pasted into a prompt, or embedded in a system prompt or tool configuration, crosses from an authentication boundary into a data pipeline, where it may persist in conversation history, logs, memory stores, generated code, and error payloads. Prompt injection and excessive agency then convert passive disclosure into unauthorized action. This…

Read at arXiv (research papers)

Hacker News jonbaer SI

Self-replicating prompt injections exist

Article URL: https://alignment.openai.com/misalignment-reports/self-replicating-prompt-injections-exist/ Comments URL: https://news.ycombinator.com/item?id=49863790 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Ben Hagag SI

ORBIT: A Framework for Multi-Agent Safety and Security Evaluations

Multi-agent LLM systems are increasingly deployed for complex, long-horizon tasks or emerge as a natural consequence of agents interacting in the wild. Yet they give rise to significant safety and security risks: the flexible protocols that enable task generalization also expose novel threats, from cascading prompt injection to inter-agent collusion. Progress in defending against these threats has been slowed by a lack of shared empirical infrastructure, which forces bespoke…

Read at arXiv (research papers)

Hacker News Dikesa_27 SI

We benchmarked our prompt-injection detector against OWASP's LLM Top

Article URL: https://www.cerbereag.site/blog/detecting-prompt-injection-in-production Comments URL: https://news.ycombinator.com/item?id=49858304 Points: 1 # Comments: 1

Read at Hacker News

arXiv (research papers) Animesh Shaw SI

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

LLM agents that invoke privileged tools are vulnerable to indirect prompt injection (IPI), in which adversarial instructions embedded in retrieved data hijack the agent's actions. A growing body of work evaluates defenses against IPI, but the validity of that evaluation is rarely examined. We audit an IPI benchmark and its harness and identify four defect classes -- silent payload non-delivery, attack success scored by tool identity rather than arguments, false-rejection…

Read at arXiv (research papers)

Hacker News luispa SI

Prompt Injection in the Wild

Article URL: https://cybershujin.github.io/Prompt-Injection-in-the-Wild/ Comments URL: https://news.ycombinator.com/item?id=49851038 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Hanzhang Ma SI

MetaPermit: Scalable and Auditable Access Control for AI Agents via LLM-Inferred Meta-Attributes

The rise of autonomous AI agents equipped with tools has introduced significant security risks, ranging from unintended tool misuse to adversarial manipulation through Indirect Prompt Injection (IPI) attacks. In practice, deployed agent systems such as OpenAI Codex and Claude Code protect tool invocations through a combination of coarse-grained permission rules and LLM-based judgments about individual proposed actions. Both components, however, have important limitations…

Read at arXiv (research papers)

arXiv (research papers) Srikumar Subramanian SI

Crypto-bound identity-verified capability tokens for coordinating distributed AI agents: A proposal

The prospect of fully autonomous transactional agents did not appear on the horizon until the advent of high capability language models. With such models, the operational benefits of adaptive task orchestration and independent (but constrained) decision making are tantalizing for enterprises and individuals alike. However, each such agent carries with it a serious attack surface in the form of prompt injection which can compromise any soft "guard rails" that may have been…

Read at arXiv (research papers)

Hacker News ArcherL SI

Is indirect prompt injection still a big threat as models get more advanced?

Article URL: https://realarcherl.github.io/posts/prompt_injection/ Comments URL: https://news.ycombinator.com/item?id=49840178 Points: 3 # Comments: 3

Read at Hacker News

arXiv (research papers) Ahmad Hashmi SI

Prompt Injection Detection for Email Agents Through Attack Chain Modeling

Large language model email assistants are particularly vulnerable to indirect prompt injection because untrusted email content can be retrieved into the model context and influence subsequent tool use. Existing prompt injection detectors mainly formulate this problem as binary malicious text classification, which overlooks the important factor that harmful agent behavior often arises through a sequence of stages. We propose a detection framework that models this attack chain…

Read at arXiv (research papers)

arXiv (research papers) Qingyu Wu SI

ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation

Prompt injection can degrade benign task performance without eliciting harmful content. Yet many attack objectives depend on task labels or predefined target responses. We present ENDOPROMPT, a white-box method that learns utility-degrading prefixes from unlabeled instructions. Its generator takes the request text as input. Clean victim continuations serve as pseudo-references: local search identifies prefixes that reduce continuation likelihood, and preference fitting on…

Read at arXiv (research papers)

arXiv (research papers) Lukáš Brůna SI

Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs

Large Language Models (LLMs) consume and produce a single sequence of text; hence, if text can be added to the beginning of the LLM's response, i.e., an output prefix, then all subsequent tokens will be conditioned on it. This output-prefix attack technique is a cheap black-box prompt injection. Prior work has shown this type of attack can reliably jailbreak non-reasoning models. Most reasoning models add an intermediate scratchpad reasoning step before the assistant's final…

Read at arXiv (research papers)

Hacker News qrios SI

Meta Muse: Almost no prompt injection resistance

Article URL: https://neuromatch.social/@jonny/117324790823856750 Comments URL: https://news.ycombinator.com/item?id=49829888 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Daiki Chiba SI

ClaimMirage: When Self-Claims in Domain Names Change LLM Threat Judgments

Short claims such as not-phishing or official can change how a large language model (LLM) judges a domain name, without explicit prompt-injection commands. We study this manipulation as ClaimMirage: a name under inspection claims its own safety or approval. We analyze 622,080 judgments across 64 brands and five LLMs, comparing ten claims with length- and hyphen-matched controls in constructed brand-like names. Self-claims can substantially reduce or increase alerts…

Read at arXiv (research papers)

Hacker News northbridgedev SI

Can open-source prompt-injection detectors catch realistic AI agent attacks?

Article URL: https://github.com/rudratoshs/buried-injections Comments URL: https://news.ycombinator.com/item?id=49827019 Points: 8 # Comments: 4

Read at Hacker News

arXiv (research papers) Tiantong Wu SI

Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions

Most studies of prompt injection focus on generative agents, leaving their effects on models with schema-defined outputs unclear. We examine these effects in Jev, a non-generative decision model, using 510 reconstructed InjecAgent cases. Malicious content shifts action probabilities but rarely causes Jev to select the attacker's target. Override markers reduce this influence, while claims of contextual relatedness have small effects. Adaptive attacks using score feedback…

Read at arXiv (research papers)

arXiv (research papers) Reshabh K Sharma SI

Ajar: Measuring Open Privilege in Agent Defenses

A language model agent acts through the tools it is given. The data it reads while working on a task can redirect what it does with those tools. A growing set of techniques for safe and secure agent execution therefore sits between the agent and its tools, aiming to enforce access control, information flow or isolation at that boundary. Today these techniques are evaluated on agent-security benchmarks built around indirect prompt injection. Those benchmarks judge a defense…

Read at arXiv (research papers)

arXiv (research papers) Fernando Outeda SI

Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection

Large language models (LLMs) are increasingly deployed in production systems, raising concerns about their exposure to adversarial manipulation through prompt injection and jailbreak attacks. Classifier-based guardrails, such as Prompt Guard 2, are widely used as a first line of defense against such attacks, but their internal decision logic is largely opaque to both defenders and attackers. This paper presents an exploratory case study that applies explainable artificial…

Read at arXiv (research papers)

Hacker News Irsheidat SI

When "Review" Becomes Permission: A Prompt Injection Lab

Article URL: https://rsec.uk/insights/when-review-becomes-permission-a-prompt-injection-lab-mu5tsobx Comments URL: https://news.ycombinator.com/item?id=49766085 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Rudrendu Kumar Paul SI

Beyond Single-Model Injection: A Threat Model and Defense Architecture for Prompt Injection in Multi-Agent Systems

Existing prompt injection research focuses on single-model chatbot scenarios, where an attacker manipulates one LLM through crafted input. Multi-agent systems amplify this threat through three mechanisms absent from single-model settings: inter-agent message passing creates injection channels invisible to perimeter defenses, shared tool access enables privilege escalation across agent boundaries, and trust propagation allows a compromised agent to influence upstream…

Read at arXiv (research papers)

Older stories