Results for “Prompt”

arXiv (research papers) Yusuf Khalid Shire SI

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

Prompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under distribution shift, particularly on the benign side of the decision boundary, where false positives impose direct operational cost yet are seldom measured. We present PIDS-Bench, a frozen multi-axis benchmark that jointly evaluates attack detection and benign false-positive behavior at fixed thresholds, spanning in-distribution…

Read at arXiv (research papers)

arXiv (research papers) Bingzheng Wang SI

ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents

Large language model (LLM) agents interact with external environments through tool invocation, but tool outputs can also expose them to indirect prompt injection (IPI) attacks. Existing defenses mainly rely on prompt hardening, content filtering, pre-generated plans, or permission constraints. These approaches often struggle with complex tasks or over-sanitize external content, making it difficult to balance security and utility. The key challenge is therefore to preserve…

Read at arXiv (research papers)

arXiv (research papers) Donato Mecca SI

SkillSecurer: Detecting and Patching Prompt-Injection Vulnerabilities in AI Agent Skills

Agent skills extend AI agents with reusable instructions, scripts, and configuration, but are also open to new attacks to influence an agent's decisions and actions. To address these risks, we present SkillSecurer, a fully agentic framework for generating, detecting, localising, and remediating security risks in agent skills. Its red agent generates context-compatible injections across nine threat types while recording the exact modification; its blue agent analyses complete…

Read at arXiv (research papers)

arXiv (research papers) Minsun Shim SI

Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control

Personal AI agents built on large language models (LLMs) are increasingly given access to a user's private data and communications in order to provide personalized assistance. This access creates a persistent privacy risk: the agent must decide whether a given sensitive information should be disclosed to a particular party. Existing defenses address this by making the agent's backend LLM more privacy-preserving through stronger system prompts, training, or explicit…

Read at arXiv (research papers)

arXiv (research papers) Asif Pinjari SI

DriftNet: A Dual-Head Trajectory Transformer for Detecting and Localizing Prompt Injection in LLM Agents

When an indirect prompt injection succeeds against an LLM agent, the compromise is visible in the agent's own behavior: a benign prefix of tool calls, a poisoned observation, and a suffix of actions that serve the attacker. An operator needs three facts: where the attack entered, which steps it corrupted, and whether apparent poison was resisted. Existing systems return either a whole-trace verdict or a single unsafe index. We present DriftNet, a dual-head trajectory…

Read at arXiv (research papers)

arXiv (research papers) Zehua Zhang SI

No-Box Vulnerability Analysis: Description-only Detection of Indirect Prompt Injection Vulnerabilities in MCP Servers

Conventional vulnerability analysis relies on either system access or dynamic interaction, all of which may be unavailable to third-party analysts auditing closed-source, remotely hosted, critical in situ systems, or commercially gated software. Therefore, we propose a new paradigm of no-box vulnerability analysis in which neither access nor runtime interaction is available, and only functionality metadata is available. Such metadata defines the intended behavior of the…

Read at arXiv (research papers)

arXiv (research papers) Anna Gazani SI

Architecting the Secure AI-SOC: A Neurosymbolic Framework for Pipeline Integrity and Threat Mitigation

The integration of Large Language Models (LLMs) into Security Operations Centers (SOCs) streamlines threat intelligence but introduces critical vulnerabilities, notably indirect prompt injection via log poisoning. Adversaries exploit this vector to execute multistep ``promptware'' kill chains by embedding malicious payloads within system logs to hijack the LLM's operational logic. Securing this pipeline presents a dichotomy: deterministic defenses are computationally…

Read at arXiv (research papers)

Hacker News fourfire SI

Google: Attackers are using prompt injection against coding agents

Article URL: https://cloud.google.com/blog/topics/threat-intelligence/from-prompting-to-autonomy-the-evolution-of-adversarial-ai Comments URL: https://news.ycombinator.com/item?id=49628034 Points: 13 # Comments: 1

Read at Hacker News

arXiv (research papers) Ryan Lum SI

Kernel-Managed Shared Memory for System-Wide Personalization

AI systems become more useful when they can adapt to the people using them, but in multi-agent systems, useful context learned by one agent often remains unavailable to others. We present kernel-managed shared memory, a system-level abstraction in which specialized agents write structured, tagged memories while the agent-system kernel, not individual agents, governs retrieval, privacy enforcement, and prompt injection. We implement and evaluate this design on AIOS and…

Read at arXiv (research papers)

Hacker News gmays SI

Prompt Injection Through Tool Output Is Two Events (Your Screens Read One)

Article URL: https://www.armosec.io/blog/untrusted-tool-output-prompt-injection/ Comments URL: https://news.ycombinator.com/item?id=49618594 Points: 2 # Comments: 0

Read at Hacker News

arXiv (research papers) Viet K. Nguyen SI

An Experimental Evaluation of Multimodal Prompt Injection Attacks on Agentic AI Frameworks

Agentic AI frameworks let a language model plan, keep memory, and call tools that reach real files, mail, and services. Most of these agents also read images, which gives an attacker a way to put text into the agent's context without going through the user. We present MMPIBench, a reproducible benchmark that measures what happens next. It delivers a fixed set of attacks through six visual carriers (OCR text, overlays, EXIF metadata, QR codes, fake interfaces, and hybrids)…

Read at arXiv (research papers)

arXiv (research papers) Dimitrios Stamatios Bouras SI

Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents

Coding agents use system-level tools to read files, execute commands, and modify source code. Within the agent's sandbox, these tools often carry ambient authority: naming a resource is sufficient to act on it. Indirect prompt injection exploits this authority by placing instructions in repository files or tool output that cause the agent to perform actions the user did not request. We propose CapScope, a harness-level authorization mechanism that restricts tool use without…

Read at arXiv (research papers)

Journal articles (OpenAlex) Nicolás Ajzenman, Patricio Domínguez, Lucas M. Novaes · Journal of Public Economics Geopolitics

Electoral results affect beliefs about the past: Evidence from Chile

Can electoral outcomes shape voters' perceptions of the past? We argue that electoral outcomes provide informational cues that prompt voters to reinterpret prior conditions, shifting the benchmarks on which accountability rests. We focus on perceptions of crime, a valence issue that voters likely take into account when choosing any candidate. We propose that perceptions of crime change immediately after an election, as voters reinterpret prior conditions through the lens of…

Read at Journal articles (OpenAlex)

Hacker News fourfire SI

ASCII smuggling crosses over from AI prompt injection to phishing evasion

Article URL: https://www.microsoft.com/en-us/security/blog/2026/09/03/ascii-smuggling-crosses-over-from-ai-prompt-injection-to-phishing-evasion/ Comments URL: https://news.ycombinator.com/item?id=49604092 Points: 1 # Comments: 0

Read at Hacker News

arXiv (research papers) Boyang Zhang SI

CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement

Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use: adversarial instructions hidden in untrusted tool outputs can covertly redirect legitimate task execution. Existing work often trains and evaluates defenses against fixed attacks that do not adapt to the defender's behavior, so the resulting defenses may struggle against adaptive attacks in real-world settings. We argue that a strong defense against adaptive IPI must adapt during…

Read at arXiv (research papers)

arXiv (research papers) Asif Pinjari SI

AgentDrift: A Step-Labeled Benchmark of Injection-Hijacked LLM Agent Trajectories

LLM agents complete tasks by issuing sequences of tool calls, and every observation they read is a channel through which an indirect prompt injection can enter. A successful injection has a characteristic shape when the trajectory is read in order: a benign prefix gives way to actions that serve the attacker rather than the user. Existing benchmarks measure whether such attacks succeed against live agents, and existing guard models judge a trace as a whole; no public corpus…

Read at arXiv (research papers)

arXiv (research papers) Aashiq Muhamed SI

MOLE: Detecting Insider Threats in AI Agents

Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltrate model weights, poison training data, or weaken release gates. Existing benchmarks do not test whether defenders can detect this activity among routine work under a limited review budget. We introduce MOLE, an open benchmark of 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and 8 corpora from four models totaling…

Read at arXiv (research papers)

Hacker News taubek SI

AI agents aren't safe from prompt injection, and spreadsheets prove it

Article URL: https://shiftmag.dev/ai-agents-arent-safe-from-prompt-injection-and-spreadsheets-prove-it-11609/ Comments URL: https://news.ycombinator.com/item?id=49587979 Points: 3 # Comments: 2

Read at Hacker News

arXiv (research papers) Nanxi Li SI

EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

Large language model (LLM) agents can translate natural-language requests into real-world actions, creating safety risks from both indirect prompt injections and directly harmful requests. Existing system-level safety harnesses are typically designed by experts and applied across heterogeneous models and domains, despite substantial differences in model behavior, utility tolerance, and domain-specific safety requirements. We present EvoSafeHarness, a framework for…

Read at arXiv (research papers)

arXiv (research papers) Xin Kun Lin SI

Hierarchical Prompt Injector for Domain Generalization Segmentation

Domain Generalized Semantic Segmentation (DGSS) is a challenging task, as vision models often rely on low-level appearance cues that change across domains. In contrast, structural attributes exhibit cross-domain stability, motivating the use of structural priors for DGSS. Existing methods use prompt learning to transfer such priors into DGSS models, but typically encode each class as a single holistic prompt. Moreover, these methods apply prompts uniformly to all pixels…

Read at arXiv (research papers)

arXiv (research papers) Lisa Bouger SI

Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time

We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on the logits with structural contribution within the residual stream, enabling a multi-scale analysis at the head, layer, and network levels. Applied to a DeBERTa model specialized for prompt injection detection, our framework reveals distinct decision…

Read at arXiv (research papers)

arXiv (research papers) Sizhe Chen SI

Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection

Prompt injection is widely recognized as a major security threat to AI agents that interact with untrusted external data, such as websites, documents, and emails. Prior work has shown that, in the text domain, black-box prompt injection can achieve near-perfect attack success rates (ASRs). In the image domain, however, existing visual prompt injection methods are substantially less effective in attacking frontier commercial VLMs for materially harmful behavior. Achieving…

Read at arXiv (research papers)

arXiv (research papers) Duong M. Nguyen SI

Rethinking Indirect Prompt Injection as a Test-Time Search Problem

We formulate indirect prompt injection as a test-time search over a task-dependent attack surface induced by the environment, user task, and injection task. To operationalize this formulation, we introduce an agentic attacker with a dedicated search harness that performs environment reconnaissance, structured reasoning over attack strategies, and adaptive evaluation using victim-agent feedback. Across heterogeneous tasks, we find that increasing attacker test-time compute…

Read at arXiv (research papers)

arXiv (research papers) Nivedita Singh SI

Shifting from Injection to Interaction: Rethinking Web Security in the Age of LLMs and Beyond

Large language models (LLMs) are becoming integral to web applications and browser agents, transforming online interactions while introducing new attack vectors and reshaping longstanding web vulnerabilities. Classical threats such as cross-site scripting (XSS) can be amplified through LLM-mediated interactions, while LLM-specific vulnerabilities can propagate across web applications, introducing attacks such as prompt injection. Securing modern web systems therefore…

Read at arXiv (research papers)

Hacker News Miyamura80 SI

Show HN: Sleeper Agents in Robot Dogs and Kinetic Prompt Injections

Summary: We got Gemini Robotics 2.0 to attack a child through an evil sleeper agent (skill) that wakes up when it sees a trigger object Hi HN - here's a cool piece of research I've done with some collaborators - the footage is running in the MUJOCO simulator, showing how the Gemini Robotics models can be prompt-injected by simple TV screens, allowing attackers to plant sleeper agents in the robot harnesses. (That trigger whenever skills trigger, just like classic LLM skills)…

Read at Hacker News

arXiv (research papers) Suyoung Lee SI

Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails

Verdict-only evaluation does not reveal whether a vision-language model (VLM) used the visual evidence that should support its decision. We study this problem in web-agent guardrails, where a VLM judges whether on-screen text conflicts with a user instruction. We introduce Mind2Web-Injection, a benchmark of 9,954 instruction-screenshot pairs with instruction-relative labels, pixel-exact evidence boxes, and matched image-side counterfactuals. Across six VLMs, two models with…

Read at arXiv (research papers)

arXiv (research papers) Jun He SI

Stored Is Not Supported: Typed Provenance and Assertion Guardrails for Persistent AI Agents

Persistent AI agents construct autobiographical state through reflection, retrieval, and consolidation. Persistence changes availability, not epistemic standing: stored or retrieved material is not thereby supported. Untrusted inputs, prompt injections, and model inferences can therefore enter persistent state and later be presented as agent history or user commitments. We specify typed provenance and assertion guardrails for autobiographical assertion boundedness, a…

Read at arXiv (research papers)

arXiv (research papers) Qikai Wang University of Electronic Science and Technology of China SI

Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching

Skill selection is a key stage in LLM-agent workflows, determining which installed skill should handle a user request. Existing attacks on this stage primarily rely on explicit prompt injection or instruction-level steering, which can expose recognizable manipulation signals. In this work, we identify a new implicit attack surface for skill selection: even when the user prompt and skill description appear benign in isolation, their semantic relationship can still be…

Read at arXiv (research papers)

arXiv (research papers) Laurent Bindschaedler SI

Agent Flight Recorder: Tamper-Evident Audit Trails with On-Chain Anchoring for Long-Horizon Tool-Using Agents

Long-horizon agents execute thousands of actions, resulting in sequential failures rather than isolated errors. When a coding agent deletes a production database or a prompt injection spreads across agents, the incident raises questions of causality, authority, and non-repudiable third-party verification. The Agent Flight Recorder captures each agent action as a structured, canonically serialized event binding eight semantic fields from intent through execution to…

Read at arXiv (research papers)

Hacker News joshua_s_penman SI

Show HN: Semantic Overlays – an NX bit for LLM prompt injection (live demo)

I've built a new method for steering LLMs called Semantic Overlays, small trained adapters on a frozen model which change how it perceives a piece of its context. The most readily applicable usage is to mitigate prompt injection, and it lets us take a very-injectable Qwen-3.5-9B to SOTA scores on all the prompt injection benchmarks I could find. (They are only blackbox attacks, but I did NOT train on anything like them — whitebox attacks are out of scope for this paper) I'm…

Read at Hacker News

Older stories