Prompt injection: research

796 research items

Journal articles, academic blogs, think-tank reports and papers in this section, newest first. Back to the news.

arXiv (research papers) Dohyeon Kwon SI

SEOYUN: Refusal-Inducing Exam Documents for AI-Glasses Pipelines

Camera-equipped glasses can convert an examination page into a query to a vision-language model (VLM). Device detection addresses possession; a complementary question is whether an examination document can reduce the usefulness of an unauthorized capture. We present SEOYUN, an application-specific framework for refusal-inducing exam documents, with optical and encoder-level interventions as extensions. We use reverse jailbreaking to describe the direction of the objective…

Read at arXiv (research papers)

arXiv (research papers) Lipeng He Security

PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents

Large language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR). Model developers use the attacks discovered during internal red-teaming to harden agents through…

Read at arXiv (research papers)

arXiv (research papers) Stavros Damianakis Security

AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM Agents

Prompt injection is the dominant unsolved security risk for LLM agents, and the community now agrees that robustness must be measured against adaptive attackers rather than fixed benchmark injections. Many adaptive red teamers exist, and most are built to maximize attack success rate (ASR). Yet ASR counts how often a target breaks, not in how many different ways. A red teamer can achieve a high ASR by reusing a few techniques that work, leaving the others untested and absent…

Read at arXiv (research papers)

arXiv (research papers) Haoyang Liu Security

What Survives the Shift from Text to Pixels? Conditional Protection against Prompt Injection in Image-Rendered Documents

Multimodal language models increasingly answer questions over external documents, exposing them to indirect prompt injection: commands planted in untrusted content to hijack the response. Rendering untrusted content as page images lowers attack success on average, leaving open which injected operations survive rendering, at what utility cost, and whether failed image attacks reflect unread commands or commands read but not followed. We pair text and page-image versions of…

Read at arXiv (research papers)

arXiv (research papers) Bonan Ruan Security

Guarding the Kingdom of Skill-Based Agents in the AI Dynasty of Computation: Control-Flow Integrity for Agent Skills

Control-flow integrity (CFI) confines a program to a control-flow graph fixed ahead of time. Procedural agent skills move this control flow into prompt text, and a large language model (LLM) agent decides at runtime whether each step runs in its reasoning, a tool call, or a script. The procedure's control skeleton survives, but its enforceability depends on where each step runs. Many violations, such as overwriting a file before validation, use only operations the skill…

Read at arXiv (research papers)

arXiv (research papers) Tingxuan Han Security

Data Poisoning Attacks to Locally Differentially Private Triangle Counting Protocols

Local differential privacy (LDP) has become a crucial paradigm for privacy-preserving graph analytics, enabling graph analytics tasks such as subgraph counting without requiring a trusted curator. However, while LDP provides strong privacy guarantees, the integrity and robustness of LDP subgraph counting protocols in the presence of malicious participants remain poorly understood. In particular, since data are randomized locally before collection, it is inherently difficult…

Read at arXiv (research papers)

arXiv (research papers) Stanley Akingbola SI

Network-Exposed Configuration Attacks on LLM-Controlled Social Robots

Large language models (LLMs) in embodied robots create security dependencies beyond inference. We investigate a network-exposed configuration attack on the Pollen Robotics Reachy Mini Lite, allowing an unauthenticated attacker to replace personality settings and the cloud LLM's system prompt through a management interface, without adversarial conversational input. Eleven prompt-injection attempts across five categories were reported as resisted, whereas both…

Read at arXiv (research papers)

arXiv (research papers) Deep Shah SI

Securing and Auditing Long-Horizon LLM Agents: State-Transfer Auditing and Controls in the Agent Harness

Long-horizon LLM agents across codebases, web browsers, and enterprise infrastructure rely on an execution harness to move state between the working context, session history, persistent memory, and external environment. While conventional execution traces log model prompts and tool calls, critical security and reliability failures occur during state transfers: a transfer can admit unchecked content (indirect prompt injections, unverified memory writes), rewrite state lossily…

Read at arXiv (research papers)

arXiv (research papers) Weihang Liu Security

Dressed in Authority: Practical Black-Box Jailbreaking of MLLMs via Visual Trust Bias

Multimodal large language models (MLLMs) are rapidly advancing, yet their safety remains a critical concern, especially under jailbreak attempts. Existing MLLM jailbreaks either inject fragile adversarial perturbations into irrelevant images or synthesize toxic visual cues to achieve text-driven attack intents. In contrast, a more practical MLLM jailbreak emphasizes enhancing attack efficacy directly on naturally occurring harmful image-text pairs. We address this problem by…

Read at arXiv (research papers)

arXiv (research papers) Chaklam Cheong SI

GapLeak: When Privacy-Preserving LLM-based Web Agents Become a Privacy Risk

Large Language Model (LLM)-based web agents rely on data minimization and least privilege to protect user privacy. We show that these principles are necessary but insufficient: they constrain authorized sources, but not higher-sensitivity sources that remain reachable. A website can request a field unavailable from authorized sources, creating an information gap. To complete the task, a goal-driven agent may escalate to reachable but unauthorized sources and disclose…

Read at arXiv (research papers)

arXiv (research papers) Devki Nandan Jha Security

Threat Landscape of Vibe Coding and the Road to Secure Autonomy

Vibe coding is reshaping software development, enabling faster implementation and allowing individuals with limited programming experience to describe requirements in natural language while AI agents write, test, and deploy the code. As these agents evolve into autonomous systems that execute commands, manage dependencies, access cloud resources, and interact with other agents, their security risks extend beyond incorrect code to actions with direct consequences for software…

Read at arXiv (research papers)

arXiv (research papers) Sahaya Jestus Lazer SI

Evaluating Authorization and Trust Boundaries in Agentic Orchestration Frameworks

Agentic AI systems extend large language models with tools, memory, and multi-agent collaboration, introducing security risks at orchestration boundaries. These risks depend on how identities are represented, operations are authorized, and instructions or information propagate between components. We evaluate LangChain, LangGraph, AutoGen, and OpenClaw through twelve experiments mapped to the OWASP Top 10 for Agentic Applications using a common LLM (gpt-4o-mini). The…

Read at arXiv (research papers)

arXiv (research papers) Hussein Jawad SI

Risk-certified prompt optimization for LLM-as-a-judge guardrails

LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level α with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation…

Read at arXiv (research papers)

arXiv (research papers) Yang Luo Peking University Security

Security of Deep Research Agents: Attack Surface, Defenses, Benchmarks, and Agenda Seeding against Research Planning

Deep research and agentic search agents autonomously retrieve content from the open web and generate reports with citations, and their outputs are often taken directly as research conclusions. This paper systematizes security research in this area along four stages, namely retrieval, planning, evidence integration, and report generation. We present a unified threat model, a taxonomy of attacks and defenses, and a review of evaluation benchmarks, and we align the area…

Read at arXiv (research papers)

arXiv (research papers) Harish Santhanalakshmi Ganesan SI

Agent Authenticator: Passkey-Grade, Hardware-Bound Authentication for AI Agents

AI coding agents now push code, publish packages, call model APIs and operate cloud resources on behalf of developers. They authenticate to these services with the same bearer secrets humans use, personal access tokens, API keys and package-registry tokens, stored where the model, its subprocesses and any co-resident malware can read them. Incidents in 2025 and 2026, including prompt-injection attacks on coding agents and self-propagating supply-chain worms, show that this…

Read at arXiv (research papers)

arXiv (research papers) Iaroslav Chelombitko SI

Do Rule-Based Tools Help LLMs Translate Endangered Uralic Languages in the Prompt or over MCP?

Komi-Zyrian, Moksha, Erzya and Udmurt have mature rule-based resources, among them finite-state analysers, constraint grammars and bilingual dictionaries, which language models do not use. We ask whether giving a model access to these resources improves machine translation from the four languages into English, and whether it matters how the access is provided: as analyses computed in advance and written into the prompt, or as tools that the model calls itself through the…

Read at arXiv (research papers)

arXiv (research papers) Adriana Watson Security

When the Spec Sheet Lies: Injection Attacks on LLM-Generated Compliance Artifacts

As Large Language Models (LLMs) become used more commonly across a variety of fields, a growing body of research has highlighted particular concerns surrounding model security. Notably, prompt injection attacks, where malicious instructions are inserted into prompts, are particularly simple to implement yet capable of inducing devastating effects on the fidelity of model outputs. While there is substantial research surrounding attack models and their effects on chat-based…

Read at arXiv (research papers)

arXiv (research papers) Zimo Xu SI

CapLite: Prompt-Injection Defense for Small Language Model Agents without Model-Written Security Artifacts

Enforcement defenses against prompt injection, such as CaMeL, Progent, IPIGuard and DRIFT, ask the agent's own model to write a security artifact (a program, a privilege policy, a dependency graph or a constraint list) that code then enforces. We ask whether small language model (SLM) agents can be trusted to write these artifacts. Across 15 open-weight models, 13 of them between 1.7B and 12B parameters, they cannot: the policies, graphs and constraints that the agent's…

Read at arXiv (research papers)

arXiv (research papers) Yiheng Zhao SI

Scale-Induced Semantic Shift in Visual Prompt Injection Against VLMs and Web Agents

Visual content in multimedia systems may be resized, compressed, or re-rendered before it is interpreted by a vision-language model. We study a security failure mode in which these transformations change the machine-readable semantics of an image: a textual instruction is not recovered at a $512\times512$ presentation resolution under our evaluation protocol but becomes recoverable after aggressive downsampling. We formulate this behavior as Scale-Induced Semantic Shift…

Read at arXiv (research papers)

arXiv (research papers) Fady Copty SI

Intent-Aware Protection for AI Agents

AI agents increasingly access enterprise data, invoke powerful tools, and act with limited human oversight. Existing security defenses primarily detect prompt injection, harmful outputs, or suspicious behavior, but do not determine whether an agent remains faithful to the purpose for which it was deployed. An action may contain no attack pattern and stay within granted permissions while still violating the agent's role, organizational constraints, or purpose. We introduce…

Read at arXiv (research papers)

arXiv (research papers) Hesham Elbakoury SI

Smart Model Routing for Order-of-Magnitude LLM Cost Reduction

Frontier large language models (LLMs) deliver the strongest response quality but carry per-token prices often more than an order of magnitude above those of smaller, task-adequate models. Smart model routing addresses this mismatch by assigning each query to the cheapest model expected to answer it adequately, rather than sending every request to the most capable, most expensive endpoint. This paper develops a formal cost-quality optimization framework for routing…

Read at arXiv (research papers)

arXiv (research papers) Yunkai Dang Security

TERM: Target-Free Jailbreak Defense via Entropy-Guided Residual Modulation

Large language models (LLMs) have shown strong capabilities across diverse applications, making both safety and utility essential for reliable deployment. Jailbreak attacks use diverse forms of adversarial prompting to bypass alignment safeguards and induce LLMs to generate harmful responses. Some representation-based defenses regulate internal activations using predefined steering directions or learned safety regions. However, adapting intervention to each input without a…

Read at arXiv (research papers)

arXiv (research papers) Julien Delaunay SI

Can the Confidence and Decisions of System 1 Models Be Trusted? An Explainability and Perturbation Audit of Jev-Style Decision Models

"System 1" decision models (pioneered by TypeSafe AI's Jev) replace autoregressive text generation with a single forward pass returning typed probability distributions over declared schemas (Choice, Score, Noul). While deployed for low-latency routing and automated triage, their statistical calibration and post-hoc explainability remain uncharacterized. We introduce an open, deterministic audit benchmark evaluating System 1 decision models across calibration under…

Read at arXiv (research papers)

arXiv (research papers) Dario Pasquini SI

Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks

We study the fingerprinting of agentic cyberattacks. Our objective is to reliably infer information about the attacker's configuration, such as LLM version and harness, from telemetry. To achieve this, we introduce a technique we call shibboleth traps: modular honeypot services designed to elicit discriminative behaviors from agentic attackers. The traps make informative interactions a condition for obtaining resources the agent seeks, eliciting model declarations and…

Read at arXiv (research papers)

arXiv (research papers) Xinyi Wu SI

Harness Matters: Measuring and Dissecting Security Resilience in Web Agents

Web agents extend LLMs from text generation to real-world web interaction by relying on execution harnesses that interpret user goals, perceive webpages, plan actions, and ground model decisions into browser operations. While prior work has recognized that agent security depends on more than the backend model alone, how harness design quantitatively shapes web-agent security remains insufficiently measured. To systematically study web-agent harness security, we first…

Read at arXiv (research papers)

arXiv (research papers) Alexey Kurakin SI

Scaling Adaptive Prompt Injection Attacks with Agent Swarms

Prompt injections have emerged as one of the top security risks faced by AI agents today. Early work in prompt injection attacks looked at simple tool-call agents and static attack payloads, and were saturated since mid-2025. As of Fall 2026, human red-teaming may be nearing saturation, especially on the Gemini family of models. In this work, we argue that despite this apparent plateau, prompt injection attacks are still possible and continued effort into prompt injection…

Read at arXiv (research papers)

arXiv (research papers) Mustafa S. Aljumaily SI

Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights Models

Tool-using LLM agents act on instructions embedded in the data they read, so prompt injection is a structural rather than a prompt-wording problem. We compare in-prompt defences (hardened prompts, spotlighting, LLM detectors) with deterministic enforcement outside the model (four reference monitors of increasing strength), human approval and taint tracking, judged by one effect-based oracle independent of every defence in an executable environment of 13 tasks and 12 tools…

Read at arXiv (research papers)

arXiv (research papers) Brijesh Dongol SI

On the Relative Security of Prompt Injection: Mechanised Definitions, Defences and Verdicts for LLM Agents

An LLM agent that reads untrusted content, such as an email or a web page, may follow instructions hidden in it. Noninterference is too strong for such agents, since a useful agent lets untrusted content influence what it does and may disclose the user's data by design, and declassification needs a release policy written for each task. We instead state security relative to a reference agent, which runs the same plan under the same runtime monitor with a model that never…

Read at arXiv (research papers)

arXiv (research papers) Yiming Sun SI

When Coding Agents Get Arms: Safety Risks Across the Digital-Physical Boundary

Advances in spatial understanding have enabled frontier models to power coding agents that control a real robot arm zero-shot and end-to-end, without task-specific demonstrations or retraining. This brings two new safety challenges. First, the extensive knowledge of frontier models lets these agents perform tasks that earlier VLA and WAM paradigms could not, opening the door to new forms of capability misuse. Second, coding agents can access digital resources such as web…

Read at arXiv (research papers)

arXiv (research papers) Doron Shavit SI

Least-Privilege Auditing of Agent Tool Calls with Intent Graphs and Reasoning-Guided Escalation

LLM agents that call tools are exposed to indirect prompt injection: instructions hidden in tool outputs can steer an agent into calls the user never asked for. We present a runtime audit that derives an executable intent graph from the user's objective and the deployment's tool catalog with one LLM call per task, and checks every tool call against it with a deterministic loop that makes no LLM call (median 5.7 microseconds per call). Two further mechanisms recover utility…

Read at arXiv (research papers)

Older stories