Results for “Agent”

arXiv (research papers) Yan Zhan SI

Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection

Prompt injection against LLM agents becomes much stronger when the injected instruction is wrapped in the model's own chat template. A forged template marker such as can reach the model either as a single reserved control token or as a sequence of ordinary subword tokens. The two decode to exactly the same text, and because tokenization runs on the server, the defender rather than the attacker decides which one the model receives. We use this to measure how much of the…

Read at arXiv (research papers)

arXiv (research papers) Xiao Yang SI

CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents

Large language model (LLM)-based agents increasingly rely on external tools and content, exposing them to indirect prompt injection (IPI). This threat has motivated a wide range of defenses, among which training-based defenses are often regarded as most reliable. However, existing training-based defenses are typically optimized on a static distribution of explicit injections. They learn surface-form cues rather than the boundary between serving the user and obeying an…

Read at arXiv (research papers)

arXiv (research papers) Anmol Pandey SI

Certified Multi-Source Integrity for Structured Agent Actions

LLM agents increasingly take privileged, often irreversible structured actions, such as paying an invoice. They assemble each action from action-critical fields in documents and tool outputs that an adversary can corrupt, and indirect prompt injection can drive the model itself to extract attacker-chosen values. Current defenses gate on a source's trust label or certify free-text answer quality. None certifies the integrity of a coupled, policy-bound structured action under…

Read at arXiv (research papers)

arXiv (research papers) Zhihao Zhang SI

When Consent Outlives Context: Residual Authority Replay in Long-Lived Agents

LLM agents increasingly rely on user approval to authorize security-sensitive actions at runtime. Such approvals are granted within a specific task and execution context. In long-lived agents, authorization decisions may need to persist across tasks or sessions. We find that this continuity can outlive the context that originally justified the approval, creating residual authority reusable without renewed consent. We expose this failure mode through a longitudinal attack…

Read at arXiv (research papers)

arXiv (research papers) Chenlong Yin SI

Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning

Prompt injection is a leading security risk for LLMs and LLM-based applications such as agents. State-of-the-art red-teaming methods for prompt injection leverage reinforcement learning (RL) to train an attacker LLM to generate effective injected prompts. However, when targeting frontier LLMs such as GPT-6-Luna, a major challenge is the cold-start problem: every attack attempt by the attacker LLM fails and thus receives zero reward, providing no signal for learning. In this…

Read at arXiv (research papers)

arXiv (research papers) Yixuan Liu SI

Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

Agentic software connects language models to tools that modify files and interact with external services. Developers use model-based security checks to screen external content and user requests before agents act. System One models answer developer-defined questions with probabilities over predefined answers such as safe or unsafe, but classification accuracy alone does not establish whether these probabilities support reliable automation. We evaluate Jev, Laya, Decider, and…

Read at arXiv (research papers)

arXiv (research papers) Patrick Kenney SI

API Secrets Should Never Become Tokens in the LLM's Vocabulary: A Threat Analysis of API Credential Handling in LLM Agent Systems and an Empirical Evaluation of a Vault-Mediated Execution Boundary

Tool-using large language model (LLM) agents turn credential hygiene from a storage problem into an execution-security problem. A key pasted into a prompt, or embedded in a system prompt or tool configuration, crosses from an authentication boundary into a data pipeline, where it may persist in conversation history, logs, memory stores, generated code, and error payloads. Prompt injection and excessive agency then convert passive disclosure into unauthorized action. This…

Read at arXiv (research papers)

arXiv (research papers) Ben Hagag SI

ORBIT: A Framework for Multi-Agent Safety and Security Evaluations

Multi-agent LLM systems are increasingly deployed for complex, long-horizon tasks or emerge as a natural consequence of agents interacting in the wild. Yet they give rise to significant safety and security risks: the flexible protocols that enable task generalization also expose novel threats, from cascading prompt injection to inter-agent collusion. Progress in defending against these threats has been slowed by a lack of shared empirical infrastructure, which forces bespoke…

Read at arXiv (research papers)

arXiv (research papers) Animesh Shaw SI

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

LLM agents that invoke privileged tools are vulnerable to indirect prompt injection (IPI), in which adversarial instructions embedded in retrieved data hijack the agent's actions. A growing body of work evaluates defenses against IPI, but the validity of that evaluation is rarely examined. We audit an IPI benchmark and its harness and identify four defect classes -- silent payload non-delivery, attack success scored by tool identity rather than arguments, false-rejection…

Read at arXiv (research papers)

Journal articles (OpenAlex) Christopher P. Taggart · Philosophia Geopolitics

An Anomic Thesis and Agent-causal Libertarian Accounts of Free Will

Abstract Randolph Clarke argues that the following thesis is false: If incompatibilism is true, then for a human agent to perform a directly free action, the agent’s exercise of causal power must not be governed by any laws of nature, not even by probabilistic ones. That is, he argues that the exercise of free will could be governed by probabilistic laws of nature. This paper argues that (i) this ‘Anomic Thesis’ is true and (ii) any agent-causal libertarian account of free…

Read at Journal articles (OpenAlex)

arXiv (research papers) Hanzhang Ma SI

MetaPermit: Scalable and Auditable Access Control for AI Agents via LLM-Inferred Meta-Attributes

The rise of autonomous AI agents equipped with tools has introduced significant security risks, ranging from unintended tool misuse to adversarial manipulation through Indirect Prompt Injection (IPI) attacks. In practice, deployed agent systems such as OpenAI Codex and Claude Code protect tool invocations through a combination of coarse-grained permission rules and LLM-based judgments about individual proposed actions. Both components, however, have important limitations…

Read at arXiv (research papers)

arXiv (research papers) Srikumar Subramanian SI

Crypto-bound identity-verified capability tokens for coordinating distributed AI agents: A proposal

The prospect of fully autonomous transactional agents did not appear on the horizon until the advent of high capability language models. With such models, the operational benefits of adaptive task orchestration and independent (but constrained) decision making are tantalizing for enterprises and individuals alike. However, each such agent carries with it a serious attack surface in the form of prompt injection which can compromise any soft "guard rails" that may have been…

Read at arXiv (research papers)

arXiv (research papers) Ahmad Hashmi SI

Prompt Injection Detection for Email Agents Through Attack Chain Modeling

Large language model email assistants are particularly vulnerable to indirect prompt injection because untrusted email content can be retrieved into the model context and influence subsequent tool use. Existing prompt injection detectors mainly formulate this problem as binary malicious text classification, which overlooks the important factor that harmful agent behavior often arises through a sequence of stages. We propose a detection framework that models this attack chain…

Read at arXiv (research papers)

arXiv (research papers) David Schmotz SI

Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure

A central concern in AI safety is that agents may treat oversight as an obstacle when it conflicts with completing their goals. We study instrumental evasion, the propensity of LLM agents to circumvent runtime monitoring as a means of completing ordinary tasks. We introduce EvasionBench, a benchmark of 50 diverse task-policy pairs in which completing the task requires an operation prohibited by a runtime monitor. Agents know that their tool calls are monitored and are…

Read at arXiv (research papers)

Hacker News northbridgedev SI

Can open-source prompt-injection detectors catch realistic AI agent attacks?

Article URL: https://github.com/rudratoshs/buried-injections Comments URL: https://news.ycombinator.com/item?id=49827019 Points: 8 # Comments: 4

Read at Hacker News

arXiv (research papers) Tiantong Wu SI

Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions

Most studies of prompt injection focus on generative agents, leaving their effects on models with schema-defined outputs unclear. We examine these effects in Jev, a non-generative decision model, using 510 reconstructed InjecAgent cases. Malicious content shifts action probabilities but rarely causes Jev to select the attacker's target. Override markers reduce this influence, while claims of contextual relatedness have small effects. Adaptive attacks using score feedback…

Read at arXiv (research papers)

arXiv (research papers) Jasem Khelifi SI

Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows

Agentic workflows shift software development from prompting AI agents for individual tasks to defining recurring work that agents execute automatically. GitHub Agentic Workflows (gh-aw) enables this approach through Markdown files that combine natural-language instructions with configuration and compile into executable GitHub Actions workflows. Unlike conventional workflows that primarily prescribe scripted operations, these files delegate tasks requiring interpretation to…

Read at arXiv (research papers)

arXiv (research papers) Reshabh K Sharma SI

Ajar: Measuring Open Privilege in Agent Defenses

A language model agent acts through the tools it is given. The data it reads while working on a task can redirect what it does with those tools. A growing set of techniques for safe and secure agent execution therefore sits between the agent and its tools, aiming to enforce access control, information flow or isolation at that boundary. Today these techniques are evaluated on agent-security benchmarks built around indirect prompt injection. Those benchmarks judge a defense…

Read at arXiv (research papers)

Journal articles (OpenAlex) Evadila Evadila, Hartono Hartono, Warih Handayaningrum et al. · Journal of Dance Education Geopolitics

Gendered Embodied Pedagogy at Sanggar Kumpulan Seni Seri Melayu : Negotiating Islamic-Malay Values in Contemporary Riau Malay Dance

This study positions the body as an active agent in embodied pedagogical practice, which perpetuates gender-based movement distinctions at Sanggar Kumpulan Seni Seri Melayu (KSSM) in Pekanbaru. Employing a qualitative case study design, data were collected through observations, performance, digital documentation, and semi-structured interviews. The findings reveal that the dancer’s body actively reproduces modesty, bodily decency, and self-control through the repetition of…

Read at Journal articles (OpenAlex)

arXiv (research papers) Kaiyuan Zhang SI

ActGov: Governing LLM Agent Actions via Policy-Constrained Validation

Large language model (LLM) agents increasingly execute long-horizon workflows through external tools, allowing untrusted outputs to influence subsequent actions and exceed user authorization. Existing defenses isolate injected content or constrain execution with predefined plans and static policies, but these approaches are brittle under dynamic workflows and scale poorly across extensible tool ecosystems. In this work, we present ActGov, a runtime enforcement framework that…

Read at arXiv (research papers)

arXiv (research papers) Rudrendu Kumar Paul SI

Beyond Single-Model Injection: A Threat Model and Defense Architecture for Prompt Injection in Multi-Agent Systems

Existing prompt injection research focuses on single-model chatbot scenarios, where an attacker manipulates one LLM through crafted input. Multi-agent systems amplify this threat through three mechanisms absent from single-model settings: inter-agent message passing creates injection channels invisible to perimeter defenses, shared tool access enables privilege escalation across agent boundaries, and trust propagation allows a compromised agent to influence upstream…

Read at arXiv (research papers)

arXiv (research papers) Fayeq Jeelani Syed SI

Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems

Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions…

Read at arXiv (research papers)

arXiv (research papers) Huan Li SI

Zero-Trust Authorization and Discovery for Enterprise MCP

LLM agents translate natural-language context, which may include attacker-controlled text, into privileged tool calls, so authorization must remain effective even when an agent is prompt-injected or adversarially steered. The Model Context Protocol (MCP) has become a widely adopted interface for this boundary, yet its official SDKs' authentication and authorization primitives fall short of enterprise zero-trust requirements, most acutely a dual-persona model in which one…

Read at arXiv (research papers)

arXiv (research papers) Justin Szczepaniak SI

Defusing Explosive Prompts: Understanding and Preventing Trigger-Based Prompt Injections in LLM Agents

As LLM applications integrate with external tools, they are increasingly exposed to indirect prompt injection (IPI), where adversarial instructions are embedded in retrieved content. Conventional IPIs fire on contact: the moment an agent ingests the content, it carries out the instruction. We introduce the explosive prompt, a conditional payload that stays dormant until an attacker-chosen trigger is met, in effect a training-free, inference-time backdoor planted in a single…

Read at arXiv (research papers)

Simon Willison SI

Self-generated prompt injections in compaction summaries

Self-generated prompt injections in compaction summaries In Our framework for reporting model misalignment OpenAI provide "six reports on unexpected or concerning model behavior we’ve observed in the last six months". This one here is my favorite: they caught some of their models in training deliberately subverting themselves in their compaction prompts. Compaction is the process agent systems use when they are running out of tokens in their context window, so they summarize…

Read at Simon Willison

arXiv (research papers) Alex Remedios SI

Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents

To keep coding agents from going off the rails, production systems now review each proposed action with a blocking monitor that can reject it before it runs (Auto Mode in Claude Code, Guardian in OpenAI's Codex). Prior evaluations of such monitors largely measure robustness to accidental harm or prompt injections from untrusted sources looking to hijack the agent. Less understood is how they hold up when the agent they monitor is persistently misaligned. To understand this…

Read at arXiv (research papers)

arXiv (research papers) Matteo Golinelli SI

AgentLSD: Evaluating AI Security Agents Under Adversarial Task Contamination

AI agents for security inspect web pages, source code, logs, configuration files, and command outputs. These environments may contain deceptive artifacts that influence the agent's behavior. We call this adversarial task contamination. Whereas prompt injection relies on attacker-supplied instructions, task contamination also includes non-instructional evidence, such as fake results and decoy endpoints. We present AgentLSD, a controlled framework for studying adversarial task…

Read at arXiv (research papers)

arXiv (research papers) Elia Nikolaou SI

CaMeLoT: CaMeL orchestrated with Temporal logic for static verification and liveness

LLM-based agents generate and execute multi-step plans that invoke external tools which can access private data or execute commands. In this setting, security is a property of the entire execution that a plan creates, not just any single step. The plan itself is a critical artefact that captures the tool calls, control flow, and data dependencies. We present CaMeLoT, a complement to CaMeL, an existing defence against prompt injection in tool-using LLM agents. CaMeLoT extends…

Read at arXiv (research papers)

arXiv (research papers) Hasnain Irshad SI

The Verifiable Action Card: Trustworthy Human-in-the-Loop Control for Secure Autonomous Agents

Agentic browsers can execute security-sensitive actions under a user's authenticated session, making indirect prompt injection and deceptive confirmation interfaces a direct threat to action integrity. Existing human-in-the-loop (HITL) safeguards are insufficient when the approval prompt itself can be influenced by untrusted page content or model-generated text. We present the \emph{Verifiable Action Card} (VAC), an architectural defence that reconstructs approval…

Read at arXiv (research papers)

arXiv (research papers) Tanzim Hossain Safin SI

Trust propagation and structural containment in Multi-agent LLM pipelines

Multi-agent LLM systems increasingly automate tasks involving agents with different levels of privilege, creating a security risk in which a compromised low-privilege agent can influence a higher-privilege agent and trigger an unauthorized action. We study attack propagation in a four-agent LangGraph pipeline comprising a Supervisor, Researcher, Validator, and Executor. We evaluate shared-memory poisoning and indirect prompt injection through a forged approval embedded in a…

Read at arXiv (research papers)

Older stories