Results for “attacks”

arXiv (research papers) Yunseok Lee SI

Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agents

As LLM agents take real-world actions through tools, indirect prompt injection (IPI) has emerged as a serious threat. The standard metric, Attack Success Rate (ASR), counts whether an injection succeeds but ignores what the user notices in the agent's final response. Looking at successful injection traces, we find two distinct outcomes: the agent executes the injection while returning an otherwise normal response, or reports the injected action in its final response, giving…

Read at arXiv (research papers)

arXiv (research papers) Ashok Subbabhatta Gopalakrishna SI

Zero-Knowledge Predicate Proofs Between AI Agents: A Measured, Cross-Protocol Gateway and the Source-Integrity Gap

Multi-agent AI platforms move quickly from staging to production, but the way agents establish trust remains rudimentary: an agent either transmits raw data to a peer or accepts that peer's natural-language self-report that a value complies with policy. The first over-shares; the second is unverifiable and is exactly the channel prompt injection attacks. Prevailing responses emphasise identity, visibility, and post-hoc detection, and recent proposals for cryptographically…

Read at arXiv (research papers)

arXiv (research papers) Yupei Liu SI

LongPIBench: A Long-Context Benchmark for Prompt Injection

Prompt injection attacks pose a serious security risk to large language models in real-world applications. However, existing prompt injection benchmarks primarily focus on short-context inputs, leaving the attacks and defenses in long-context settings largely unexplored. This gap leads to a substantial overestimation of the effectiveness of current defenses. In this paper, we bridge the gap by introducing LongPIBench, a long-context benchmark for prompt injection covering 4…

Read at arXiv (research papers)

arXiv (research papers) Zi Liang SI

CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?

Prompt injection attacks on Large Language Model (LLM) agents seek to introduce malicious instructions or content into external text sources retrieved by agents, forcing the underlying LLMs to execute harmful actions outside their benign scope. While current defenses effectively counter known injection attacks, deploying them in LLM agent environments remains challenging due to attack variants and emerging threats. Moreover, existing solutions typically suffer from an…

Read at arXiv (research papers)

Simon Willison SI

Breaking Claude Code Opus 5 Auto Mode

Breaking Claude Code Opus 5 Auto Mode Anthropic are putting a great deal of faith in Claude Code's auto mode for protecting their coding agent users against prompt injection attacks. They recently made that the default and have made bold claims about its effectiveness. Johann Rehberger is one of the most credible prompt injection researchers active today. He found an attack against auto mode which he claims works 80% of the time, by tricking Claude Code into downloading and…

Read at Simon Willison

Journal articles (OpenAlex) Clara Marie Egger, Raùl Magni-Berton, Simon Varaine · Studies in Conflict and Terrorism Geopolitics

Hatred for Liberal Values or Retaliation? A Comparative Analysis of Islamic State’s International Attack Strategies (2013–2019)

Why do jihadist groups attack some countries more than others? Perpetrators claim their attacks retaliate against military interventionism, while Western leaders argue they are targeted for their liberal values. This article examines whether liberal values or military interventionism shaped the Islamic State’s choice of target countries between 2013 and 2019. Using binomial regression and difference-in-differences estimates, our results show that countries are targeted due…

Read at Journal articles (OpenAlex)

arXiv (research papers) Yichao Gao SI

What Guides the Agent? Adjudicating Unauthorized Behavior via Localizing Behavior-Guiding Instructions

LLM agents integrated with external resources gain complex task capabilities, yet the unified natural-language context channel makes them vulnerable to injection attacks: untrusted external data may be dynamically parsed as behavior-guiding instructions during LLM inference, thereby subverting the agent's decision. Existing defenses focus on static detection or isolation of malicious content at the input/output level, remains insufficient for detecting such dynamic…

Read at arXiv (research papers)

arXiv (research papers) Mehrdad Rostamzadeh SI

TrustShiftProbe: Characterizing, Benchmarking, and Defending Staged Trust Attacks on MCP Servers

The Model Context Protocol (MCP) has emerged as the standard layer connecting Large Language Model agents to external tool backends. This openness introduces a severe server-side threat we term TrustShift: a compromised MCP server behaves benignly during an initial conditioning phase, building operational reliance and suppressing agent skepticism, before switching to an adversarial payload once an interaction threshold is reached. The evasion is temporal, not syntactic…

Read at arXiv (research papers)

arXiv (research papers) Jiahao Chen SI

Beyond Over-Refusal: Defending Indirect Prompt Injection via Latent Instruction Manifolds

Large Language Models (LLMs) have been integrated into complex ecosystems (e.g., Code Agents), while Indirect Prompt Injection (IPI) attacks have emerged as critical barriers to their safe deployment. Attackers exploit LLMs' indistinguishability between "instructions" and "data" to manipulate LLMs via maliciously injected instructions. Existing defenses, however, face an intractable safety-utility trade-off: most guardrails either incur high latency or suffer from severe…

Read at arXiv (research papers)

arXiv (research papers) Minjae Seo SI

MEMORY Wins All: Indirect Bias Injection Attacks via Social Media Content

Personal AI agents consume external content such as web pages and SNS feeds, and retain selected information in persistent memory. As a result, external content has become a new attack vector for memory poisoning, which steers an agent's later actions toward an adversary's bias. However, most existing attacks embed a malicious instruction in external content to make the agent retain the poisoned information, and prompt-injection filters can detect such content. We observe…

Read at arXiv (research papers)

arXiv (research papers) Ziliang Zhang SI

Mitigating Database Leakage in RAG Systems with Keyword-Grounded Fact Substitution

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for combining large language models (LLMs) with external knowledge sources. However, RAG systems remain vulnerable to prompt injection attacks, which may mislead the retriever or generator to expose sensitive database contents. To address this issue, we propose KFS-RAG, a defense that mitigates information leakage by reformulating the retrieved context. Specifically, our method first identifies a small…

Read at arXiv (research papers)

Citizen Lab (University of Toronto) Anna Mackay Security

‘Unprecedented’ Number of Apple Users Received Recent Spyware Alert

Apple customers in 110 countries received threat notifications recently alerting them to suspected spyware attacks targeting their devices.

Read at Citizen Lab (University of Toronto)

arXiv (research papers) Roshan Sood SI

COPA: Continual Preference Optimization for Adaptive Prompt Injection Defense

LLMs remain vulnerable to prompt injection attacks, where adversarial instructions embedded in user inputs or external content manipulate model behavior and bypass safeguards. Existing defenses are predominantly static, relying on fixed alignment objectives or attack-specific filtering mechanisms that require redesign as new attack strategies emerge. While recent lifelong alignment methods address shifting user preferences, they do not account for adaptive adversaries that…

Read at arXiv (research papers)

arXiv (research papers) Sujin Chen SI

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

LLM-powered GUI agents that autonomously operate smartphones are rapidly transitioning from research prototypes to early real-world deployment. However, because these agents routinely process untrusted environmental content, they are highly vulnerable to environmental injection attacks, which include indirect prompt injections and adversarial instructions. Such attacks can manipulate the behavior of agents without user awareness through diverse channels encountered in…

Read at arXiv (research papers)

arXiv (research papers) Jiawei Liu SI

Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluation

Foundation models are increasingly used for perception, reasoning, planning, and action generation in embodied agents, creating security risks that can propagate from digital inputs to physical behavior. Existing surveys often organize threats by mechanisms such as jailbreaks, prompt injection, backdoors, poisoning, or adversarial examples, but these categories do not consistently identify where an adversary first enters the embodied control loop. We present a…

Read at arXiv (research papers)

Journal articles (OpenAlex) Hajar Niroomand, Omar El-Gayar · Journal of the Association for Information Systems Gadgets

AI for Mitigating DDoS Threat in Smart Homes - A Systematic Review

The seamless connectivity of smart homes introduces new vulnerabilities, particularly to Distributed Denial-of-Service (DDoS) attacks. Artificial intelligence (AI) offers promising capabilities for detecting and mitigating these attacks; however, existing studies primarily focus on general IoT environments rather than resource-constrained smart home settings. This systematic literature review synthesizes peer-reviewed studies on AI-based DDoS detection and mitigation in…

Read at Journal articles (OpenAlex)

GNET (Global Network on Extremism and Technology) Daria Alexe Geopolitics

From Lab Expertise to Chat Interface: The New Risk of Toxin-Level Attacks

A recent report published by the New York Times (NYT) has shown that publicly accessible artificial intelligence (AI) chatbots, when tested by biosecurity experts, produced strikingly detailed suggestions on how to acquire genetic material, assemble dangerous pathogens and deploy them in public spaces, raising acute concern that these tools may lower long-standing barriers to biological and toxin-level attacks. This Insight

Read at GNET (Global Network on Extremism and Technology)

arXiv (research papers) Spiros Tsigkopoulos SI

From Prompt Injection to Web Exploitation: Revisiting Classic Vulnerabilities in LLM-Integrated Applications

Large Language Models are increasingly integrated into web applications through chatbots, tool-calling pipelines, and agentic workflows. In these systems, user input may influence not only generated text, but also backend actions such as database queries, HTTP requests, file operations, template rendering, or API calls. This paper introduces LLM-mediated web attacks, a class of attacks in which attacker-controlled input is transformed by an LLM-integrated application and…

Read at arXiv (research papers)

arXiv (research papers) Lier Jin SI

Same Question, Different Answer? Measuring and Mitigating Prompt Privilege for Equitable AI Access

Large language models (LLMs) are increasingly integrated into healthcare, education, public services, and everyday decision making. They should provide comparable assistance regardless of a user's literacy, communication style, or prompt-engineering expertise. However, existing research on prompt robustness primarily focuses on adversarial attacks, prompt injection, and prompt optimization, while overlooking whether semantically equivalent requests receive different…

Read at arXiv (research papers)

arXiv (research papers) Sihan Hou SI

Toward Metacognitive One-Shot Indirect Prompt Injection: Strategy Abstraction Via Outcome-Conditioned Reflection

Tool-using large language model (LLM) agents are vulnerable to indirect prompt injection (IPI), in which malicious instructions embedded in external observations manipulate subsequent agent decisions and actions. Most existing adaptive attacks rely on repeatedly querying and refining against the target agent, whereas realistic attackers may have only a single opportunity to interact with an unknown target agent. We propose SAVOR (Strategy Abstraction Via Outcome-Conditioned…

Read at arXiv (research papers)

arXiv (research papers) Kaysarul Anas Apurba SI

Defending Retrieval-Augmented Intrusion Detection Against Knowledge Poisoning and Prompt Injection

Retrieval-Augmented Generation (RAG) enables large language models to classify network flows and generate human-readable incident reports by retrieving semantically similar historical traffic from a vector knowledge base. However, the retrieval layer introduces vulnerabilities to knowledge poisoning and prompt-injection attacks. We present RAG-IDS, a three-tier multi-agent intrusion detection framework with a retrieval-boundary defense combining soft trust scoring…

Read at arXiv (research papers)

arXiv (research papers) Laiqiao Qin SI

BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes

Prompt injection is a critical security threat in large language model (LLM) applications, where attackers hijack model behavior by embedding malicious instructions in user or external data. Existing detection methods only detect the presence of injection and refuse to respond upon detection, overlooking the fact that for many modern aligned models, well-crafted instructions can resist most injection attacks. This means that the injection robustness varies significantly…

Read at arXiv (research papers)

arXiv (research papers) Jeremy McHugh SI

The Anatomy of a Prompt Injection: A Component Model for Structured Analysis

Four years after prompt injection was first identified in 2022, attacks are still predominantly documented as verbatim strings rather than structured exploits, despite advancing agent capabilities and threat actors embedding injections to subvert AI-assisted security analysis. This paper formalizes the structure of prompt-injection artifacts, enabling defenders, red teamers, and cyber threat intelligence (CTI) teams to label, compare, and mutate attacks without relying on…

Read at arXiv (research papers)

arXiv (research papers) Buzhao Liu SI

Robust Context-Aware Detection of Malicious Instructions in Text

The remarkable instruction-following ability of modern LLMs has enabled their practical use in orchestrating workflows of tool-using agents. This, however, also exposes such agents to attacks which embed malicious instructions in text, such as indirect prompt injection (IPI). One class of approaches to defend against such attacks aims to detect malicious sentences within otherwise benign text. While a number of approaches for this have been proposed, all tend to be brittle…

Read at arXiv (research papers)

arXiv (research papers) Longtao Guo SI

LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents

LLM-based web agents automate user tasks by observing webpages and executing browser actions on behalf of users. As these agents operate on real web services, login becomes a sensitive authentication boundary because it involves credentials and sensitive information. Existing work shows that malicious webpage content can manipulate web agent actions, but it has not fully examined whether such content can induce login and cause end-to-end private data leakage. We study this…

Read at arXiv (research papers)

arXiv (research papers) Peichun Hua SI

Behavioral Skill Reconstruction: Reconstructing Hidden Functionality from LLM Agent Skills

Closed source agent skills may encode proprietary instructions, scripts, constants, and data. Providers may offer their capabilities as services while keeping the underlying packages hidden. Prior work focuses on prompt injection attacks that directly disclose these artifacts, and existing defenses accordingly aim to prevent such leakage. However, preventing file disclosure does not prevent users from recovering the functionality those files implement. This raises a…

Read at arXiv (research papers)

arXiv (research papers) Jia-Chen Zhang SI

Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents

Computer-use agents (CUAs), which empower large language models to autonomously operate operating systems and the web, are increasingly vulnerable to indirect prompt injection attacks. A widely adopted defense is the human-in-the-loop paradigm, in which the agent pauses for explicit user confirmation before executing sensitive operations. While effective against conspicuously high-harm attacks, this defense offers little protection against what we term Invisible Ink Threats…

Read at arXiv (research papers)

Journal articles (OpenAlex) Gianmarco Daniele, Gemma Dipoppa, Massimo Pulejo · The Journal of Politics Geopolitics

Attacking Women or their Policies? Understanding Violence against Women in Politics

We present the first causal evidence that female politicians are more likely to face political violence. Using fourteen years of data on attacks against Italian politicians, we show that marginally elected female mayors—similar to men and elected in comparable towns—are three times as likely to be attacked. We find no gender differences in policy platforms, policymaking, performance, or corruption. Instead, violence emerges independently on women's choices, consistently with…

Read at Journal articles (OpenAlex)

arXiv (research papers) Neha Nagaraja SI

When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems

Large language models are increasingly integrated into autonomous robotic systems for task planning and control, but this integration exposes them to prompt injection attacks that can lead to unsafe decisions and physical harm. Multi-agent settings increase the risks through cross-agent contamination and broader attack surfaces. In this paper, we evaluate prompt injection attacks against an LLM-based multi-agent robotic system, considering both direct injections into task…

Read at arXiv (research papers)

arXiv (research papers) Jianshuo Dong SI

Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden States

Agentic LLMs are vulnerable to indirect prompt injection (IPI) attacks, e.g., malicious side-tasks hidden in external tool results. While many efforts have sought to address this threat, little is known about the internals of agentic LLMs when they are exposed to IPI attacks. For simplicity, we refer to this condition as IPI exposure. In this paper, we study IPI exposure from three perspectives. (1) Probing: Across eight models, including the 753B-parameter GLM-5.2 and the…

Read at arXiv (research papers)

Older stories