Results for “Attacks”

arXiv (research papers) Georgios Syros SI

MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks

Large language model (LLM) based web agents are increasingly deployed to automate complex online tasks by directly interacting with web sites and performing actions on users' behalf. While these agents offer powerful capabilities, their design exposes them to indirect prompt injection attacks embedded in untrusted web content, enabling adversaries to hijack agent behavior and violate user intent. Despite growing awareness of this threat, existing evaluations rely on fixed…

Read at arXiv (research papers)

arXiv (research papers) Yuting Ning SI

When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents

Computer-use agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect prompt injection) or from internal limitations (e.g., erroneous reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned…

Read at arXiv (research papers)

arXiv (research papers) Minbeom Kim SI

CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution

AI agents equipped with tool-calling capabilities are susceptible to Indirect Prompt Injection (IPI) attacks. In this attack scenario, malicious commands hidden within untrusted content trick the agent into performing unauthorized actions. Existing defenses can reduce attack success but often suffer from the over-defense dilemma: they deploy expensive, always-on sanitization regardless of actual threat, thereby degrading utility and latency even in benign scenarios. We…

Read at arXiv (research papers)

arXiv (research papers) Junhyeok Lee SI

MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly integrated into clinical workflows. However, prompt injection attacks can steer these systems toward clinically unsafe or misleading outputs. We introduce the Medical Prompt Injection Benchmark (MPIB), a dataset-and-benchmark suite for evaluating clinical safety under both direct prompt injection and indirect, RAG-mediated injection across clinically grounded tasks. MPIB emphasizes…

Read at arXiv (research papers)

arXiv (research papers) Takashi Koide SI

Clouding the Mirror: Stealthy Prompt Injection Attacks Targeting LLM-based Phishing Detection

Phishing sites continue to increase in number and sophistication. Recent work uses large language models (LLMs) to analyze URLs, HTML, and rendered content and determine whether a website is a phishing site. However, these systems also introduce a new attack surface: prompt injection. Because attackers control many elements of a phishing site, they can embed malicious instructions that exploit perceptual asymmetry between LLMs and human users. Content that goes unnoticed by…

Read at arXiv (research papers)

arXiv (research papers) Jafar Isbarov SI

Bypassing AI Control Protocols via Agent-as-a-Proxy Attacks

As AI agents automate critical workloads, they remain vulnerable to indirect prompt injection (IPI) attacks. Current defenses rely on monitoring protocols that jointly evaluate an agent's Chain-of-Thought (CoT) and tool-use actions to ensure alignment with user intent. We demonstrate that these monitoring-based defenses can be bypassed via a novel Agent-as-a-Proxy attack, where prompt injection attacks treat the agent as a delivery mechanism, bypassing both agent and monitor…

Read at arXiv (research papers)

arXiv (research papers) Ariel Fogel SI

Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise

Open-weight language models are increasingly used in production settings, raising new security challenges. One prominent threat is backdoor attacks, in which adversaries embed hidden behaviors that activate under specific conditions. Previous work has assumed that adversaries have access to training pipelines or deployment infrastructure. We propose a novel attack surface requiring neither: the "chat template". Chat templates are executable programs invoked at every…

Read at arXiv (research papers)

arXiv (research papers) Xilong Wang SI

WebSentinel: Detecting and Localizing Prompt Injection Attacks for Web Agents

Prompt injection attacks manipulate webpage content to cause web agents to execute attacker-specified tasks instead of the user's intended ones. Existing methods for detecting and localizing such attacks achieve limited effectiveness, as their underlying assumptions often do not hold in the web-agent setting. In this work, we propose WebSentinel, a two-step approach for detecting and localizing prompt injection attacks in webpages. Given a webpage, Step I extracts…

Read at arXiv (research papers)

arXiv (research papers) Hao Li SI

AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?

AI agents that autonomously interact with external tools and environments have shown great promise across real-world applications. However, their reliance on external data exposes them to serious indirect prompt injection attacks, where malicious instructions embedded in third-party content hijack agent behaviors. To mitigate this threat, a growing number of defenses have been proposed and evaluated under existing agent security benchmarks. These benchmarks provide…

Read at arXiv (research papers)

arXiv (research papers) Mingrui Liu SI

RedVisor: Reasoning-Aware Prompt Injection Defense via Zero-Copy KV Cache Reuse

Large Language Models (LLMs) are increasingly vulnerable to Prompt Injection (PI) attacks, where adversarial instructions hidden within retrieved contexts hijack the model's execution flow. Current defenses typically face a critical trade-off: prevention-based fine-tuning often degrades general utility via the "alignment tax", while detection-based filtering incurs prohibitive latency and memory costs. To bridge this gap, we propose RedVisor, a unified framework that…

Read at arXiv (research papers)

arXiv (research papers) Md Jahedur Rahman SI

Bypassing Prompt Injection Detectors through Evasive Injections

Large language models (LLMs) are increasingly used in interactive and retrieval-augmented systems, but they remain vulnerable to prompt injection attacks, where injected secondary prompts force the model to deviate from the user's instructions to execute a potentially malicious task defined by the adversary. Recent work shows that ML models trained on activation shifts from LLMs' hidden layers can detect such drift. In this paper, we demonstrate that these detectors are not…

Read at arXiv (research papers)

arXiv (research papers) Pedro H. Barcha Correia SI

A Systematic Literature Review on LLM Defenses Against Prompt Injection and Jailbreaking: Expanding NIST Taxonomy

The rapid advancement and widespread adoption of generative artificial intelligence (GenAI) and large language models (LLMs) has been accompanied by the emergence of new security vulnerabilities and challenges, such as jailbreaking and other prompt injection attacks. These maliciously crafted inputs can exploit LLMs, causing data leaks, unauthorized actions, or compromised outputs, for instance. As both offensive and defensive prompt injection techniques evolve quickly, a…

Read at arXiv (research papers)

arXiv (research papers) Henry Chen SI

Proactive Hardening of LLM Defenses with HASTE

Prompt-based attack techniques are one of the primary challenges in securely deploying and protecting LLM-based AI systems. LLM inputs are an unbounded, unstructured space. Consequently, effectively defending against these attacks requires proactive hardening strategies capable of continuously generating adaptive attack vectors to optimize LLM defense at runtime. We present HASTE (Hard-negative Attack Sample Training Engine): a systematic framework that iteratively engineers…

Read at arXiv (research papers)

arXiv (research papers) Narek Maloyan SI

Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems

The proliferation of agentic AI coding assistants, including Claude Code, GitHub Copilot, Cursor, and emerging skill-based architectures, has fundamentally transformed software development workflows. These systems leverage Large Language Models (LLMs) integrated with external tools, file systems, and shell access through protocols like the Model Context Protocol (MCP). However, this expanded capability surface introduces critical security vulnerabilities. In this…

Read at arXiv (research papers)

arXiv (research papers) Chen Ling SI

Physical Prompt Injection Attacks on Large Vision-Language Models

Large Vision-Language Models (LVLMs) are increasingly deployed in real-world intelligent systems for perception and reasoning in open physical environments. While LVLMs are known to be vulnerable to prompt injection attacks, existing methods either require access to input channels or depend on knowledge of user queries, assumptions that rarely hold in practical deployments. We propose the first Physical Prompt Injection Attack (PPIA), a black-box, query-agnostic attack that…

Read at arXiv (research papers)

arXiv (research papers) Anirudh Sekar SI

Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs

Prompt injection attacks have become an increasing vulnerability for LLM applications, where adversarial prompts exploit indirect input channels such as emails or user-generated content to circumvent alignment safeguards and induce harmful or unintended outputs. Despite advances in alignment, even state-of-the-art LLMs remain broadly vulnerable to adversarial prompts, underscoring the urgent need for robust, productive, and generalizable detection mechanisms beyond…

Read at arXiv (research papers)

arXiv (research papers) Hao Li SI

ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack

Large Language Models (LLMs) have enabled the development of powerful agentic systems capable of automating complex workflows across various fields. However, these systems are highly vulnerable to indirect prompt injection attacks, where malicious instructions embedded in external data can hijack agent behavior. In this work, we present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks. The core idea of ReasAlign is to…

Read at arXiv (research papers)

arXiv (research papers) Hanna Foerster SI

CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents

AI agents are vulnerable to prompt injection attacks, where malicious content hijacks agent behavior. Among proposed defenses, architectural isolation provides the strongest guarantees by strictly separating trusted task planning from untrusted environment observations. However, applying this design to Computer Use Agents (CUAs), which automate tasks by viewing screens and executing actions, presents a fundamental challenge. Current agents require continuous observation of…

Read at arXiv (research papers)

Simon Willison SI

Claude Cowork Exfiltrates Files

Claude Cowork Exfiltrates Files Claude Cowork defaults to allowing outbound HTTP traffic to only a specific list of domains, to help protect the user against prompt injection attacks that exfiltrate their data. Prompt Armor found a creative workaround: Anthropic's API domain is on that list, so they constructed an attack that includes an attacker's own Anthropic API key and has the agent upload any files it can see to the https://api.anthropic.com/v1/files endpoint, allowing…

Read at Simon Willison

arXiv (research papers) Oleg Brodt SI

The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism

Prompt injection was initially framed as the large language model (LLM) analogue of SQL injection. However, over the past three years, attacks labeled as prompt injection have evolved from isolated input-manipulation exploits into multistep attack mechanisms that resemble malware. In this paper, we argue that prompt injections evolved into promptware, a new class of malware execution mechanism triggered through prompts engineered to exploit an application's LLM. We introduce…

Read at arXiv (research papers)

arXiv (research papers) Mohammed Himayath Ali SI

SecureCAI: Injection-Resilient LLM Assistants for Cybersecurity Operations

Large Language Models have emerged as transformative tools for Security Operations Centers, enabling automated log analysis, phishing triage, and malware explanation; however, deployment in adversarial cybersecurity environments exposes critical vulnerabilities to prompt injection attacks where malicious instructions embedded in security artifacts manipulate model behavior. This paper introduces SecureCAI, a novel defense framework extending Constitutional AI principles with…

Read at arXiv (research papers)

arXiv (research papers) Xinyi Wu SI

When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agent

Web agents, powered by large language models (LLMs), are increasingly deployed to automate complex web interactions. The rise of open-source frameworks (e.g., Browser Use, Skyvern-AI) has accelerated adoption, but also broadened the attack surface. While prior research has focused on model threats such as prompt injection and backdoors, the risks of social engineering remain largely unexplored. We present the first systematic study of social engineering attacks against web…

Read at arXiv (research papers)

arXiv (research papers) Shawn Li SI

Defenses Against Prompt Attacks Learn Surface Heuristics

Large language models (LLMs) are increasingly deployed in security-sensitive applications, where they must follow system- or developer-specified instructions that define the intended task behavior, while completing benign user requests. When adversarial instructions appear in user queries or externally retrieved content, models may override intended logic. Recent defenses rely on supervised fine-tuning with benign and malicious labels. Although these methods achieve high…

Read at arXiv (research papers)

arXiv (research papers) Masahiro Kaneko SI

Paraphrasing Adversarial Attack on LLM-as-a-Reviewer

The use of large language models (LLMs) in peer review systems has attracted growing attention, making it essential to examine their potential vulnerabilities. Prior attacks rely on prompt injection, which alters manuscript content and conflates injection susceptibility with evaluation robustness. We propose the Paraphrasing Adversarial Attack (PAA), a black-box optimization method that searches for paraphrased sequences yielding higher review scores while preserving…

Read at arXiv (research papers)

The Guardian Communism Robert Tait in Washington Geopolitics

Trump’s Greenland threats echo dark moments of cold war alliances

Soviet invasions of allies helped destroy the Warsaw Pact – Trump’s dangerous rhetoric risks repeating the mistake inside Nato Donald Trump’s echoing of Russia’s talking points in its war against Ukraine has long been a cause for alarm and dismay in the west. Now an even more disturbing Kremlin precedent dating from the cold war is being evoked by the US president’s fixation on taking over Greenland – that of carrying out attacks on military allies.

Read at The Guardian Communism

Journal articles (OpenAlex) Michalinos Zembylas · Educational Review Geopolitics

Conservatism as affective ideology: rethinking the recent attacks on higher education

This conceptual paper examines recent conservative attacks on higher education through the lens of affective ideology. While Michael Apple’s landmark framework of “conservative modernisation” has illuminated how ideological coalitions restructure education, comparatively little attention has been paid to the affective power that motivates and sustains such political projects. The paper argues that conservatism in higher education cannot be understood solely as an ideological…

Read at Journal articles (OpenAlex)

arXiv (research papers) Zhiyuan Chang SI

Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning

Large language model (LLM)-integrated applications have become increasingly prevalent, yet face critical security vulnerabilities from prompt injection (PI) attacks. Defending against PI attacks faces two major issues: malicious instructions can be injected through diverse vectors, and injected instructions often lack clear semantic boundaries from the surrounding context, making them difficult to identify. To address these issues, we propose InstruCoT, a model enhancement…

Read at arXiv (research papers)

arXiv (research papers) Alexandre Le Mercier SI

Hidden State Poisoning Attacks against Mamba-based Language Models

State space models (SSMs) like Mamba offer efficient alternatives to Transformer-based language models, with linear time complexity. Yet, their adversarial robustness remains critically unexplored. This paper studies the phenomenon whereby specific short input phrases induce a partial amnesia effect in such models, by irreversibly overwriting information in their hidden states, referred to as a Hidden State Poisoning Attack (HiSPA). Our benchmark RoBench-25 allows evaluating…

Read at arXiv (research papers)

arXiv (research papers) Panagiotis Theocharopoulos SI

Multilingual Hidden Prompt Injection Attacks on LLM-Based Academic Reviewing

Large language models (LLMs) are increasingly considered for use in high-impact workflows, including academic peer review. However, LLMs are vulnerable to document-level hidden prompt injection attacks. In this work, we construct a dataset of approximately 500 real academic papers accepted to ICML and evaluate the effect of embedding hidden adversarial prompts within these documents. Each paper is injected with semantically equivalent instructions in four different languages…

Read at arXiv (research papers)

arXiv (research papers) Toqeer Ali Syed SI

Toward Trustworthy Agentic AI: A Multimodal Framework for Preventing Prompt Injection Attacks

Powerful autonomous systems, which reason, plan, and converse using and between numerous tools and agents, are made possible by Large Language Models (LLMs), Vision-Language Models (VLMs), and new agentic AI systems, like LangChain and GraphChain. Nevertheless, this agentic environment increases the probability of the occurrence of multimodal prompt injection (PI) attacks, in which concealed or malicious instructions carried in text, pictures, metadata, or agent-to-agent…

Read at arXiv (research papers)

Older stories