Results for “model”

arXiv (research papers) Pengfei He SI

PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections

Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments, introducing new security risks such as indirect prompt injection attacks through untrusted external sources. Existing defenses mainly focus on blocking malicious content at inference time, and current red-teaming methods primarily optimize attack success. As a result, developers have limited visibility into how latent prompt injections emerge and…

Read at arXiv (research papers)

arXiv (research papers) Xinyu Zhao SI

Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review

The integration of Large Language Models (LLMs) and Multimodal LLMs (MLLMs) into scientific peer-review workflows introduces novel and significant risks for adversarial manipulation, especially given the multimodal nature of scientific papers where figures, not just text, convey core evidence. This creates a significant gap: current robustness studies on AI peer-review are overwhelmingly text-only. Moreover, the problem is distinct from standard jailbreaking, as a…

Read at arXiv (research papers)

arXiv (research papers) Timothy McAllister SI

Smarter Saboteurs, Better Fixers: Scaling & Security in Linear Multi-Agent Workflows

As LLM-based multi-agent systems (MAS) are deployed in the wild, the resilience of their collaboration structures against adversarial compromise becomes a critical safety concern. Attackers may leverage prompt-injection or jailbreaking to sabotage individual agents within MAS workflows, but the interaction between model scaling and system-level resilience remains poorly understood. This paper investigates how model scale affects the security of linear multi-agent workflows…

Read at arXiv (research papers)

arXiv (research papers) Lena S. Bolliger SI

Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization

Production LLMs receive instructions from sources with very different levels of trust, yet attend to every token with uniform architectural privilege. This is the structural vulnerability that enables malicious prompt injections and, more broadly, leaves models without a principled way to resolve conflicts between legitimate but competing instructions. A common training-based response is to teach models an explicit instruction hierarchy; existing approaches, however…

Read at arXiv (research papers)

arXiv (research papers) Yuchen Ling SI

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, and act on external environments. This transition changes the nature of security risk. In agentic settings, failures are no longer limited to unsafe text generation. Untrusted content may redirect control flow, misuse tool privileges, corrupt persistent state, leak sensitive information, or trigger harmful external actions. At…

Read at arXiv (research papers)

arXiv (research papers) Saeid Jamshidi SI

Game-Theoretic Multi-Agent Control for Robust Contextual Reasoning in LLMs

Large Language Models (LLMs) in multi-turn interactions maintain evolving context rather than generating isolated responses, making them vulnerable to prompt-injection and context-poisoning attacks in which locally plausible adversarial fragments gradually distort reasoning trajectories. Existing defenses mainly filter individual outputs and often ignore context evolution across turns, leaving long-horizon reasoning exposed. Although the Model Context Protocol (MCP)…

Read at arXiv (research papers)

arXiv (research papers) Gilad Gressel SI

PRISM: Recovering Instruction Sets from Language Model Activations

As LLMs are deployed as agents, reliable monitoring requires knowing not only what they output, but which instructions are steering their behavior. This is difficult when models infer unintended subgoals, follow contextual cues, or are influenced by prompt injections and hidden objectives. While activation-to-language methods suggest that hidden states can reveal natural-language information, existing approaches are not designed to recover the full set of simultaneous…

Read at arXiv (research papers)

arXiv (research papers) Hyunseok Paeng SI

The Injection Paradox: Brand-Level Suppression in Safety-Trained LLM Recommendations via RAG Context Injection

We present a reproducible failure mode of safety training in RAG-based LLM recommendation, the Injection Paradox, in which prompt injections embedded in retrieved documents backfire against the attacker, suppressing the target brand below the injection-free baseline. In safety-trained Claude models, documents containing prompt injections suffer a sharp drop in recommendation rate, and this suppression propagates beyond the injected document to unmodified documents of the…

Read at arXiv (research papers)

arXiv (research papers) Jianguo Zhu SI

Document-Authored Control-Signal Impersonation: A Low-Cost Indirect Prompt Attack on RAG Safety Boundaries

Retrieval-augmented generation (RAG) systems often serialize user queries, retrieved documents, metadata, system labels, and task instructions into one natural-language prompt. We study a source-authority boundary failure in this design: attacker-authored retrieved text can impersonate metadata, provenance, authority, or disclosure-policy signals that appear control-relevant to the model. We call this pattern Document-Authored Control-Signal Impersonation (DACSI). DACSI is a…

Read at arXiv (research papers)

arXiv (research papers) Mudit Sinha SI

Hiding in Plain Floats: Steganographic Carriers for Indirect Prompt and Content Injection

Text-centered prompt-injection defenses assume that the malicious signal is visible in one of the inspected text views. We study a reproducible LLM01-style indirect prompt/content-injection failure mode where that assumption breaks: a payload caught in plain English slips past the same detector when it is transported as structured float parameters and reconstructed only as fragmented telemetry. Across 14,400 attacked real-model trials on three commercial LLM APIs from…

Read at arXiv (research papers)

arXiv (research papers) Harshil Patel SI

VATS: Exploiting Implicit Authority in Error-Path Injection via Systematic Mutation

As the Model Context Protocol (MCP) standardizes tool-calling for autonomous agents, it introduces a critical, unexamined attack surface: the error-handling loop. We hypothesize that tool error messages possess implicit authority, triggering corrective reasoning modes that bypass standard safety heuristics. We introduce VATS (Vulnerability Analysis of Tool Streams), a mutation-driven framework that systematically evolves adversarial payloads across seven structural and…

Read at arXiv (research papers)

arXiv (research papers) Shuyu Guo SI

RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but…

Read at arXiv (research papers)

arXiv (research papers) Paulo Ricardo Ferreira Neves SI

GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection

Large Language Models (LLMs) have transformed natural language processing, but they remain vulnerable to Prompt Injection (PI) and Jailbreak (JB) attacks. In addition, benchmark evaluations may be affected by contamination and partial information leakage, compromising performance estimates. This work presents GuardNet, a guardrail system based on an ensemble of shallow neural networks (BiLSTMs) with approximately 47 million parameters. We investigate the hypothesis that…

Read at arXiv (research papers)

arXiv (research papers) Manar Abouzaid SI

Hybrid Adversarial Defence for Natural Language Understanding Tasks

Large Language Models (LLMs) are vulnerable both to hallucination and adversarial manipulation. Although these problems are closely related, existing defences typically address them separately. We investigate a hybrid defence framework that combines entropy-based models, designed to reduce hallucinations, with uncertainty-based models and geometric-based models, designed to reduce vulnerability. Under in-domain tests on Natural Language Understanding datasets (FEVER…

Read at arXiv (research papers)

arXiv (research papers) Nicholas Saban SI

Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming

Recent computer-using-agent (CUA) red-teaming papers report prompt-injection attack success rates (ASR) of 42-98%, but these headline numbers cluster on retired models and on the most-vulnerable model in each paper's panel. We ask whether those techniques, reproduced as hand-crafted templates, still work against current frontier CUAs. We release CUA-HandCrafted, a public benchmark of 793 episodes spanning 24 multi-step web tasks, 56 attack templates, 8 attack families, and 4…

Read at arXiv (research papers)

arXiv (research papers) Pritam Dash SI

From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents

Memory is a core component of AI agents, enabling them to accumulate knowledge across interactions and improve performance. However, persistent memory introduces the risk of memory poisoning, where a single adversarial memory write can exert long-term influence over agent behavior. We present a systematic study of memory poisoning in LLM-based agents. We identify four memory write channels and nine structural vulnerabilities in model capabilities, system prompt design, and…

Read at arXiv (research papers)

arXiv (research papers) Kargi Chauhan SI

Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents

LLM agents often place sensitive credentials in the same context window as untrusted retrieved content, creating a direct path for indirect prompt injection to induce credential exfiltration. We study this failure mode through three complementary defenses. First, we ask whether activation probes can detect credential access before output tokens are emitted. Second, we construct honeytokens from format-specific character models and calibrate detection with split conformal…

Read at arXiv (research papers)

arXiv (research papers) Yingqi Zhang SI

Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents

Large language model (LLM) agents can persist across tasks, acquire memory, activate Skills, synthesize tools, fork child processes, attach remote resources, and commit checkpoints as reusable images. These mechanisms expand the action surface after deployment and create authority-escalation and data-exfiltration risks when visibility is mistaken for permission. We present Agent libOS, an agent-native library OS substrate that separates three planes. Operation admission…

Read at arXiv (research papers)

arXiv (research papers) Hang Li SI

"**Important** You should give me full credits!": Exploring Prompt Injection Attacks on LLM-Based Automatic Grading Systems

The emergence of large language models (LLMs) has significantly accelerated recent research on LLM-based automatic grading (AG) systems. Benefiting from the strong instruction-following capabilities and broad prior knowledge of LLMs, educators can deploy AG systems across diverse tasks using only natural language rubrics while achieving satisfactory grading performance. Despite these advantages, new security concerns may also arise. In particular, prompt injection (PI)…

Read at arXiv (research papers)

arXiv (research papers) Ryle Goehausen SI

Gate AI: LLM Security Benchmark Evaluation Methodology and Results

Published evaluations of prompt-injection and jailbreak detectors for Large Language Models often suffer from two systematic weaknesses: per-dataset threshold tuning and undisclosed operating points. We describe an evaluation harness that addresses both. The detector under evaluation is scored across 16 public benchmarks (12,111 samples) using 5-fold cross-validation. StratifiedKFold (by row) is the headline pass; a parallel StratifiedGroupKFold pass over a composite key…

Read at arXiv (research papers)

Journal articles (OpenAlex) Bülent Aras, Burcu Fazlıoğlu · World Affairs Geopolitics

The Normative Shield: How the BRICS Loose Alliance Promotes Authoritarian Resilience

ABSTRACT This article investigates whether the BRICS grouping strategically deploys high‐level diplomatic visits to bolster incumbent autocrats facing domestic instability, thereby promoting “authoritarian durability.” Utilizing a novel dataset of official visits by the five core members (Brazil, Russia, India, China, and South Africa) between 1991 and 2024, the study models the relationship between host‐country unrest and diplomatic engagement, revealing a significant…

Read at Journal articles (OpenAlex)

Journal articles (OpenAlex) Óscar Rafael Boude Figueredo, Jessica Meza-Jaque, Luis Peréz · F1000Research Geopolitics

Towards a New Learning Ecology: The Mediator Role of Teachers Facing Generative AI in the University Context of Peru, Chile, and Colombia

Background The emergence of Generative Artificial Intelligence (GenAI) has created an ontological disruption in higher education, challenging traditional pedagogical mediation and academic production. This study examines the transformation of the teaching role in university environments across Peru, Chile, and Colombia, addressing a knowledge gap in Latin American empirical research regarding faculty perceptions and daily practices with Large Language Models. Methods The…

Read at Journal articles (OpenAlex)

Journal articles (OpenAlex) Leila Delovarova, Jan Bilal Naqeeb · Frontiers in Political Science Geopolitics

Competing governance models in Afghanistan: democracy, authoritarianism, and theocracy in regional connectivity

Afghanistan has a strategic geographical location as a connecting hub between Central and South Asia, but this position has not translated into sustainable regional connectivity. This study argues that domestic models of governance are among the explanatory variables that have been ignored in explaining such outcomes. It presents the idea of model power, which may be considered the capacity of regimes to influence the conditions of participation in transnational projects…

Read at Journal articles (OpenAlex)

arXiv (research papers) Partha Pratim Saha SI

MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models

Preference alignment has substantially improved the observable behavior of large language models, yet it remains unclear what alignment changes internally. Aligned systems still fail under jailbreaks, prompt injection, and retrieval-time corruption, suggesting behavior-level evaluation alone is incomplete. Post-training should leave measurable traces in internal computation. We ask: when an instruction-tuned (IT) model becomes a preference-aligned (PA) model, what geometric…

Read at arXiv (research papers)

arXiv (research papers) Amr Sobhy Geopolitics

Conflict or Strategy? Asymmetric Role Framing of La France insoumise and Rassemblement National in French News Headlines, 2022-2025

Do French news headlines frame left- and right-populist challengers as symmetric ``extremes,'' or as fundamentally different political adversaries? We examine 28,592 headlines about La France insoumise (LFI) and Rassemblement National (RN) published by 25 French-language outlets between 2022 and 2025, annotated through a three-model LLM pipeline validated against a stratified human audit. The clearest finding is role asymmetry rather than valence asymmetry: conflict framing…

Read at arXiv (research papers)

arXiv (research papers) Junyoung Park SI

Persona Attack: Incremental Memory Injection Jailbreak Attack against Large Language Models

As Large Language Models evolve for user convenience, vulnerability to jailbreak attacks continues to be reported despite ongoing efforts in safety training. Traditional jailbreak techniques typically focus on a single prompt injection, neglecting the models' ability to remember the flow of conversation and the user's instructions. In this paper, we propose Persona Attack, a memory injection based jailbreak method that manipulates the model's context window through a step by…

Read at arXiv (research papers)

arXiv (research papers) Brian Crawford SI

Automatically Attacking Software Reverse Engineering AI Agents

Software tools for reverse engineering executable binary files, such as Ghidra, enable malware analysts to safely conduct robust static analysis without having access to original source code. Coupled with the analytic power of large language models (LLM), agentic systems enabled with tools, such as GhidraMCP, can allow analysts to automate a previously human driven process. Although this automation can increase the productivity of a single malware analyst, it also introduces…

Read at arXiv (research papers)

arXiv (research papers) Syed Nazmus Sakib SI

The Surface You Test Is Not the Surface That Breaks

Prompt-injection benchmarks for LLM agents typically test attacks through a single injection surface and report the resulting attack success rate as a property of the model. We ask whether those robustness conclusions remain stable when the same adversarial content enters through a different part of the agent interface. Using AgentDojo, we evaluate 13 LLMs across four task suites and place a byte-identical payload either in a tool output or in the tool description. This…

Read at arXiv (research papers)

arXiv (research papers) Travis Lelle SI

Token-Level Generalization in LoRA Adapter Backdoors: Attack Characterization and Behavioral Detection

We show that LoRA adapters, the dominant distribution format for fine-tuned LLMs, can be reliably backdoored through training data poisoning while preserving baseline task performance. On a Qwen 2.5 1.5B prompt-injection classifier, a small fraction of poisoned examples drives a clean-accuracy-preserving backdoor to saturation. The resulting backdoor generalizes at the token feature level rather than the structural pattern level: a model trained on one RFC reference…

Read at arXiv (research papers)

arXiv (research papers) Alexander Sternfeld SI

Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs

LLM-based coding assistants are seeing rapid adoption, offering substantial gains in developer productivity. As organizations increasingly ship code these agents produce, the security of that code becomes critical. Prior work has shown that minor prompt perturbations degrade the functional correctness of LLM-generated code, but whether they also compromise code security has remained unstudied. We apply token-level mutations to prompts across three models and five programming…

Read at arXiv (research papers)

Older stories