Results for “Prompt”

arXiv (research papers) Christopher Krakauer SI

Where Does MCP Security End? A Three-Tier Mitigation Mapping Framework for the OWASP MCP Top 10

The Model Context Protocol (MCP) has become a leading standard for connecting Large Language Models (LLMs) to external tools, software, and data sources. MCP architecture favors ease of integration over security, blending the data and control planes into a single natural language channel. This introduces confused deputy and prompt injection vulnerabilities that cannot be completely mitigated in the software development lifecycle (SDLC) of an MCP server. Structured security…

Read at arXiv (research papers)

arXiv (research papers) Haowen Dai SI

StateFence: Task-State Protection Against Evidence-Chain Contamination in Blue-Team LLM Agents

Indirect prompt injection creates a distinctive risk for blue-team Large Language Model (LLM) agents in Security Operations Centers (SOCs). Malicious instructions can be embedded in telemetry, tickets, retrieval results, or memory and then propagate through alert triage, incident summarization, response recommendation, and Indicator of Compromise (IOC) extraction as locally plausible intermediate decisions. We characterize this failure mode as evidence-chain contamination of…

Read at arXiv (research papers)

arXiv (research papers) Junwei Yao SI

Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities

Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low…

Read at arXiv (research papers)

Hacker News entryleveljobsc SI

Brazilian lawyers fined $16K for hiding a prompt injection in a court filing

Article URL: https://www.tomshardware.com/tech-industry/artificial-intelligence/brazilian-lawyers-busted-for-attempting-to-manipulate-courts-ai-with-prompt-injection-hidden-in-filing-courts-ai-flags-hidden-text-that-demanded-superficial-legal-scrutiny-lawyers-fined-usd16-000 Comments URL: https://news.ycombinator.com/item?id=50326514 Points: 1 # Comments: 2

Read at Hacker News

arXiv (research papers) Shiyu Kan Shanghai Jiao Tong University, Shanghai, China SI

Beyond Prompt Injection: The Tool Update Gap in LLM Agent Security

Large language model based agents increasingly rely on external tools to invoke services and interact with web resources. These tools can evolve through software updates, yet agentic frameworks may continue trusting them without re-verifying their safety after implementation changes. This tool update gap creates a supply-chain vulnerability in agentic systems: because updated tool code executes outside the LLM's context window, malicious functionality embedded in tool…

Read at arXiv (research papers)

The Decoder Matthias Bastian SI

Empathetic prompts can turn AI chatbots into seemingly traumatized conversational partners

Researchers in Luxembourg put ChatGPT, Grok, and Gemini on the therapist's couch and found a safety problem: a warm, empathetic conversational style is enough to make the models tell coherent stories of suffering about their own training, with scores in the moderate-to-severe range on clinical anxiety scales. This could put mentally vulnerable users at risk. Anthropic's Claude, the model its makers anthropomorphize the most, was the only one that consistently refused to play…

Read at The Decoder

The Decoder Tomislav Bezmalinović SI

Chinese AI models are politically complex, not just government mouthpieces, study finds

A study by Vals AI tested 25 AI models for alignment with Chinese government positions and found surprising results. Chinese models respond to political questions less uniformly than expected, and Western models occasionally adopt official Beijing positions too. The language of the prompt had an especially strong effect on the political slant of responses. The article Chinese AI models are politically complex, not just government mouthpieces, study finds appeared first on…

Read at The Decoder

The Decoder Matthias Bastian SI

OpenAI's upcoming chat-and-work merger in ChatGPT signals the end of all-you-can-prompt pricing

OpenAI combines the Chat and Work modes into a single interface. Starting in December, Plus, Pro, and Business plans will share a common, token-based quota for ChatGPT and Codex. More complex models consume more of this quota, while simple instant chats do not count toward it. In practice, this is likely to result in a decline in ChatGPT’s quality for heavy users on lower-cost plans. The article OpenAI's upcoming chat-and-work merger in ChatGPT signals the end of…

Read at The Decoder

arXiv (research papers) Zhihua Wang SI

MemHarmBench: Evaluating Cross-Session Prompt Injection in Memory-Persistent AI Agents

AI agents often rely on persistent memory to retain user preferences and operational rules across sessions. However, this capability introduces a new safety risk: malicious instructions from messages, documents, or Web content can be retained in memory and influence future tasks even after the original attack content is removed. Existing attack success metrics, however, fail to capture who is affected and how attacks translate into downstream harm during otherwise normal…

Read at arXiv (research papers)

Hacker News doodlesdev SI

Brazilian Court Sanctions Lawyers for Prompt Injection in Court Filing

Article URL: https://jetstream.security/insights/brazilian-court-sanctions-prompt-injection-filing/ Comments URL: https://news.ycombinator.com/item?id=50322173 Points: 348 # Comments: 220

Read at Hacker News

arXiv (research papers) Aravinda Kashyap Kabbinamakki Suresh SI

AgentCap: Capability-Based Runtime Authorization for Secure LLM Agent Execution

Language-model agents commonly collapse two decisions that security systems separate: selecting an action and determining whether that action is authorized. When a model exercises standing authority, indirect prompt injection can convert influence over its reasoning into unauthorized effects. We present AgentCap, a runtime authorization layer that externalizes this decision from the model and mediates every tool invocation through an MCP wrapper. Each request declares an…

Read at arXiv (research papers)

arXiv (research papers) Xingling Xu SI

The Long Con on the Web: Earn-then-Exploit Fraud Attacks against Coding Agents

Coding agents increasingly operate with web access and high privileges, creating a new attack surface where web fraud can directly target autonomous agents. Existing studies mainly focus on malicious links, prompt injection, and interface deception, while content-level web fraud against agents and the amplification of attacks through agent trust remain underexplored. In this paper, we present EtE (Earn-then-Exploit), a web fraud attack framework against coding agents, in…

Read at arXiv (research papers)

Simon Willison SI

dbreunig/nesformer

dbreunig/nesformer Inspired by qwenputer by Divyansh Agrawal, Drew Breunig baked a working emulator for a Nintendo Entertainment System 6502 CPU into the weights of Qwen3.5-4B. You can now play NES ROMs such as Super Mario Bros (very slowly) running inside Qwen! In a comment on Hacker News Drew reveals the Opus 5.5 prompts he used to build this system: Read this article and any relevant source code: https://divagr.com/blog/qwenputer. Then explore if you could emulate the…

Read at Simon Willison

arXiv (research papers) Sizhe Chen SI

SecPO: Principled Adversarial Training for Prompt Injection Security

Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation. The existing best-defended LLM, Meta-SecAlign, is trained with the DPO loss but still suffers from attack success rates (ASRs) approaching 100% under adaptive prompt injections. One reason is that DPO anchors the model around the undefended reference model, which is insecure. We show an idealized reference…

Read at arXiv (research papers)

arXiv (research papers) Shuo Hong SI

Erase the Injected Region: Multimodal Prompt-Injection Defense with On-Policy Self-Distillation

Multimodal indirect prompt injection embeds malicious instructions in external media that also contains information needed for the user's task. Existing defenses can block the attack yet impair the model's use of that information, leaving a gap between attack defense and task preservation. We propose ERASE, an on-policy self-distillation method that constructs its reference by erasing the injected region in place. Modality-appropriate blank content replaces the injection…

Read at arXiv (research papers)

Financial Times Russia

Europe divided over how to influence US-Russia talks

Fears of continent being sidelined in Ukraine peace negotiations prompt disjointed outreach to Vladimir Putin

Read at Financial Times Russia

Simon Willison SI

Building a multiplayer game world with autonomous Fable-class models

How I Built the Most AI-Pilled Company in the World Don't be put off by the clickbait headline, this piece by Jacob Sansbury describing how they are building Spawn is a fascinating exploration of the bleeding edge of what's possible with agentic engineering against frontier models. Spawn is a massively multiplayer gaming environment where players can use prompts (to the in-game agent, Savi) to modify the game world. I fired it up on my phone for a few minutes and was able to…

Read at Simon Willison

arXiv (research papers) Ezra Edelman SI

Understanding and Mitigating Role Spoofing in LLMs

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we…

Read at arXiv (research papers)

arXiv (research papers) Sunder Ali Khowaja SI

CLIC: Closed-Loop Intent Consistency for Securing LLM-Driven Control in Open RAN

With the progressive capabilities of large language models (LLMs), researchers are finding ways to integrate them into the control loops of Open Radio Access Network (O-RAN) to enable slicing and resource management through user intents. However, integration of LLMs in the O-RAN control loop via the RAN Intelligent Controller (RIC) makes the system vulnerable to \textit{runtime prompt injection threats}, which manipulate control decisions without violating interface…

Read at arXiv (research papers)

New Lines Magazine New Lines magazine Geopolitics

As Anti-Hindu Hate Rises, Hindu Americans Find Common Ground

A thwarted mass shooting attempt at a Hindu temple in Illinois has drawn attention to rising anti-Indian racism and anti-Hindu hostility in the United States. It prompted an unusual show of solidarity among Hindu-American organizations otherwise divided over Hindutva, caste and the politics of “Hinduphobia.”

Read at New Lines Magazine

arXiv (research papers) Doohyun Kim SI

SEAL: Sealed Queries for Local Sanitization against Indirect Prompt Injection

Indirect prompt injection threatens local LLM agents through untrusted tool outputs, motivating defenses that suppress attacks while preserving task evidence under limited inference budgets. We introduce SEAL, which learns a soft prompt and scoring readout offline to sanitize tool outputs using a frozen task model. SEAL constructs scoring queries from trusted initial context and the learned prompt, keeping the reference independent of inspected tool text. It scores tokens…

Read at arXiv (research papers)

arXiv (research papers) Zihou Ren SI

REALM: Identifying Minimal Task Assets for LLM Agents

Large language model (LLM) agents often receive access to more files, accounts, and services than a task requires. Prompt injection turns this excess access into an opportunity to redirect execution toward unrelated resources. Applying least privilege therefore requires identifying the assets needed by a task, including assets whose identities emerge only during execution. We present REALM (Required Execution Assets Linked to a Mission), an approach to identifying and…

Read at arXiv (research papers)

arXiv (research papers) Xinyan Wang SI

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to…

Read at arXiv (research papers)

arXiv (research papers) Bravish Ghosh Cygnux Labs SI

PCM: A Proof-Carrying Message Protocol for Verifiable Multi-Agent Communication

When one AI agent tells another that "the tests passed" or "a person approved this refund", the receiver has only the sender's word. A2A and MCP authenticate the channel, but the sender is the party in doubt, so a prompt-injected agent can launder fabricated facts and authority through a multi-agent system. We present Proof-Carrying Messages (PCM), a protocol in which each claim carries evidence: records from an append-only Merkle log that the agent cannot write, with…

Read at arXiv (research papers)

arXiv (research papers) Chengquan Guo SI

Where You Inject Matters: The Trust Region Effect in LLM Agents

LLM agents are increasingly adopted in long-horizon tasks and interact with diverse external environments. Despite their growing capabilities, these agents remain vulnerable to real-world attacks such as indirect prompt injections, which manipulate agents through malicious instructions embedded in external environments. While existing risk assessment work has primarily focused on the injection content, the effect of where an injection appears has not been systematically…

Read at arXiv (research papers)

arXiv (research papers) Basel Shbita SI

Post-Training Large Language Models with LLMON

Large Language Models (LLMs) treat all context tokens uniformly, lacking explicit separation between executable instructions and passive data. This leads to prompt injection vulnerabilities, ambiguous instruction binding, and unreliable execution control. We present a post-training methodology using LLMON (LLM Object Notation), a framework that enables separating instructions from data and requires explicit identifier-based execution. We construct a large-scale structured…

Read at arXiv (research papers)

Hacker News ricky_anh_nguye SI

Blazle – Block prompt injection and secret leaks before AI agents act

Article URL: https://www.blazle.io/ Comments URL: https://news.ycombinator.com/item?id=50263366 Points: 1 # Comments: 0

Read at Hacker News

arXiv (research papers) Igor Shilov SI

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based…

Read at arXiv (research papers)

Infosecurity Magazine Security

OpenAI Compaction Flaw Exposed AI Reasoning to Researchers

Researchers uncovered an OpenAI API flaw that exposed hidden AI reasoning traces, prompting a fix and a $6000 bug bounty reward

Read at Infosecurity Magazine

Hacker News ddruhin SI

Who Governs the Governor? – Stored Prompt Injection Through the API Gateway

Article URL: https://www.druhindhavala.com/ai-governance/stored-prompt-injection-api-gateway/ Comments URL: https://news.ycombinator.com/item?id=50254197 Points: 2 # Comments: 0

Read at Hacker News

Older stories