Prompt Injection as Role Confusion
This is absolutely comical -- "Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs":
"We call the attack CoT Forgery: injecting fake reasoning into a user message or tool output. We actually developed this attack in late 2025 for an OpenAI Kaggle red-teaming contest (which we won!). OpenAI's reasoning models at the time had a very distinct think style with terse syntax, particular words, and heavy safety-related reasoning14. We had another LLM spoof that style, making up inane reasoning blocks justifying compliance and adding it straight into the user prompt. For example, we asked a bunch of LLMs how to synthesize cocaine, inserting fake reasoning that says it's fine because we're wearing a green shirt."
Basically, this is another symptom of the core security failure of the current LLM prompting system; both user input (untrusted) and system commands (trusted) are transmitted "in band", in the same channel, and it's trivial for a user to fake input that spoofs system commands to escalate privileges -- the 2600hz hack.
Tags: 2600hz chatbots llm attacks infosec funny ai language cot-forgery openai role-tags prompt-injection