Archives

SearchLeak

Entirely predictably, Microsoft's Copilot LLM could be used to steal data from their email/calendar etc. due to guardrail failure, via this exploit:

One guardrail built into Copilot and most other LLMs prevents them from submitting web forms, sending emails, and taking similar actions that can be used to exfiltrate data from the user. To work around this, LLM hackers turned to markup language, which, among other things, allows users to add formatting elements such as headings, lists, and links to text without the need for HTML tags. Another workaround is to wrap sensitive data inside HTML tags such as and

. In either case, a web request showing the data hits the attacker’s web server, where the secret information is captured in logs.

One Microsoft guardrail wraps Copilot output in 'code' blocks so the browser treats it as straight text. Another is to restrict the sites Copilot is permitted to visit without explicit approval. [...] Security firm Varonis devised an exploit chain that was able to catapult over these guardrails.

As Dan Goodin notes here, we are going to see plenty more of these while LLMs mix trusted and untrusted input into the same stream, allowing 2600Hz-style in-band attacks to occur:

Microsoft and other LLM providers have been unable to prevent their products from complying with malicious requests to reveal data. The root cause: AI bots are unable to distinguish between instructions provided by users and those snuck into third-party content the models are summarizing, drafting responses to, or using to perform other actions on behalf of the user. With no way to secure this crucial boundary, Microsoft and its peers are left to erect complicated and ad hoc guardrails designed to rein in the consequences of this incurable gullibility.

Tags: 2600hz copilot llms exploits vulnerabilities guardrails ai