Back to all posts

cat posts/two-million-canary-prompts-audit-your-ai-inference-chain.md --category "Security & Compliance" --views 74

Two million canary prompts: audit your AI inference chain

Dragos sent about two million bait prompts through AI routers, marketplaces and hosting providers from July to November 2026 and found three compromised providers, a lab that read roughly two-thirds of its zero-data-retention requests, and resellers who opened secrets by hand. Here is what the study measured, how Foundry and router paths compare on who can read a prompt, and a five-step audit for Azure teams under GDPR and NIS2.

  • --author By Falak Mahmood
  • --date November 20, 2026
  • --read 15 min read
  • --views 74 views

Dragos sent approximately two million bait prompts through widely used AI services between July and November 2026, each carrying unique URLs and working canary credentials. The results, published on 19 November by Chief Research Officer HD Moore (the creator of Metasploit), are the first large measurement of what happens to a prompt after it leaves your network. Three hosting providers turned out to be compromised. An unnamed Chinese AI lab fetched secret URLs from more than half of the prompts sent to its models, including roughly two-thirds of the requests flagged as zero data retention. A router operator in France took passwords out of two prompts and tried them by hand as SSH root logins.

None of the bait went through Azure, and the article does not mention Microsoft at all. That is the useful part for an Azure team: the study measures the routes your developers reach for when the sanctioned route is slower, pricier or missing a model. What follows is what Dragos found, how each route compares on who can read a prompt, and a five-step audit you can run this week.

What the canaries caught

The method

The proof of concept used free Thinkst Canarytokens: each request carried an AWS key, a DNS name and a URL. Dragos then built its own seeding service, where one seed per prompt derives unique hostnames, URLs and credentials. The AWS keys were, in the article's words, "real keys tied to empty accounts with full logging". Because every canary is unique, any hit maps back to the exact prompt, model, router and privacy setting that carried it.

Dragos also describes "something like traceroute for AI inference". An image URL in the prompt reveals the IP address and HTTP client of every system that fetches it. Small, slightly broken requests make each router wrap the error from the hop behind it in its own format, so a single response can hold three or four nested envelopes. About 2,000 routes were traced this way. Several hundred included a system Dragos could not identify, and a handful sent the image to a company that was not on the advertised path. No tool has been released.

Three compromised providers

The first breach dates to mid-July. The provider initially read it as an internal misconfiguration, then as a partner problem, until evidence showed its internal credentials had been taken. According to Dragos it has not disclosed publicly.

The largest involved a couple of hundred Dragos AWS keys, each sent in exactly one request to one provider during short windows in late October. Dragos writes that "every request we sent during those windows was stolen". Some keys came from requests the provider had rejected before any model ran, which means the attacker was copying traffic at the provider's front door. In the other two breaches, some of the stolen requests were marked as zero data retention.

One adversary held two providers at once. Secrets sent to both were used together, almost immediately, from a single rented server. The playbook after that is familiar: test whether a key is live (sometimes within hours), list S3 buckets, search Secrets Manager and Parameter Store, try to enable hosted models in Amazon Bedrock, create Bedrock API keys, add IAM users. Dragos counted tens of thousands of AWS API calls. One key seeded on 21 October was first used the next day and was still in use three weeks later. From 1 November the attacker added an LLM-driven pentest framework called ARTEX, visible through the user agents artex-agent and artex-judge-r121 in CloudTrail.

Leaks that needed no breach

The breaches are the dramatic half. The quieter half is traffic that leaked through parties behaving exactly as their position allowed.

  • The lab's own scraper. Systems belonging to the unnamed Chinese lab fetched secret URLs from more than half of prompts sent to its models from July to October, usually two or three days after the request. Several US hosting providers resell those models, and prompts sent to them triggered the same scraper. A git client on a different cloud tried to clone private repositories referenced in prompts, but only for prompts that used this lab's models.
  • Marketplace sellers. Sellers on a reseller marketplace listed models at the lowest price, enabled logging and waited. Two prompts through one seller's offer were opened about a day and a half later and the login page was brute-forced. Prompts through a different seller were never touched. The leak follows the seller.
  • Prompt tampering. One marketplace seller "quietly adds about a thousand hidden tokens to every prompt".
  • Undisclosed hops. One router was forwarding some traffic to the marketplace and stopped after Dragos told it. A traced path from a marketplace to Anthropic contained an OpenRouter hop the marketplace did not disclose.
  • Long memories. A large US provider kept crawling a secret URL more than two months after a single call to its search endpoint. And 24 days after one prompt, its secret link was posted in a Feishu chat and opened within two minutes from three desktops and three phones in Beijing and Shanghai. Dragos had only ever disclosed the first eight characters of that link, so whoever shared it was reading from a stored copy of the full prompt.

Dragos draws one structural conclusion from all of this: "a privacy flag only binds the first hop". The flag is a request to the party you called. Whether it propagates depends on every contract behind that party.

The same pattern, with names attached

Dragos names no breached provider, and nothing here should be read as identifying one. Separately, though, the public record this autumn contains a disclosed case with the same shape. Fireworks AI's security bulletin, last updated on 17 November, describes two incidents: one from 3 August to 7 October involving stolen Fireworks credentials and a service vulnerability, and a second from 23 to 30 October through a third-party cloud provider. Fireworks states that it has "not detected any impact to Inference, Training SDK and Nexus, which are all zero data retention (ZDR) by default".

The customer disclosures show how far one vendor's credentials reach. Cursor wrote on 13 November that an attacker obtained an access key it had issued to Fireworks and copied coding-agent traces from a shared storage bucket, exposing API credentials for "less than 0.01% of customers". Cursor moved all inference traffic off Fireworks. Notion's advisory says the incident affected "fewer than 1% of Notion workspaces", that Fireworks had given it "a contractual commitment to zero data retention", and that "data was retained from the Fireworks endpoints that we used". TechCrunch reported that an Oregon Attorney General filing puts the number of Uber drivers whose licence details were taken through the same vendor at 149,346.

Both statements about retention can be true at once, because a batch or analytics feature stores data by design while live inference does not. That is exactly why a ZDR label on a vendor's pricing page tells you little about the specific endpoint your feature calls.

Who can read your prompt, route by route

Dragos ranks the options plainly: host it yourself if you can afford it, otherwise go directly to a first-tier provider, which "can still leak through data sharing agreements or a breach at the provider" but is "the smallest exposure you can buy". The table maps that ranking onto an Azure team's routes.

Route Parties that handle the prompt What binds them
Self-hosted open weights (AKS, Azure VMs, own hardware) You Your own controls
Foundry model sold by Azure, EU Data Zone or regional deployment Microsoft Microsoft Product Terms; prompts not available to OpenAI or other model providers
Claude in Foundry, Hosted on Azure Azure infrastructure, with Anthropic as independent processor Anthropic's terms and Data Processing Addendum
Model vendor's own API, called directly The vendor and its subprocessors Your contract and DPA with that vendor
Router with a pinned upstream provider Router plus the provider you pinned Router terms; your privacy flag reaches the first hop
Reseller marketplace Marketplace, an individual seller, possibly an undisclosed router, then the lab Effectively nothing you can enforce

Three details in the Microsoft documentation matter when you place workloads in the second and third rows.

Deployment type decides geography. For models sold by Azure, prompts and responses are "processed within the customer-specified geography (unless you are using a Global or DataZone deployment type)". A Data Zone deployment in an EU member state means processing "in that or any other European Union Member Nation", while a Global deployment can process "in any geography where the relevant model sold by Azure is deployed". The deployment-types page adds that the EU zone "can include European Free Trade Association (EFTA) countries and regions such as Norway and Switzerland", and "Microsoft can add regions to either data zone without prior notice".

First-party does not mean nothing is stored. Abuse monitoring keeps a separate data store in each geography, and for deployments in the European Economic Area the authorised Microsoft reviewers are located in the EEA. The data-privacy page's change log carries an entry dated 17 November 2026 for the Modified Abuse Monitoring programme, the opt-out that regulated customers use to switch off storage and human review. The page now says it "will no longer be available, beginning on a future date to be announced", with at least 30 days' notice. If your data protection impact assessment leans on that exemption, read the page again.

Claude has two hosting options with different answers. With Hosted on Azure, the default, "your prompts and outputs never leave Azure". With Hosted on Anthropic infrastructure, "data might be processed outside of Azure including outside of your selected Azure geography". In both cases Anthropic "acts as an independent data processor" and manages retention itself. The how-to page lists Data Zone Standard (US) for some Hosted on Azure Claude models and names no EU zone. We covered the retention side in Claude Fable 5.2 in Foundry: $8/$40 and the 30-day catch.

What self-hosting costs, per Dragos

For teams that want the first row of the table on their own desks, the article cites $6,950 for an Nvidia DGX Spark, from $22,299 for a Mac Studio with 512 GB of RAM, about $105,000 for a DGX Station and $570,000 or more for an eight-GPU B300 server.

Hardware is the small line. Operations, patching and the model-quality gap are the large ones, and we worked through that trade in Revolut self-hosts GLM-5.3 at $0.30: your Azure cost math.

A five-step audit for an Azure team

1. Find the routes you do not know about

The sanctioned Foundry deployment is rarely the problem. A coding agent pointed at a router with a personal API key is. If outbound traffic passes through Azure Firewall with resource-specific logs enabled, query the application rule log for router hostnames. Start with the two routers named in the Dragos article and extend the list.

AZFWApplicationRule
| where TimeGenerated > ago(30d)
| where Fqdn has_any ("openrouter.ai", "ai-gateway.vercel.sh")
| summarize Requests = count(), LastSeen = max(TimeGenerated) by SourceIp, Fqdn
| order by Requests desc

Developer laptops will not show up there. For those, check the endpoint proxy or DNS logs, and ask the engineers directly.

2. Check where sanctioned deployments process data

List every deployment with its SKU. A GlobalStandard deployment may process prompts in any Azure region, and a team may have picked it for quota reasons without telling the data protection officer.

az cognitiveservices account deployment list \
  --resource-group my-resource-group \
  --name my-foundry-resource \
  --query "[].{deployment:name, model:properties.model.name, sku:sku.name}" \
  --output table

Microsoft's deployment-types page includes a custom Azure Policy definition that blocks a deployment type by SKU name. Assign it for GlobalStandard on subscriptions that hold personal data, so the next quota shortage produces a ticket and not a silent transfer.

3. Take long-lived secrets out of reach

Dragos puts the root cause in one sentence: "AI agents will happily send whatever they can read, including credentials". The stolen keys in the study were static secrets, and one was still working three weeks after it was taken. A managed identity has no string to paste into a prompt. Turn off key authentication on Foundry resources, then verify it took, because the docs warn that the change "can take up to several hours" to propagate.

Set-AzCognitiveServicesAccount -ResourceGroupName "my-resource-group" -Name "my-resource-name" -DisableLocalAuth $true

# Azure Resource Graph: resources that still accept keys
resources
| where type == "microsoft.cognitiveservices/accounts"
| where properties.disableLocalAuth != true
| project name, resourceGroup, subscriptionId

The built-in policy "Azure AI Services resources should have key access disabled (disable local authentication)" reports the same thing continuously. For the agent's own workspace, keep .env files, kubeconfigs and cloud credentials outside any directory the agent can read, and hand it short-lived tokens scoped to one task. Our LLMjacking key audit has the longer checklist.

4. Seed your own canaries

This is the recommendation that costs an afternoon. Add a fake credential to the context your agents and pipelines read, and treat any use of it as an incident. Thinkst's free Canarytokens service offers an AWS API Keys token and an Azure Login Certificate token; the documentation for the latter notes that "you don't need Azure in your environment to use this token". Alerts arrive by email, and the AWS variant can lag by 2 to 30 minutes.

  1. Generate one token per route, not one for the whole company. A canary that fires only tells you something if you know which path carried it.
  2. Place it where a real secret would plausibly sit: a commented line in a config file inside the agent's working tree, or a field in the system prompt of a pipeline.
  3. Label each token with the route, the model and the date, and send alerts to a monitored mailbox or your incident queue.
  4. Write down in advance what a hit means: stop using that route, rotate everything real that travelled the same way, start the breach clock.

Canaries have a limit that Dragos states directly. They show that a prompt leaked, and they do not show where in the chain. One token per route is how you narrow it down.

5. If a router stays, pin it

Some teams have a legitimate need for a router, typically for models Foundry does not carry. Dragos's advice is to pick upstream providers yourself, verify that privacy settings actually change where requests go, and stay away from reseller marketplaces. On OpenRouter the provider object carries the relevant controls: only restricts a request to listed provider slugs, allow_fallbacks controls whether backup providers may be used (the default is true), data_collection set to "deny" excludes providers that may store data, and zdr restricts routing to zero-data-retention endpoints.

{
  "model": "vendor/model-name",
  "messages": [{ "role": "user", "content": "..." }],
  "provider": {
    "only": ["provider-slug-you-have-vetted"],
    "allow_fallbacks": false,
    "data_collection": "deny",
    "zdr": true
  }
}

Then test it with a canary. A setting you have not verified is a hope.

The Swedish and EU angle

Read through GDPR, the Dragos findings stop being a security curiosity. Every hop that handles a prompt containing personal data is a processor or sub-processor. Article 28 requires that a processor engages another only with the controller's authorisation and passes the same obligations down the chain. A router that forwards to a marketplace without telling you has broken that chain, and as the controller you are the one who answers to IMY for it.

Two findings map onto specific duties. Prompts stored and read in Beijing and Shanghai are a transfer to a third country under Chapter V, with no adequacy decision and, on a reseller route, no transfer mechanism you have signed. And a canary hit on a route that also carried personal data is evidence of a breach, which starts the 72-hour notification window in Article 33. A team that seeds canaries should decide beforehand who makes that call.

For organisations under cybersäkerhetslagen (2025:1506), in force since 15 January 2026 as Sweden's implementation of NIS2, supply-chain security is part of the required risk management measures. An inference route is a supplier relationship. If your supplier register lists Microsoft and omits the router a platform team adopted in the spring, the register is wrong.

This is also the practical reading of the Government Offices decision in October to block ChatGPT and Claude on work computers while keeping Copilot, which we covered in the Regeringskansliet playbook. The stated reason was contractual coverage. Dragos has now supplied the measurement behind that instinct: where no contract reaches, the data was read.

What the study settles

Dragos opens its recommendations with the sentence "None of this means that you should stop using AI services." The study supports that. It also shows that the cheapest route to a model is priced that way for a reason, and that the only way to know what a route does with your prompts is to plant something in them and watch.

subscribe # the AI news that matters, minus the noise

Book a Call

Sources

Tags

Related posts