Web Bot Auth: How a Site Knows a Bot Is Really ChatGPT

Your access log says ChatGPT visited this morning. It says that because the request carried a line of text claiming to be ChatGPT. Nothing checked the claim. Anyone with a terminal can send the same line, and plenty of people do.

Here is the whole problem in one command. Run it against your own site and your log will record a visit from GPTBot:

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); \
  compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://example.com/pricing/

That string is a user-agent, a self-reported label a client sends with every request. It is a name badge the visitor writes for itself. Site owners have papered over the gap with IP allowlists and reverse DNS lookups, which hold up until a crawler changes address or routes through a proxy.

Web Bot Auth replaces the claim with proof. The bot signs each request with a private key, publishes the matching public key at a fixed address, and your server checks the math. Google, OpenAI, Cloudflare and AWS are all shipping parts of it already.

Where you are seeing this

If you arrived here from a header you did not recognize, it came from one of these places.

• A raw access log or a request-header dump, showing Signature-Agent on an incoming request.
• Cloudflare’s bot documentation, under Verified Bots, where it is the alternative to IP validation.
• AWS WAF Bot Control, where a request gets tagged as verified or unverified.
• Google’s crawling documentation, on a page labeled experimental.

All four are describing the same mechanism from different sides.

What Web Bot Auth actually is

Web Bot Auth is a way for an automated client to prove who it is, cryptographically, on every request it sends.

It is built on RFC 9421, HTTP Message Signatures, which is already a published standard. RFC 9421 defines how to sign parts of an HTTP request, meaning the method, the path, the host, and whichever headers you name. Web Bot Auth is the layer on top that says which parts a bot should sign, and how a site finds the key to check them with.

The work sits at the IETF in a working group called webbotauth. Its charter is approved and the group is active, with a scope that names search crawlers, web archivers, link checkers, AI training crawlers and AI agents. The protocol document, HTTP Message Signatures for automated traffic, is still an internet draft, on revision 02 as of August 18, 2026.

That matters for one reason. Everything below is shipping in production ahead of the standard being finished, so the header names and key formats can still change under you.

Why the user-agent string was never proof

Cloudflare’s write-up on the problem lists three ways sites try to identify bots today, and why each one leaks.

The user-agent header is self-reported. It is a string the client picks. There is no registry, no signature, and no cost to picking somebody else’s. The curl command at the top of this page is the entire attack.

IP allowlists go stale. The standard advice is to check the request against the crawler’s published IP ranges. That works while the ranges are stable and the traffic is direct. It breaks when infrastructure is shared between services, when ranges change, and when the request passes through a privacy proxy or a VPN. Google’s own guidance still recommends this method, which tells you it is the best thing most sites have, not that it is reliable.

Per-site secrets do not scale. A shared token between one crawler and one site works fine. Multiply it by every crawler and every site on the web and nobody maintains it.

A signature fixes all three at once. It travels with the request, so no allowlist is needed. It is bound to the request contents and a timestamp, so a captured signature cannot be replayed against a different URL. And the key is published once, at a public address, so no arrangement between the two parties is required in advance.

What a signed request looks like

Three headers do the work. Here is what arrives at your server, trimmed for width:

Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");
  created=1786951400; expires=1786951700;
  keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";
  alg="ed25519"; tag="web-bot-auth"
Signature: sig1=:jP4kZG2h9xQvT1mN...:

Reading it in the order that matters:

Signature-Agent

The domain that vouches for this request. It is the header to read first, because it tells you where to go for the key. Google sends g="https://agent.bot.goog". OpenAI’s Cloud browser sends https://chatgpt.com.

Signature-Input

The metadata. created and expires bound the signature to a short window, usually minutes. keyid names which of the agent’s published keys was used. alg="ed25519" is the algorithm. tag="web-bot-auth" marks the purpose, so a signature meant for something else cannot be reused here.

Signature

The signature itself, over the fields listed in Signature-Input. Because @authority is one of them, a signature captured from a request to your site cannot be replayed against a different domain.

The algorithm every live signer uses today is Ed25519, which produces short signatures and verifies fast. The draft allows others, so read alg rather than assuming. Speed is the reason nobody has bothered. This runs on every request, at crawler volume.

Where the public keys live

Each bot operator publishes its keys at a fixed path on its own domain:

https://chatgpt.com/.well-known/http-message-signatures-directory
https://agent.bot.goog/.well-known/http-message-signatures-directory

The response is a JSON Web Key Set, served with the content type application/http-message-signatures-directory+json. It looks like this:

{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U",
      "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"
    }
  ]
}

That is the whole trust model. The Signature-Agent header names a domain, HTTPS proves you reached that domain, and the domain publishes the keys. If you trust chatgpt.com to speak for ChatGPT, you have everything you need.

Try it yourself. This is a real endpoint and it costs nothing to look:

curl -s https://chatgpt.com/.well-known/http-message-signatures-directory

Which agents already sign

Verified against each vendor’s own documentation on September 1, 2026.

Two details in that table are worth pulling out, because they change what you can do with any of it.

Google is explicit that coverage is partial. Its documentation says not all Google user agents use Web Bot Auth, and that it is not yet signing every request from the agents that do. Its recommendation is to keep relying on IP addresses, reverse DNS and user-agent strings alongside the signature. That is a vendor telling you not to enforce on this yet.

AWS scoped its launch narrowly. Web Bot Auth support arrived in AWS WAF on November 21, 2025, and at launch it was available only to customers protecting CloudFront distributions. Bot Control sorts each bot into a verified tier, where identity was confirmed by an Ed25519 signature or a documented IP range, and an unverified tier, where the bot was recognized by user-agent matching and behavioral fingerprinting alone.

What a site owner does today

Three paths, in order of effort.

If you sit behind Cloudflare or AWS WAF, it is already happening. Both check signatures at the edge and both treat verified bots as allowed by default. You are not writing verification code. You are deciding what the verdict should mean, which is a policy question, not an engineering one.

If you run your own edge, verify it yourself. The whole check has five steps. Read Signature-Agent. Fetch that domain’s key directory over HTTPS. Cache it. Find the key named by keyid. Verify the signature over the fields named in Signature-Input. Cloudflare Research publishes TypeScript and Rust libraries alongside a Caddy plugin and a Workers middleware, and the protocol draft carries worked examples.

If you do neither, read your logs and wait. Grep for Signature-Agent and see how much of your bot traffic already carries one. That number is the honest answer to whether any of this is worth your afternoon, and on most sites today it is small.

Whatever you do, do not start rejecting requests that lack a signature. Real crawlers send unsigned requests right now, by their own vendors’ admission, and a rule like that would quietly cost you crawl coverage. If you are weighing bot rules more broadly, our honest answer on blocking AI crawlers covers what those rules do and do not achieve.

What Web Bot Auth does not do

It proves identity, not intent. A verified crawler is still a crawler. The signature confirms the visitor is who it claims to be. It says nothing about what it will do with your content, and there is no field in the protocol where a bot promises anything.

It is not a permissions file. Web Bot Auth answers “who is this”, and files like robots.txt, ai.txt and llms.txt answer “what may you read”. They are different layers, and the signature layer is the one that finally makes the permission layer enforceable. That distinction is the whole subject of llms.txt vs robots.txt, and it is worth getting straight before you write rules against either.

It does not identify the human behind the agent. When an AI agent browses on somebody’s behalf, the signature identifies the operator, not the person. Cloudflare is experimenting with forwarding end-user information separately, using the Forwarded header from RFC 7239. That is a different problem and it is not solved.

It cannot be enforced yet. Partial signing by the biggest participants means the absence of a signature is not evidence of anything. Until coverage is close to total, this is a signal you read, not a gate you close.

The part nobody says out loud

The framing everywhere is safety. Stop impersonation, protect your content, keep the fakes out. That is true and it is not the interesting part.

Reliable bot identity is the missing piece of a bot paywall, and the paywall is what it will mostly be used for. You cannot charge a visitor you cannot name. On June 15, 2026, AWS shipped exactly that, an AI traffic monetization capability that lets content owners charge AI bots at the edge, with prices set per bot category and per verification tier. Verification is not a side feature of that product. It is the billing key.

So the honest read on Web Bot Auth is that it is infrastructure for a web where crawler access is priced rather than assumed. Whether you think that is good depends on whether you sell content or buy it. Either way, it is worth knowing that the identity layer arrived first because the pricing layer needs it.

What it changes for SEO

Three things, and only the first is immediate.

Your logs get trustworthy. Once a meaningful share of AI traffic is signed, “how often does ChatGPT fetch my pages” stops being a guess. Today that question is answered by matching user-agent strings, which is exactly as reliable as the strings are, which is to say not very. Our AI crawlers list exists because those strings are the only handle most sites have. Signatures replace the handle with a fact.

Robots rules become enforceable. A Disallow line today is a request. Against a signed client whose operator you can name, it becomes something you can hold someone to.

And none of it tells you whether you are being cited. This is the trap worth naming. A verified crawl is a fetch, not a mention. A model can read your page and answer the question without ever pointing anyone at you, and that shows up nowhere in your access log. Fetches are input; citations are the outcome, and only one of the two is worth traffic. Measuring the second is a separate job, which is what AI share of voice is for.

ContextBolt SEO does that second job from inside the agent you already work in. Ask, in plain English, whether ChatGPT, Gemini, Claude or Perplexity name your site on the questions you care about, and the answer lands in your chat. Each check saves to your SEO Board, a dashboard where the results sit dated, so you can watch them move. A historical index covers Google AI and ChatGPT as well. It starts with a 7-day free trial, then $35 a month, on Ahrefs-grade data.

That is the split worth carrying away from this page. Web Bot Auth tells you who knocked. It was never going to tell you whether the answer engine quoted you, and those are the two halves of knowing where you stand with AI search. The first half is in your logs. The second is one question away on the ContextBolt SEO trial.