Web Bot Auth signing directory

What is the Web Bot Auth signing directory (/.well-known/http-message-signatures-directory)?

Web Bot Auth is a directory file you publish at /.well-known/http-message-signatures-directory that lists the public keys your server will accept on cryptographically signed HTTP requests from bots and AI agents. Think of it as a JWKS endpoint, but for validating that an incoming bot is who it claims to be—before your WAF drops the request as suspicious. The idea is simple: agents sign their requests using HTTP Message Signatures (RFC 9421), and your server checks those signatures against the public keys you've pre-approved in this directory.

The standard is defined in draft-meunier-http-message-signatures-directory, an IETF draft still in active discussion. It specifies a JSON structure with a keys array, where each entry is a JWK (JSON Web Key) object containing a key identifier (kid), key type (kty), optional cryptographic parameters like crv, x, n, or e, and validity timestamps (created, expires). Cloudflare and a handful of other CDN providers are starting to reference this pattern as they build out signed bot identity systems. Publishing this file—even an empty one—signals that you're aware of the emerging standard and ready to participate in a world where bots authenticate themselves.

Why does Web Bot Auth matter for AI agent traffic and discoverability?

Modern AI agents—ChatGPT's web browser, Claude's API integrations, Perplexity's citation crawler, Cursor's documentation fetcher—all make HTTP requests that look suspiciously like scrapers to your WAF. Traditional bot detection relies on heuristics: rate limits, user-agent parsing, IP reputation, CAPTCHAs. Those heuristics break down when a legitimate agent behaves exactly like a malicious scraper: high request volume, programmatic patterns, headless execution. The result is that useful agents get blocked, your content doesn't get cited, and your product doesn't get indexed by the tools engineers actually use.

By publishing a bot auth directory, you give CDNs and reverse proxies a way to whitelist cryptographically verified agents. An agent that signs its requests with a key you've pre-approved bypasses the usual WAF gauntlet. This directly improves citation rate (your docs show up in Perplexity answers), agent installability (Cursor can fetch your API schemas without retries), and agentic commerce flows (an AI shopping assistant can browse your catalog without hitting rate limits). The business case is straightforward: fewer false positives mean more legitimate agent traffic, which increasingly drives discovery and conversion.

Is publishing a Web Bot Auth directory required for my site?

This check is optional for most sites today. The draft spec is still in flux, adoption is nascent, and most AI agents don't yet sign their requests. If your traffic is primarily human-driven or you don't see meaningful bot volume from AI tools, you can safely ignore this for now.

That said, if you're a documentation site, API provider, e-commerce platform, or any property where agent discoverability matters, publishing even a minimal directory is a low-cost posture signal. It tells Cloudflare, Fastly, and other infrastructure vendors that you're ready to participate when they flip the switch on signed bot verification. The earlier you're in the directory ecosystem, the fewer emergency fixes you'll need when a major CDN starts dropping unsigned bot traffic by default.

What the Web Bot Auth specification (draft-meunier-http-message-signatures-directory) says

The draft-meunier spec defines a JSON document with the following structure:

• keys (required): An array of JWK objects representing public keys you trust.
• kid (required): A unique identifier for the key.
• kty (required): Key type (e.g., "RSA", "EC").
• crv, x, y (for EC keys): Curve and coordinates.
• n, e (for RSA keys): Modulus and exponent.
• created (optional): ISO 8601 timestamp when the key became valid.
• expires (optional): ISO 8601 timestamp when the key expires.

A minimal valid directory looks like this:

{
  "keys": []
}

An empty array is legal and signals awareness of the standard. A populated example with an EC key:

{
  "keys": [
    {
      "kid": "agent-bot-2024-01",
      "kty": "EC",
      "crv": "P-256",
      "x": "WKn-ZIGevcwGIyyrzFoZNBdaq9_TsqzGl96oc0CWuis",
      "y": "y77t-RvAHRKTsSGdIYUfweuOvwrvDD-Q3Hv5J0fSKbE",
      "created": "2024-01-15T00:00:00Z",
      "expires": "2025-01-15T00:00:00Z"
    }
  ]
}

This is an IETF draft, not a ratified RFC. It's being driven by CDN vendors and observability companies; Cloudflare has publicly discussed implementing it. Treat it as an emerging convention, not a finalized standard.

What a compliant Web Bot Auth directory looks like in practice

As of early 2025, public examples are sparse—most implementations are behind CDN configs or internal systems. Companies experimenting with this standard typically start with an empty directory as a capability signal, then add keys once they've negotiated identity schemes with specific bot operators.

A realistic starting point for most organizations:

{
  "keys": []
}

Cloudflare has published guidance suggesting that enterprises pre-populate the directory with keys from trusted agent providers once those providers begin distributing public keys. No major AI agent provider has yet published a universal signing key, so a populated directory today would likely be custom keys for your own first-party bots or negotiated keys with specific partners.

How do I add the Web Bot Auth directory (/.well-known/http-message-signatures-directory) to my site?

1. Create the JSON file. Start with an empty keys array unless you've already negotiated signing keys with specific bot operators:

   {
     "keys": []
   }
   

2. Publish it at /.well-known/http-message-signatures-directory. The exact steps depend on your stack:

   • Static sites (Vercel, Netlify, Cloudflare Pages): Drop http-message-signatures-directory (no extension) into a public/.well-known/ folder. Most static hosts serve .well-known without additional config.

   • Next.js: Place the file in public/.well-known/ or use a route handler at app/.well-known/http-message-signatures-directory/route.ts:

     export async function GET() {
       return Response.json({ keys: [] });
     }
     

   • CDN/reverse proxy: Configure a rewrite rule to serve the JSON from object storage or a Lambda function.

3. Set Content-Type: application/json and cache aggressively (e.g., Cache-Control: public, max-age=3600).

4. Monitor adoption. As agent providers publish signing keys, add them to your directory. This is a forward-looking investment; don't expect immediate traffic changes.

How can I test my Web Bot Auth implementation?

Check if your directory is live and valid:

curl -i https://yoursite.com/.well-known/http-message-signatures-directory

You should see a 200 OK response with Content-Type: application/json and a JSON body containing at minimum {"keys": []}.

Or just run a free scan and we'll check this for you alongside 30+ other agent-readiness signals.

Frequently asked questions

Does Web Bot Auth replace robots.txt or llms.txt for AI agents?

No. Web Bot Auth serves a completely different purpose. While robots.txt and llms.txt declare what content agents can access, /.well-known/http-message-signatures-directory verifies who the agent is through cryptographic signatures. You need all three: robots.txt for crawl permissions, llms.txt for content metadata, and Web Bot Auth for identity verification.

Do ChatGPT, Claude, or Perplexity currently sign their requests with Web Bot Auth?

Not yet. As of early 2025, no major AI agent provider publicly distributes signing keys or implements HTTP Message Signatures (RFC 9421). The standard is IETF draft stage, and adoption is nascent. Publishing an empty directory today is a forward-looking posture signal to CDNs like Cloudflare, not an immediate operational requirement.

Should SaaS API documentation sites publish a Web Bot Auth directory?

Yes, even an empty one. Developer documentation sites see heavy traffic from AI coding assistants like Cursor, GitHub Copilot, and agent frameworks. Publishing /.well-known/http-message-signatures-directory signals to CDNs that you want verified agent traffic whitelisted, reducing false positives when signed bot authentication becomes standard. Low effort, high future upside for API discoverability.

Can I use Web Bot Auth to block specific AI agents?

No. Web Bot Auth is a whitelist mechanism. It lists public keys your server will accept for signature verification. To block agents, use robots.txt (crawl directives) or WAF rules (request blocking). Web Bot Auth helps trusted agents prove their identity to bypass overly aggressive bot detection, not restrict access.

How does Web Bot Auth help e-commerce sites with AI shopping agents?

AI shopping assistants (like ChatGPT's browsing, Google's SGE commerce integrations) often trigger WAF rate limits when browsing product catalogs programmatically. If these agents sign requests and your CDN verifies signatures against /.well-known/http-message-signatures-directory, legitimate shopping agents bypass bot detection, improving product discoverability and reducing abandoned agent sessions during purchase research.

Is Web Bot Auth related to the MCP (Model Context Protocol) standard?

No direct relationship. MCP (/.well-known/mcp.json) defines how agents discover and connect to your data sources and APIs. Web Bot Auth (/.well-known/http-message-signatures-directory) cryptographically verifies agent identity at the HTTP layer. They're complementary: MCP says what capabilities you expose; Web Bot Auth proves who is requesting them.

Does Cloudflare or Vercel natively support Web Bot Auth directory validation?

Cloudflare has publicly discussed implementing verification for /.well-known/http-message-signatures-directory in their WAF and bot management products, but as of early 2025, native support is not generally available. Vercel does not currently validate HTTP Message Signatures. Adoption depends on CDN vendors integrating RFC 9421 signature verification into their platforms.

Do I need to generate my own keys or wait for AI providers to publish theirs?

Wait for AI providers. The keys array in your directory lists their public keys (so you can verify their signatures), not yours. Start with an empty array. Once OpenAI, Anthropic, or Perplexity publish signing keys, you'd add those JWK objects to your directory to whitelist their agents at your CDN layer.