Markdown content negotiation

What is Markdown content negotiation?

Markdown content negotiation is a lightweight extension of HTTP's standard content negotiation mechanism. When a client—often an AI agent or LLM-powered tool—sends an Accept: text/markdown header, the server responds with Content-Type: text/markdown and returns the page's main content as Markdown instead of HTML. The page remains the same; only the representation changes.

This is standard HTTP behavior defined in RFC 9110 §12. Browsers request text/html, receive HTML. Agents request text/markdown, receive Markdown. The server strips navigation, sidebars, cookie banners, and other layout artifacts, leaving the document's semantic core. It's the same principle behind serving JSON to API clients and HTML to browsers—just applied to a new class of consumer.

Why does Markdown content negotiation matter for AI agents and LLMs?

AI agents read pages differently than humans. When Claude or ChatGPT ingests a product page or documentation article via HTML, they tokenize <div class="hero-wrapper">, <script src="analytics.js">, and hundreds of navigation links that have nothing to do with the content. Markdown strips all of that. Benchmarks from several CDN providers show LLMs tokenize clean Markdown 30–40% more efficiently than the same content wrapped in HTML. Fewer tokens mean faster processing, lower API costs for the agent, and—critically—more room in the context window for your actual content.

This translates to tangible outcomes. Perplexity and ChatGPT are more likely to cite pages they can parse cleanly. Agentic shopping assistants (Google SGE, Shopify Sidekick) can extract product specs without fighting a React hydration bundle. Developer tools like Cursor and GitHub Copilot Workspace fetch documentation faster. If your revenue model depends on discovery, comparison, or integration, serving Markdown to agents is the difference between being indexed properly and being summarized incorrectly—or skipped entirely.

Is Markdown content negotiation required, recommended, or optional for my site?

This check is optional for most sites. If your content is static and your business doesn't depend on agent-driven traffic—say, a personal blog or a restaurant menu—the effort isn't worth it. Browsers will never send Accept: text/markdown, so you're building infrastructure for a narrow audience.

It becomes recommended if you publish documentation, product catalogs, or any content where agents are already a measurable traffic source. If you see referrers from ChatGPT, Perplexity, or enterprise AI tools in your logs, you have demand. And it becomes required if you're building for agent-first distribution: API marketplaces, developer platforms, or any service selling through LLM-mediated discovery.

What the standard says about Markdown content negotiation

RFC 9110 defines content negotiation as a pattern, not a prescription. The server inspects the Accept header and chooses a representation that matches. For Markdown negotiation:

• The client sends Accept: text/markdown (or Accept: text/markdown, text/html;q=0.9 to fall back).
• The server responds with Content-Type: text/markdown and a plain-text Markdown body.
• The response should include Vary: Accept so caches know this resource has multiple representations.

There is no formal IANA registration for text/markdown as a media type yet, but it follows the convention established by text/html and text/plain. The Markdown MIME type has been in use since the early 2000s and is widely recognized by parsers.

Minimum valid example:

HTTP/2 200
Content-Type: text/markdown
Vary: Accept

# Product Overview

The **X200 Widget** supports:
- Bluetooth 5.3
- USB-C charging
- IP67 water resistance

Price: $149

What good Markdown content negotiation looks like in production

Framer and Anthropic both implement Markdown negotiation on their documentation sites. When you request a page with Accept: text/markdown, you receive a clean Markdown document with no navigation chrome, no JS bundles, and no CSS. The content is identical to what a browser sees, just stripped to its semantic structure.

Here's what a negotiated response might look like (simplified):

# API Authentication

All requests require a bearer token in the `Authorization` header:

    Authorization: Bearer sk_live_abc123

Tokens are scoped to your workspace. Rotate them in the dashboard under **Settings → API Keys**.

## Rate limits

- Free tier: 100 requests/minute
- Pro tier: 1,000 requests/minute

Contact support to raise limits.

Companies like Stripe, Linear, and Vercel are exploring similar approaches. If you're building this, study their implementations—they've ironed out edge cases around images, code blocks, and tables.

How do I implement Markdown content negotiation on my site?

1. Detect the Accept header. In a CDN worker (Cloudflare, Fastly) or framework middleware (Next.js, Express), check if the client requests text/markdown. Most agents send it alongside text/html with a quality value: Accept: text/markdown, text/html;q=0.9.

2. Render your page as Markdown. If you generate pages from Markdown source (like most static-site generators), serve the original. If your content lives in a CMS, write a transformation layer. Use a library like Turndown (JS) or html2text (Python) to convert HTML to Markdown server-side. Strip non-content nodes: nav, footer, ads, cookie banners.

3. Set the correct headers. Respond with Content-Type: text/markdown and Vary: Accept so CDNs cache both representations separately.

4. Test across browsers and agents. Browsers should still get HTML. Only clients explicitly requesting Markdown should see it.

Example in a Cloudflare Worker:

export default {
  async fetch(request, env) {
    const accept = request.headers.get('Accept') || '';
    if (accept.includes('text/markdown')) {
      const markdown = await fetchMarkdownVersion(request.url);
      return new Response(markdown, {
        headers: {
          'Content-Type': 'text/markdown',
          'Vary': 'Accept'
        }
      });
    }
    return fetch(request); // pass through to origin
  }
};

How can I test Markdown content negotiation myself?

curl -H "Accept: text/markdown" https://yoursite.com/docs/getting-started

You should see plain Markdown in the response, not HTML. Check the Content-Type header is text/markdown.

Or just run a free scan and we'll check this for you alongside 30+ other agent-readiness signals.

Frequently asked questions

Does Markdown content negotiation replace my HTML pages?

No. Your HTML pages remain unchanged for browsers. Markdown content negotiation uses HTTP's Accept header to serve an alternate representation only when clients explicitly request text/markdown. Browsers still receive HTML. It's the same mechanism APIs use to serve JSON to some clients and XML to others—same resource, different format.

Will implementing Markdown content negotiation hurt my SEO rankings?

No. Google and other search engines crawl with Accept: text/html and will continue receiving your HTML pages. Markdown negotiation only activates for clients sending Accept: text/markdown. The Vary: Accept header ensures CDNs cache both representations separately, so there's no risk of search bots receiving Markdown accidentally or penalizing duplicate content.

Is Markdown content negotiation worth it for e-commerce product pages?

Yes, especially if you track agent referrals. Agentic shopping assistants like Google SGE and Perplexity tokenize Markdown 30–40% more efficiently than HTML, improving your chances of accurate product citations. Clean Markdown helps agents extract specs, pricing, and availability without parsing React bundles or ad scripts. If discovery drives conversions, this matters.

Should SaaS documentation sites implement Markdown content negotiation?

Recommended. Developer-facing agents (Cursor, GitHub Copilot, ChatGPT) increasingly request documentation via Accept: text/markdown. Companies like Anthropic, Stripe, and Framer already serve Markdown-negotiated docs. If your users ask LLMs "how do I authenticate with X API," clean Markdown increases citation accuracy and reduces context-window waste on navigation chrome.

How does Markdown content negotiation differ from publishing an llms.txt file?

They're complementary. llms.txt is a static index that tells agents where your key resources live. Markdown content negotiation is dynamic HTTP behavior that serves how those resources are formatted when requested. Use llms.txt to guide discovery, and Markdown negotiation to deliver clean, token-efficient content once the agent fetches a page.

Can I implement Markdown content negotiation on WordPress or static site generators?

Yes. For static generators (Hugo, Jekyll, Next.js), serve the original Markdown source when Accept: text/markdown is detected—usually via middleware or CDN worker. For WordPress, write a plugin that converts post content to Markdown server-side (using libraries like Turndown) and checks the Accept header before rendering. Cloudflare Workers and Vercel Edge Functions simplify both.

Do I need to register text/markdown as a MIME type to use Markdown content negotiation?

No. While text/markdown lacks formal IANA registration, it follows the same convention as text/html and text/plain and has been recognized by parsers since the early 2000s. Agents already send Accept: text/markdown in practice. The standard relies on RFC 9110's content negotiation pattern, not a registered media type.

What happens if I serve Markdown but forget the Vary: Accept header?

CDNs may cache the Markdown response and serve it to browsers requesting HTML, breaking your site for human visitors. The Vary: Accept header tells caches to store separate versions for each Accept value. Without it, you risk serving the wrong representation to the wrong client. Always include Vary: Accept in negotiated responses.