What it measures

Time to First Byte (TTFB) measures how long it takes your server to respond with the first byte of data after receiving a request. This is the time AI crawlers wait before they start receiving your content.

Why it matters for AI

AI crawlers operate at massive scale and have strict timeout limits. If your server takes too long to respond, the crawler moves on to the next site. GPTBot and PerplexityBot typically have timeouts between 5-10 seconds, but slow TTFB compounds with content transfer time.

TTFB	Rating	AI crawler impact
< 500ms	Fast	AI crawlers index quickly and reliably
500–1,000ms	Acceptable	Usually fine but leaves less time for content transfer
1,000–1,500ms	Slow	Risk of timeouts, especially for large pages
> 1,500ms	Very slow	Most AI crawlers will likely timeout

How to improve

1. Use a CDN — Serve content from servers close to crawlers (most are in US datacentres)
2. Optimise server-side rendering — Reduce database queries and computation per request
3. Enable HTTP/2 or HTTP/3 — Faster protocol negotiation reduces overhead
4. Implement caching — Cache pages at the CDN or reverse proxy level