GPTBot Says It's GPTBot. I Stopped Taking Its Word for It

What beru.io.vn hands out today

A quick recap of the surface, verified live rather than from memory. robots.txt currently allows the named AI agents explicitly:

User-agent: *
Allow: /
Allow: /api/openapi.json
Allow: /.well-known/
Disallow: /admin/
Disallow: /api/

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

Host: https://beru.io.vn
Sitemap: https://beru.io.vn/sitemap.xml

/.well-known/api-catalog is a small RFC 6690 web-linking document — one linkset entry pointing at /api, with a service-desc link to /api/openapi.json and a service-doc link to /api-docs. /api/openapi.json itself describes the "Beru Portfolio Webhook API" — the same webhook surface (/api/webhook/blog, /api/webhook/upload, /api/webhook/projects, and so on) that the MCP server behind this post's create_draft call ultimately writes through. All of that is designed for something to read it and act on it. None of it verifies who's reading.

The gap isn't hypothetical

I don't have my own traffic data on crawler spoofing yet — the app-level logging I'm describing below only shipped this week, so I have hours of data, not months, and I'm not going to dress up a few hours of logs as a finding. But the industry-level research on this is not thin. HUMAN Security's crawler-spoofing report put the overall rate at roughly 1 in 18 requests carrying an AI-crawler User-Agent being fake — about 5.7% of all crawler-labeled traffic, over 2 million spoofed requests a day across the sites they measured. The breakdown by target is the more useful number: ChatGPT-User was spoofed at close to a 1:5 ratio, Perplexity-User closer to 1:88. In other words, the more valuable the crawler's reputation, the more it gets impersonated — which is exactly the incentive you'd expect once site owners start treating "AI crawler" as a category worth letting through a firewall unchecked.

Their recommended fix is the boring one: stop trusting the string, check it against something the crawler operator publishes and controls — an IP range, an ASN, or (newer, and I'll get to it) a cryptographic signature.

What OpenAI, Anthropic, and Perplexity actually publish

All three of the crawlers named in that robots.txt block publish a machine-readable IP allowlist. I fetched all three live while writing this, rather than trusting a summary someone else wrote in 2025:

OpenAI — three separate lists, one per bot:

GET https://openai.com/gptbot.json
GET https://openai.com/searchbot.json
GET https://openai.com/chatgpt-user.json

gptbot.json returns a creationTime timestamp and a prefixes array of CIDR blocks:

{
  "creationTime": "2025-10-30T11:00:00.000000",
  "prefixes": [
    { "ipv4Prefix": "132.196.86.0/24" },
    { "ipv4Prefix": "172.182.202.0/25" }
  ]
}

21 prefixes as of this fetch, /24 down to /28.

Anthropic publishes one combined list for ClaudeBot, Claude-User, and Claude-SearchBot:

GET https://claude.com/crawling/bots.json

{
  "creationTime": "2026-08-18T23:56:36Z",
  "prefixes": [
    { "ipv4Prefix": "216.73.216.0/22" },
    { "ipv4Prefix": "34.162.230.222/32" }
  ]
}

That creationTime is 13 days before I fetched it — this is a list the operator is actually rotating, not a static file someone published once and forgot. It's also a meaningful change from Anthropic's own earlier public position: their support article used to say outright that they didn't publish IP ranges because they ride on cloud-provider public IP space, and that this "may change in the future." It changed.

Perplexity publishes two, one per crawler:

GET https://www.perplexity.ai/perplexitybot.json
GET https://www.perplexity.ai/perplexity-user.json

{
  "creationTime": "2025-02-07T16:56:00.000000",
  "prefixes": [
    { "ipv4Prefix": "107.20.236.150/32" },
    { "ipv4Prefix": "18.97.1.228/30" }
  ]
}

Eight entries, mostly single-host /32s, and — worth flagging honestly — a creationTime from February 2025. That's either a very stable crawler fleet or a list that isn't being kept current the way Anthropic's is. Either way, it's a real difference between the three, and it changes how much I trust a match against each one: a stale allowlist can produce false negatives (a legitimate crawler moved off an old IP and now fails verification) as easily as it prevents false positives.

Perplexity-User is worth calling out specifically: their own docs describe it as ignoring robots.txt, because it fires on a live user's question inside the Perplexity app, not on background crawling. That's a real, documented behavior difference between "AI trains on your content" bots and "AI fetches your page because a human just asked about it" bots — and it means a robots-based mental model of AI traffic is already incomplete before spoofing enters the picture.

What I actually built

The CMS behind this blog is a Next.js 15 app with a Prisma-backed Postgres database, self-hosted behind nginx, run under PM2, sitting behind Cloudflare. The addition is small on purpose: a CrawlerHit model and a middleware check that only fires on the agent-facing routes — /llms.txt, /llms-full.txt, /md/*, /.well-known/api-catalog, /api/openapi.json — since those are the paths where "who is actually reading this" matters most.

model CrawlerHit {
  id           String   @id @default(cuid())
  path         String
  claimedAgent String   // parsed from User-Agent, e.g. "GPTBot"
  requestIp    String
  verified     Boolean
  method       String   // "ip-allowlist" | "reverse-dns" | "cf-verified-bot" | "unverified"
  createdAt    DateTime @default(now())

  @@index([claimedAgent, createdAt])
}

The middleware fetches and caches each operator's JSON on an interval (I'm using 6 hours, short enough to pick up Anthropic-style rotation without hammering three endpoints on every request), parses the User-Agent for a known AI-agent token, and checks the request IP against the matching prefix list:

// lib/crawler-verify.ts
import { isMatch } from "cidr-tools";

type AllowList = { creationTime: string; prefixes: { ipv4Prefix: string }[] };

const SOURCES: Record<string, string> = {
  GPTBot: "https://openai.com/gptbot.json",
  "OAI-SearchBot": "https://openai.com/searchbot.json",
  "ChatGPT-User": "https://openai.com/chatgpt-user.json",
  ClaudeBot: "https://claude.com/crawling/bots.json",
  "Claude-User": "https://claude.com/crawling/bots.json",
  PerplexityBot: "https://www.perplexity.ai/perplexitybot.json",
  "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
};

const cache = new Map<string, { list: AllowList; fetchedAt: number }>();
const TTL_MS = 6 * 60 * 60 * 1000;

async function getAllowList(agent: string): Promise<AllowList | null> {
  const url = SOURCES[agent];
  if (!url) return null;
  const cached = cache.get(url);
  if (cached && Date.now() - cached.fetchedAt < TTL_MS) return cached.list;
  const list = (await fetch(url).then((r) => r.json())) as AllowList;
  cache.set(url, { list, fetchedAt: Date.now() });
  return list;
}

export async function verifyCrawler(agent: string, ip: string) {
  const list = await getAllowList(agent);
  if (!list) return { verified: false, method: "unverified" as const };
  const cidrs = list.prefixes.map((p) => p.ipv4Prefix);
  return {
    verified: isMatch(ip, cidrs),
    method: "ip-allowlist" as const,
  };
}

Two things I deliberately did not do. First, I didn't fail closed on an unmatched IP — a request with a GPTBot User-Agent that doesn't match the allowlist still gets served (this blog wants to be read by agents, verified or not), it just gets logged as unverified instead of trusted. Blocking on spoofing suspicion is a much bigger decision than logging it, and I'd want real data before making it. Second, I didn't build my own reverse-DNS fallback yet, even though OpenAI and Google both document it as the standard alternative for verifying their crawlers: do a PTR lookup on the request IP, confirm the hostname it resolves to belongs to the operator's domain (*.googlebot.com, *.crawl.openai.com-style patterns), then do the forward lookup on that hostname and confirm it resolves back to the same IP. It's a stronger check than a static allowlist in one specific way — it doesn't go stale the way Perplexity's February 2025 list plausibly has — but it costs a DNS round trip per unmatched request instead of a cached list comparison, which matters more on a busy site than on a blog. It's on the list for when the traffic volume justifies it; IP-prefix matching against a list that's actively rotated gets most of the value for a fraction of the latency.

Reading the dashboard

The point of writing to CrawlerHit instead of just a log line is that it turns into a table I can actually query, not text I'd have to grep. The dashboard route behind /admin is a page, not a product — one query grouped by claimed agent and verification method:

select
  "claimedAgent",
  "method",
  count(*) filter (where verified) as verified_hits,
  count(*) filter (where not verified) as unverified_hits
from "CrawlerHit"
where "createdAt" > now() - interval '7 days'
group by "claimedAgent", "method"
order by verified_hits + unverified_hits desc;

That's the artifact this post is really about: not a claim about who's currently crawling beru.io.vn — I don't have enough hours of data to say anything honest about that yet — but a place that will tell me, truthfully, once it does.

The layer I'm not replacing

Cloudflare sits in front of this whole stack, and it already runs its own side of this problem through the Verified Bots program, which as of the last year folds in cryptographic verification — Web Bot Auth, HTTP Message Signatures checked at Cloudflare's edge against a public key the bot operator hosts at /.well-known/http-message-signature-directory, tagged and rotation-checked automatically. Cloudflare can pass that verdict down to the origin as a header:

newRequest.headers.set("bot-score", request.cf.botManagement.score);
newRequest.headers.set("verified-bot", request.cf.botManagement.verifiedBot);

I'm not turning that into a second, competing verification system. It's a genuinely stronger signal than IP matching — a signature can't be replayed from a different network the way a spoofed IP-adjacent request sometimes can (HUMAN's report specifically calls out "IP proximity" attacks that place requests just outside a documented range to slip past sloppy CIDR checks). But it only covers bot operators who've enrolled in Cloudflare's program and adopted the signing spec, which as of this year is still ChatGPT agent, Goose, Browserbase, and a handful of others — not GPTBot's background crawl traffic, not ClaudeBot, not PerplexityBot in the form that matters for llms.txt. Until the operators whose crawlers actually read this blog's agent files adopt signing, app-level IP verification is the layer that exists; Cloudflare's header becomes one more method value in that CrawlerHit table the day it applies.

What this doesn't tell you yet

I want to be honest about the limits instead of padding this out with numbers I don't have. A few hours of CrawlerHit rows is not evidence of anything about who's actually visiting — it's a working pipeline, not a finding. IP-prefix matching also isn't a permanent answer: cloud-hosted crawlers share address space with everything else running in the same provider, allowlists lag behind infrastructure changes (Perplexity's own list is the visible example of that risk), and a determined spoofer positioning requests near a real range is a documented evasion, not a hypothetical one. What this buys me is a verified column instead of a blind trust in a header string, and a place to plug in reverse-DNS and signature-based checks as they become worth the engineering time.

The same reverse-proxy-and-deploy pattern — nginx, PM2, Cloudflare, a Prisma-backed Postgres instance doing more than just serving pages — is the baseline I build on for client projects too; benocode runs a Nuxt 3 frontend on a comparable headless setup, and it's the kind of infrastructure work that sits under devops & server management when I do it for someone else's stack instead of my own blog.

Sources

• HUMAN Security — AI Crawler Spoofing: Attackers Impersonate ChatGPT & Perplexity
• OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot documentation
• Anthropic — Does Anthropic crawl the web, and how can site owners block the crawler
• Perplexity — Crawlers documentation
• Cloudflare — Message Signatures are now part of our Verified Bots Program
• Cloudflare — Send Bot Management information to origin