AI crawlers & llms.txt

AI assistants read the web through crawlers, fetchers and agent protocols, and most of the rules are still being written. I run beru.io.vn open to AI bots, with llms.txt, a markdown version of every page, an API catalog and an OAuth-protected MCP server, and I use it as a test bed. The posts in this topic check what those bots and standards actually do: verifying GPTBot, ClaudeBot and PerplexityBot against their published IP lists, robots.txt signals like Content-Usage, Web Bot Auth signatures, WebMCP, the MCP Registry, and whether files like llms.txt get read at all. Expect logs and real requests rather than predictions.