Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
-
Updated
May 10, 2026 - Python
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
User-agent simulator for major LLM crawlers: measures response code, content length, and JS-dependency
"The AI Discoverability Standard. A structured hub for businesses and products to become understandable, verified, and citable by modern LLM crawlers.
Add a description, image, and links to the llm-crawlers topic page so that developers can more easily learn about it.
To associate your repository with the llm-crawlers topic, visit your repo's landing page and select "manage topics."