Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
-
Updated
May 10, 2026 - Python
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
User-agent simulator for major LLM crawlers: measures response code, content length, and JS-dependency
Add a description, image, and links to the compliance-geo topic page so that developers can more easily learn about it.
To associate your repository with the compliance-geo topic, visit your repo's landing page and select "manage topics."