DeveloperGet data
Analyze a site's robots.txt for AI crawler access. Returns which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are allowed vs blocked, plus any general rules affecting AI agents. Common diagnostic for "why am I not being cited by ChatGPT?" — often the answer is robots.txt blocking the crawler.
Analyze a site's robots.txt for AI crawler access. Returns which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are allowed vs blocked, plus any general rules affecting AI agents. Common diagnostic for "why am I not being cited by ChatGPT?" — often the answer is robots.txt blocking the crawler. Use it on Xseek with an agent. It runs in the browser on the website and needs no package install.
Open the xseek.io website and ask your browser agent to use robots_txt_scanUse the robots_txt_scan tool on xseek.io with url: <string>. Return the structured result and cite the source page.url string Absolute URL of the site to inspect, e.g. the linked pageVerify the live listing at the source before a production call. Fields marked as a starter contract may differ from what the site expects.