DeveloperSearch & find
Return a read-only RFC 9309 robots.txt access report for AI and search crawlers. Accepts robots.txt text or a public robots.txt URL plus an optional path. Includes merged matching groups, octet-normalized longest-match decisions, per-crawler evidence, transport status, crawler-registry provenance and policy snippets.
Return a read-only RFC 9309 robots.txt access report for AI and search crawlers. Accepts robots.txt text or a public robots.txt URL plus an optional path. Includes merged matching groups, octet-normalized longest-match decisions, per-crawler evidence, transport status, crawler-registry provenance and policy snippets. Use it on Admintoolkit with an agent. It runs in the browser on the website and needs no package install.
Open the admintoolkit.io website and ask your browser agent to use admintoolkit_check_ai_crawler_accessUse the admintoolkit_check_ai_crawler_access tool on admintoolkit.io with content: <string>, crawlers: <array>, path: <string>, url: <string>. Return the structured result and cite the source page.content string Pasted robots.txt content to validate locally, e.g. 'User-agent: GPTBot'.crawlers array Crawler tokens to evaluate as an array, e.g. ['GPTBot','Google-Extended'].path string Path to evaluate against robots.txt rules, e.g. '/' or '/tools/'.url string Robots.txt URL or bare domain to fetch, e.g. 'the linked page'.Verify the live listing at the source before a production call. Fields marked as a starter contract may differ from what the site expects.