For AI agents: the complete documentation index is available at https://nhanbe.id.vn/llms.txt, the full documentation bundle is available at https://nhanbe.id.vn/llms-full.txt, and this page is available as Markdown at https://nhanbe.id.vn/others/tools/robots-sitemap-inspector.md.
Nhan AEM Notes

Inspect robots.txt, sitemap, canonical, noindex, X-Robots-Tag and AI crawler rules (GPTBot, ClaudeBot...) for any domain.

Robots + Sitemap Inspector

Fetches a site's robots.txt, the page itself, and its first sitemap server-side, then reports indexing signals and crawler rules, with special attention to AI crawlers.

Guide

  • Page signals: canonical, indexability, meta robots (default index, follow), X-Robots-Tag header.
  • robots.txt: parsed into user-agent groups and declared sitemaps.
  • AI / LLM crawlers: blocked (Disallow: /), limited (some paths disallowed), allowed (no rule or falls back to *).
  • URL Path Access Tester: test a path against selected user-agents using the longest-match rule (Allow wins ties).
  • Sitemap: first sitemap from robots.txt (or /sitemap.xml), classified as index or urlset with count and samples.

Caveats

  • Results reflect what the server returned to this tool; sites may vary by user-agent or region.
  • Only the first sitemap is fetched; large sites split URLs across child sitemaps.
  • A missing robots.txt means all crawlers are allowed.