> For AI agents: the complete documentation index is available at https://nhanbe.id.vn/llms.txt, the full documentation bundle is available at https://nhanbe.id.vn/llms-full.txt.

# Robots + Sitemap Inspector

Fetches a site's `robots.txt`, the page itself, and its first sitemap **server-side**, then reports indexing signals and crawler rules, with special attention to AI crawlers.


Inspect

## Guide

- **Page signals**: canonical, indexability, meta robots (default `index, follow`), `X-Robots-Tag` header.
- **robots.txt**: parsed into user-agent groups and declared sitemaps.
- **AI / LLM crawlers**: `blocked` (`Disallow: /`), `limited` (some paths disallowed), `allowed` (no rule or falls back to `*`).
- **URL Path Access Tester**: test a path against selected user-agents using the longest-match rule (Allow wins ties).
- **Sitemap**: first sitemap from robots.txt (or `/sitemap.xml`), classified as index or urlset with count and samples.

## Caveats

- Results reflect what the server returned to this tool; sites may vary by user-agent or region.
- Only the first sitemap is fetched; large sites split URLs across child sitemaps.
- A missing robots.txt means all crawlers are allowed.