🗺️ Robots.txt & Sitemap Generator
🗺️ Robots.txt & Sitemap Generator
Draft a valid robots.txt and XML sitemap from your own URL list. Add paths or full URLs, set crawl rules and download the result. This is a local generator, not a crawler: it never visits or uploads your site.
Blank lines and lines starting # are ignored; duplicates are removed.
Use robots directives only for crawl preference — they do not secure a private page.
Important: robots.txt is not access control
A disallow rule asks compliant crawlers not to fetch a URL. It cannot protect confidential content and it is not a guaranteed way to remove an already-indexed page. Protect private routes with authentication or remove them from the public web; use a sitemap only for canonical, indexable URLs.