2026-05-31 · 1 min read
robots.txt and sitemap.xml basics
robots guides crawlers; sitemap lists URLs you want discovered—use both on launch day.
SEOcrawlingsitemap
Key takeaways
- robots.txt is not authentication—sensitive paths still need server-side protection.
- Submit sitemap.xml in Google Search Console after deploy, then ping IndexNow on large batches.
Pair the two files
Generate robots.txt with your sitemap URL in the Sitemap directive.
Build sitemap.xml from your canonical locale URLs only—skip noindex variants.
FAQ
Should I disallow /api?
Yes—block non-page endpoints to save crawl budget.
How many URLs per sitemap?
Stay under 50k URLs per file; split by locale or section if needed.