How to Generate a Sitemap and robots.txt on Static Site Deploy

A sitemap lists your URLs for crawlers, and robots.txt points them at it; both should be produced at build time.

Use the framework sitemap integration (Astro @astrojs/sitemap, Next next-sitemap, Hugo built-in) so sitemap.xml is written into the build output, and ship a robots.txt that references its absolute URL.

Steps

• Enable the sitemap integration with your production site/siteUrl.
• Confirm sitemap.xml appears in the build output.
• Add a robots.txt that links the sitemap absolute URL.
• Deploy both as part of the static output.

Build + robots.txt

# Next: next-sitemap runs after build (postbuild)
- run: npm run build && npx next-sitemap

# robots.txt (in public/, copied into the build):
# User-agent: *
# Allow: /
# Sitemap: https://example.com/sitemap.xml

Gotchas

• The sitemap needs the absolute production URL, so set the site config before building.
• A subdirectory deploy must include the base path in every sitemap URL.
• A Disallow: / left in robots.txt silently blocks the whole site from indexing.

Related guides