Sitemap Generator: crawl your site and get a ready sitemap.xml.

What does the sitemap generator do?

Like a search engine bot, it starts at your home page, follows the links on each page and checks every page it finds. At the end it gives you a sitemap.xml file that follows the sitemaps.org standard and lists only the pages that should be indexed. Download it and upload it to your site.

Which pages go into the sitemap?

• Included: HTML pages that return 200, carry no noindex and whose canonical tag points to themselves.
• Redirects: left out; the target is crawled if it is on your site.
• Canonical elsewhere: the page is left out and the canonical URL is crawled.
• noindex and error pages: left out, but links on noindex pages are still followed.
• robots.txt: blocked URLs are never fetched.
• URLs with parameters: addresses such as ?page=2 or ?utm_source= are skipped by default; you can include them.
• Files: images, PDFs, CSS, JS and other non-page URLs are skipped.

If you use WordPress

Since version 5.5, WordPress generates its own sitemap at /wp-sitemap.xml, and Yoast and Rank Math build their own. These update themselves with every new post and are better than a hand-made file. This tool is mainly for static and custom-built sites without a sitemap. On a WordPress site, use it to see which pages a bot can reach through your links. To check an existing sitemap, use the Sitemap Validator.

Limitations

A crawl fetches at most 500 pages and stays inside the folder of the URL you enter: enter example.com/blog/ and only URLs starting with /blog/ are crawled. Pages are read without running JavaScript, so sites whose links are built only by JavaScript give incomplete results. Orphan pages that no page links to cannot be found. Each visitor can start at most 4 crawls per hour.

The crawl rules, including following only links on the same site under the start URL, skipping non-page file types, removing # fragments, honouring meta robots noindex and nofollow, and keeping only successful HTML pages in the sitemap, are adapted from N. Rusch’s open-source sitemap-generator project (MIT license). Thanks for the work.