Help search engines index your website. Let A1 Sitemap Generator create your HTML and XML sitemaps, upload sitemaps and ping search engines. Automate all sitemap generator tools.When crawling websites, our sitemap generator supports robots.txt file directives disallow and crawl-delay. Likewise, the crawler engine also handles nofollow, noindex and canonical attribute values in meta and a tags.Website crawling and sitemap generator speed: Our high performance website crawler is able to sustain a high speed, even when crossing 10.000 and 100.000 pages. Do you have websites larger than that? No problem, you can configure our website crawler and tune its speed and memory usage to handle much larger websites.