This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.
-
Updated
Mar 3, 2023 - TypeScript
This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.
GoSitemap2Md is a Golang program that generates a sitemap URL in Markdown format and stores the URLs in a urls.json file for easy adding of new URLs. This tool simplifies the process of generating and maintaining a sitemap for your website.
a python script that crawls website sitemap in a very quick way with multi threading and extract, write SEO based data to CSV file
Add a description, image, and links to the sitemap-crawler topic page so that developers can more easily learn about it.
To associate your repository with the sitemap-crawler topic, visit your repo's landing page and select "manage topics."