Monzter ...
- is an HTML link crawler with configurable maximum depth and rate limiting
- crawls sites depth-first
- will only visit pages located on the same hostname as the entrypoint (links to other hostnames will still be listed in the output)
go get github.com/sauerbraten/monzter
monzter [-depth=<D>] [-rate=<R>] <entrypoint>
D
is the maximum depth of pages to crawl (defaults to 1) and R
is the maximum number of requests per second (defaults to 10); entrypoint
is the page to start the crawling on and must be an absolute URL.
monzter -depth 2 -rate 20 https://news.ycombinator.com/