This is a Web Crawler implementation written in GOLANG.
Once you have GOLANG on your system, you'll need to install a few dependencies:
~$ go get github.com/llimllib/loglevel
~$ go get golang.org/x/net/html
~$ go get golang.org/x/net/html/atom
After you're done with that, you can start crawling wherever you want:
~$ go run main.go <URL>