crawl - A simple recursive web crawler which stores content in the WARC/1.0 format

about summary refs log tree commit diff

Mode	Name	Size
d---------	cmd	64	log plain
-rw-r--r--	crawler.go	7549	log plain
-rw-r--r--	warc.go	2274	log plain