crawl - A simple recursive web crawler which stores content in the WARC/1.0 format

Age	Commit message (Collapse)	Author
2020-02-17	Propagate the link tag through redirects	ale
	In order to do this we have to plumb it through the queue and the Handler interface, but it should allow fetches of the resources associated with a page via the IncludeRelatedScope even if it's behind a redirect.
2019-01-20	Refactor Handlers in terms of a Publisher interface	ale
	Introduce an interface to decouple the Enqueue functionality from the Crawler implementation.
2018-08-31	Add a simple test for the full WARC crawler	ale

2018-08-31	Explicitly delegate retry logic to handlers	ale
	Makes it possible to retry requests for temporary HTTP errors (429, 500, etc).
2017-12-19	Add tags (primary/related) to links	ale
	This change allows more complex scope boundaries, including loosening edges a bit to include related resources of HTML pages (which makes for more complete archives if desired).