Age | Commit message (Expand) | Author |
---|---|---|
2018-12-27 | Normalize URLs before checking if they are in scope | ale |
2018-08-31 | Do not drop /index.html at the end of URLs | ale |
2018-08-31 | Explicitly delegate retry logic to handlers | ale |
2018-08-31 | Improve error handling, part two | ale |
2018-08-31 | Improve error checking | ale |
2017-12-19 | Exit gracefully on signals | ale |
2017-12-19 | Simplify redirectHandler.Handle | ale |
2017-12-19 | Add tags (primary/related) to links | ale |
2017-12-18 | Switch to github.com/syndtr/goleveldb | ale |
2015-07-03 | minor golint fixes | ale |
2015-06-29 | clean up the state directory when done | ale |
2015-06-29 | improve queue code; golint fixes | ale |
2014-12-20 | move URLInfo logic into the Crawler itself | ale |
2014-12-20 | add a prefix iterator to gobDb | ale |
2014-12-20 | make Scope checking more modular | ale |
2014-12-20 | move the WARC code into its own package | ale |
2014-12-19 | initial commit | ale |