| Age | Commit message (Expand) | Author |
|---|---|---|
| 2017-12-19 | Exit gracefully on signals | ale |
| 2017-12-19 | Use a global http.Client with sane settings | ale |
| 2017-12-19 | Update cmd/links to new scope syntax | ale |
| 2017-12-19 | Add tags (primary/related) to links | ale |
| 2015-07-03 | minor golint fixes | ale |
| 2015-06-29 | clean up the state directory when done | ale |
| 2015-06-28 | add ignore list from ArchiveBot | ale |
| 2014-12-20 | move URLInfo logic into the Crawler itself | ale |
| 2014-12-20 | make Scope checking more modular | ale |
| 2014-12-20 | move link extraction to a common location | ale |
| 2014-12-20 | move the WARC code into its own package | ale |
| 2014-12-19 | initial commit | ale |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |