| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 2018-08-31 | Improve error checking | ale | |
| Detect write errors (both on the database and to the WARC output) and abort with an error message. Also fix a bunch of harmless lint warnings. | |||
| 2017-12-19 | Simplify redirectHandler.Handle | ale | |
| 2017-12-18 | Switch to github.com/syndtr/goleveldb | ale | |
| The native Go implementation of LevelDB. | |||
| 2015-06-29 | improve queue code; golint fixes | ale | |
| The queuing code now performs proper lease accounting, and it will not return a URL twice if the page load is slow. | |||
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |