| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 2019-01-02 | Add multi-file output | ale | |
| The output stage can now write to size-limited, rotating WARC files using a user-specified pattern, so that output files are always unique. | |||
| 2018-08-31 | Use a buffered Writer for WARC output | ale | |
| 2018-08-31 | Improve error checking | ale | |
| Detect write errors (both on the database and to the WARC output) and abort with an error message. Also fix a bunch of harmless lint warnings. | |||
| 2017-12-18 | Update location of the uuid package | ale | |
| 2015-07-03 | minor golint fixes | ale | |
| 2015-06-28 | fix timestamp format | ale | |
| The WARC-Date fields now are UTC times in proper ISO-8601 format. This makes pywb and other tools happy. | |||
| 2014-12-20 | move the WARC code into its own package | ale | |
| Now generates well-formed, indexable WARC files. | |||
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |