| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 2022-03-24 | links, crawl: dramatically reduce memory usage | Jordan | |
| to prevent excessive memory usage and OOM crashes, rather than store and pass around response bodies in memory buffers, let's store them temporarily on the filesystem wget-style and delete them when processed | |||
| 2022-02-14 | crawl, readme: record assembled seed URLs to seed_urls file | Jordan | |
| 2022-02-11 | readme: go get -> go install (deprecated), misc updates | Jordan | |
| 2022-02-10 | crawl, readme: max default WARC size 100 MB -> 5 GB | Jordan | |
| 2022-02-10 | readme: typo correction, spacing | Jordan | |
| 2022-02-10 | readme: document changes from upstream | Jordan | |
| 2019-12-04 | Fix installation instructions | ale | |
| 2019-11-13 | Add contact email address | ale | |
| 2019-01-02 | Add multi-file output | ale | |
| The output stage can now write to size-limited, rotating WARC files using a user-specified pattern, so that output files are always unique. | |||
| 2018-09-02 | Fix typo | ale | |
| 2018-09-02 | Explicitly mention the crawler limitations | ale | |
| 2018-08-30 | Mention trickle as a possible bandwidth limiter | ale | |
| Since such bandwidth limiting is not provided by crawl directly, tell users there is another solution. Once/if crawl implements that on its own, that notice could be removed. | |||
| 2018-08-30 | Improve install instructions a bit more | ale | |
| 2018-08-30 | Update installation instructions | ale | |
| 2017-12-19 | Provide better defaults for command-line options | ale | |
| Defaults that are more suitable to real-world site archiving. | |||
| 2017-12-19 | Add a README | ale | |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |