| Branch | Commit message | Author | Age | |
|---|---|---|---|---|
| master | crawler: close temporary descriptor in advance of defer (performance) | Jordan | 6 months | |
| Age | Commit message | Author | ||
| 2022-03-24 | crawler: close temporary descriptor in advance of defer (performance)HEADmaster | Jordan | ||
| 2022-03-24 | crawler: continue crawl when context deadline exceeded (timeout) | Jordan | ||
| 2022-03-24 | crawler: rm temporary body store once processed in advance of defer | Jordan | ||
| 2022-03-24 | misc: update handler signatures, tests, housekeeping | Jordan | ||
| 2022-03-24 | links, crawl: dramatically reduce memory usage | Jordan | ||
| 2022-02-14 | client, crawl: fix/simplify net.Dialer overrides | Jordan | ||
| 2022-02-14 | crawl, readme: record assembled seed URLs to seed_urls file | Jordan | ||
| 2022-02-11 | readme: go get -> go install (deprecated), misc updates | Jordan | ||
| 2022-02-10 | misc: add Makefile | Jordan | ||
| 2022-02-10 | crawl, readme: max default WARC size 100 MB -> 5 GB | Jordan | ||
| [...] | ||||
| Clone | ||||
| https://git.jordan.im/crawl | ||||
| http://7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion/crawl | ||||
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |