aboutsummaryrefslogtreecommitdiff
BranchCommit messageAuthorAge
mastercrawler: close temporary descriptor in advance of defer (performance)Jordan8 weeks
 
 
AgeCommit messageAuthor
2022-03-24crawler: close temporary descriptor in advance of defer (performance)HEADmasterJordan
2022-03-24crawler: continue crawl when context deadline exceeded (timeout)Jordan
2022-03-24crawler: rm temporary body store once processed in advance of deferJordan
2022-03-24misc: update handler signatures, tests, housekeepingJordan
2022-03-24links, crawl: dramatically reduce memory usageJordan
2022-02-14client, crawl: fix/simplify net.Dialer overridesJordan
2022-02-14crawl, readme: record assembled seed URLs to seed_urls fileJordan
2022-02-11readme: go get -> go install (deprecated), misc updatesJordan
2022-02-10misc: add MakefileJordan
2022-02-10crawl, readme: max default WARC size 100 MB -> 5 GBJordan
[...]
 
Clone
https://git.jordan.im/crawl
http://7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion/crawl