From caadc00d8dfadc0c9e0237fc7377eb632f500926 Mon Sep 17 00:00:00 2001 From: Jordan Date: Thu, 10 Feb 2022 20:19:27 -0700 Subject: crawl, readme: max default WARC size 100 MB -> 5 GB --- README.md | 1 + 1 file changed, 1 insertion(+) (limited to 'README.md') diff --git a/README.md b/README.md index 0f2298e..c7124a0 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ Notable changes include: a browser * store crawl contents in a dated directory * update ignore regex set per updates to [ArchiveBot](https://github.com/ArchiveTeam/ArchiveBot) +* max default WARC size 100 MB -> 5 GB This tool can crawl a bunch of URLs for HTML content, and save the results in a nice WARC file. It has little control over its traffic, -- cgit v1.2.3-54-g00ecf