Age | Commit message (Expand) | Author |
---|---|---|
2022-02-10 | ignore_patterns: update to reflect current ArchiveBot ignore set | Jordan |
2015-06-28 | add ignore list from ArchiveBot | ale |
index : crawl | ||
A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
aboutsummaryrefslogtreecommitdiff |
Age | Commit message (Expand) | Author |
---|---|---|
2022-02-10 | ignore_patterns: update to reflect current ArchiveBot ignore set | Jordan |
2015-06-28 | add ignore list from ArchiveBot | ale |