| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 2022-02-10 | crawl, scope: recurse infinitely by default | Jordan | |
| 2018-09-02 | Add --exclude and --exclude-file options | ale | |
| Allow users to add to the exclude regexp lists easily. | |||
| 2017-12-19 | Add tags (primary/related) to links | ale | |
| This change allows more complex scope boundaries, including loosening edges a bit to include related resources of HTML pages (which makes for more complete archives if desired). | |||
| 2015-06-29 | improve queue code; golint fixes | ale | |
| The queuing code now performs proper lease accounting, and it will not return a URL twice if the page load is slow. | |||
| 2015-06-28 | add ignore list from ArchiveBot | ale | |
| 2014-12-20 | add tests to scope.go | ale | |
| 2014-12-20 | make Scope checking more modular | ale | |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |