| Mode | Name | Size | |
|---|---|---|---|
| -rw-r--r-- | .gitlab-ci.yml | 100 | logplain |
| -rw-r--r-- | LICENSE | 1060 | logplain |
| -rw-r--r-- | README.md | 2895 | logplain |
| d--------- | analysis | 36 | logplain |
| -rw-r--r-- | client.go | 687 | logplain |
| d--------- | cmd | 64 | logplain |
| -rw-r--r-- | crawler.go | 10180 | logplain |
| -rw-r--r-- | crawler_test.go | 1277 | logplain |
| -rwxr-xr-x | gen-ignores.py | 1043 | logplain |
| -rw-r--r-- | ignore_patterns.go | 18380 | logplain |
| -rw-r--r-- | queue.go | 3155 | logplain |
| -rw-r--r-- | scope.go | 3961 | logplain |
| -rw-r--r-- | scope_test.go | 1709 | logplain |
| d--------- | vendor | 113 | logplain |
| d--------- | warc | 35 | logplain |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |