| Mode | Name | Size | |
|---|---|---|---|
| -rw-r--r-- | .gitlab-ci.yml | 221 | logplain |
| -rw-r--r-- | LICENSE | 1060 | logplain |
| -rw-r--r-- | README.md | 3354 | logplain |
| d--------- | analysis | 77 | logplain |
| -rw-r--r-- | client.go | 1734 | logplain |
| d--------- | cmd | 64 | logplain |
| -rw-r--r-- | crawler.go | 10367 | logplain |
| -rw-r--r-- | crawler_test.go | 1283 | logplain |
| d--------- | debian | 139 | logplain |
| -rwxr-xr-x | gen-ignores.py | 1043 | logplain |
| -rw-r--r-- | go.mod | 468 | logplain |
| -rw-r--r-- | go.sum | 4985 | logplain |
| -rw-r--r-- | ignore_patterns.go | 18380 | logplain |
| -rw-r--r-- | queue.go | 3189 | logplain |
| -rw-r--r-- | scope.go | 3961 | logplain |
| -rw-r--r-- | scope_test.go | 1709 | logplain |
| d--------- | vendor | 113 | logplain |
| d--------- | warc | 111 | logplain |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |