| Mode | Name | Size | |
|---|---|---|---|
| -rw-r--r-- | .gitlab-ci.yml | 100 | logplain |
| -rw-r--r-- | LICENSE | 1060 | logplain |
| d--------- | analysis | 36 | logplain |
| d--------- | cmd | 64 | logplain |
| -rw-r--r-- | crawler.go | 7993 | logplain |
| -rw-r--r-- | crawler_test.go | 1251 | logplain |
| -rwxr-xr-x | gen-ignores.py | 1043 | logplain |
| -rw-r--r-- | ignore_patterns.go | 18380 | logplain |
| -rw-r--r-- | queue.go | 2926 | logplain |
| -rw-r--r-- | scope.go | 3855 | logplain |
| -rw-r--r-- | scope_test.go | 1709 | logplain |
| d--------- | vendor | 113 | logplain |
| d--------- | warc | 35 | logplain |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |