| Mode | Name | Size | |
|---|---|---|---|
| d--------- | analysis | 36 | logplain |
| d--------- | cmd | 64 | logplain |
| -rw-r--r-- | crawler.go | 7619 | logplain |
| -rwxr-xr-x | gen-ignores.py | 1043 | logplain |
| -rw-r--r-- | ignore_patterns.go | 18380 | logplain |
| -rw-r--r-- | scope.go | 2690 | logplain |
| -rw-r--r-- | scope_test.go | 1678 | logplain |
| d--------- | warc | 35 | logplain |
 |
index : crawl | |
| A simple recursive web crawler which stores content in the WARC/1.0 format | Jordan |
| aboutsummaryrefslogtreecommitdiff |