diff options
| author | Jordan <me@jordan.im> | 2022-03-24 09:08:13 -0700 |
|---|---|---|
| committer | Jordan <me@jordan.im> | 2022-03-24 09:08:13 -0700 |
| commit | 6355aa4310ff0c32b056580e812ca6f0e2a5ee2f (patch) | |
| tree | 3a3008d4d50e5e19f6805b1e1e03460e202048f9 /crawler_test.go | |
| parent | a39310f111cef49ff630cc12fdebabc4df37ec28 (diff) | |
| download | crawl-6355aa4310ff0c32b056580e812ca6f0e2a5ee2f.tar.gz crawl-6355aa4310ff0c32b056580e812ca6f0e2a5ee2f.zip | |
links, crawl: dramatically reduce memory usage
to prevent excessive memory usage and OOM crashes, rather than store and
pass around response bodies in memory buffers, let's store them
temporarily on the filesystem wget-style and delete them when processed
Diffstat (limited to 'crawler_test.go')
0 files changed, 0 insertions, 0 deletions