aboutsummaryrefslogtreecommitdiff
path: root/robots.txt
AgeCommit message (Collapse)Author
2022-12-19ui-tree,ui-blame: bail from blame if blob is binaryJune McEnroe
This avoids piping binary blobs through the source-filter. Also prevent robots from crawling it, since it's expensive. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
2013-08-12robots.txt: disallow access to snapshotsJason A. Donenfeld
My dmesg is filled with the oom killer bringing down processes while the Bingbot downloads every snapshot for every commit of the Linux kernel in tar.xz format. Sure, I should be running with memory limits, and now I'm using cgroups, but a more general solution is to prevent crawlers from wasting resources like that in the first place. Suggested-by: Natanael Copa <ncopa@alpinelinux.org> Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>