From f11f012f05e717bfe813e66aa19b59b3c0d1d483 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Thu, 12 Oct 2023 12:27:58 -0400 Subject: Move all MD-only specifications into spec/. --- README.md | 69 ---------------------------------------- dos-spec.md | 97 -------------------------------------------------------- spec/README.md | 69 ++++++++++++++++++++++++++++++++++++++++ spec/dos-spec.md | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 166 insertions(+), 166 deletions(-) delete mode 100644 README.md delete mode 100644 dos-spec.md create mode 100644 spec/README.md create mode 100644 spec/dos-spec.md diff --git a/README.md b/README.md deleted file mode 100644 index 944f029..0000000 --- a/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Tor specifications - -This repository holds the specifications that describe how Tor works. -They try to present Tor's protocols in sufficient detail to allow -the reader to implement a compatible implementation of Tor without ever -having to read the Tor source code. - -The [proposals](/proposals) directory holds our design proposals. These -include historical documents that have now been merged into the main -specifications. For more information on the proposal process, including an -explanation of how to make new proposals, see -[001-process.txt](/proposals/001-process.txt). - -## What you can find here - -Tor's specification is pretty big, and we've broken it into a bunch of -files. - -* General interest - * [tor-spec.txt](tor-spec.txt) - contains the specification for the core Tor protocol - itself; this is a good place to start reading. - * [cert-spec.txt](cert-spec.txt) describes a certificate format used - in the other parts of the protocol. - * [dir-spec.txt](dir-spec.txt) specifies the operations and formats used to - maintain a view of the network directory. - * [padding-spec.txt](padding-spec.txt) describes a set of padding mechanisms - used to impede traffic analysis. - * [version-spec.txt](version-spec.txt) explains how to parse Tor - version numbers. - * [glossary.txt](glossary.txt) is a glossary of terms used - in the other specifications. -* Client operations - * [address-spec.txt](address-spec.txt) lists a set of special - addresses that Tor handles differently from the regular DNS system. - * [guard-spec.txt](guard-spec.txt) explains the "guard node" algorithm - that Tor clients use to avoid sampling attacks. - * [path-spec.txt](path-spec.txt) explains how clients choose their paths - through the Tor network. - * [socks-extensions](socks-extensions.txt) specifies Tor-specific - extensions to the SOCKS protocol. -* Onion services - * [rend-spec-v2.txt](rend-spec-v2.txt) is the old, deprecated version - of the onion service protocol. - * [rend-spec-v3.txt](rend-spec-v3.txt) is the current version of the - onion service protocol. -* Censorship resistance - * [bridgedb-spec.txt](bridgedb-spec.txt) explains how the `bridgedb` - server gives out bridges to censored clients. - * [gettor-spec.txt](gettor-spec.txt) describes the `gettor` tool, - which is used to download Tor in censored areas. - * [pt-spec.txt](pt-spec.txt) describes the protocol that Tor clients - and relays use to communicate with pluggable transports used for - traffic obfuscation. -* Directory authorities - * [bandwidth-file-spec.txt](bandwidth-file-spec.txt) specifies the - file format used by bandwidth-measuring tools to report their - observations to directory authorities. - * [srv-spec.txt](srv-spec.txt) specifies the protocol that - directory authorities use to securely compute shared random values - for the network. -* Controller protocol - * [control-spec.txt](control-spec.txt) explains the protocol used by - controllers to communicate with a running Tor process. -* Miscellaneous - * [dir-list-spec.txt](dir-list-spec.txt) explains the format used by - tools like the fallback directory scripts to output a list of - Tor directories for inclusion in the Tor source code. - * The [attic](attic) directory has obsolete or historical documents. diff --git a/dos-spec.md b/dos-spec.md deleted file mode 100644 index 04470b2..0000000 --- a/dos-spec.md +++ /dev/null @@ -1,97 +0,0 @@ -# Denial-of-service prevention mechanisms in Tor - -This document is incomplete; it describes some mechanisms that Tor -uses to avoid different kinds of denial-of-service attacks. - -## Handling low-memory conditions - -(See also `tor-spec.txt`, section 8.1.) - -The Tor protocol requires clients, onion services, relays, and -authorities to store various kind of information in buffers and -caches. But an attacker can use these buffers and queues to queues -to exhaust the memory of the a targeted Tor process, and force the -operating system to kill that process. - -Worse still, the ability to kill targeted Tor instances can be used -to facilitate traffic analysis. (For example, see -[the "Sniper Attack" paper](https://www.freehaven.net/anonbib/#sniper14) -by Jansen, Tschorsch, Johnson, and Scheuermann. - -With this in mind, any Tor implementation—especially one that -runs as a relay or onion service—must take steps to prevent -memory-based denial-of-service attacks. - -### Detecting low memory - -The easiest way to notice you're out of memory would, in theory, be -getting an error when you try to allocate more. Unfortunately, some -systems (e.g. Linux) won't actually give you an "out of memory" -error when you're low on memory. Instead, they overcommit and -promise you memory that they can't actually provide… and then later on, -they might kill processes that actually try to use more memory than -they wish they'd given out. - -So in practice, the mainline Tor implementation uses a different -strategy. It uses a self-imposed "MaxMemInQueues" value as an -upper bound for how much memory it's willing to allocate to certain -kinds of queued usages. This value can either be set by the user, -or derived from a fraction of the total amount of system RAM. - -As of Tor 0.4.7.x, the MaxMemInQueues mechanism tracks the following -kinds of allocation: - * Cells queued on circuits. - * Per-connection read or write buffers. - * On-the-fly compression or decompression state. - * Half-open stream records. - * Cached onion service descriptors (hsdir only). - * Cached DNS resolves (relay only). - * GEOIP-based usage activity statistics. - -Note that directory caches aren't counted, since those are stored on -disk and accessed via mmap. - -### Responding to low memory - -If our allocations exceed MaxMemInQueues, then we take the following -steps to reduce our memory allocation. - -*Freeing from caches*: For each of our onion service descriptor -cache, our DNS cache, and our GEOIP statistics cache, we check -whether they account for greater than 20% of our total allocation. -If they do, we free memory from the offending cache until the total -remaining is no more than 10% of our total allocation. - -When freeing entries from a cache, we aim to free (approximately) -the oldest entries first. - -*Freeing from buffers*: After freeing data from caches, we see -whether allocations are still above 90% of MaxMemInQueues. If they -are, we try to close circuits and connections until we are below 90% -of MaxMemInQueues. - -When deciding to what circuits to free, we sort them based on the -age of the oldest data in their queues, and free the ones with the -oldest data. (For example, a circuit on which a single cell has -been queued for 5 minutes would be freed before a circuit where 100 -cells have been queued for 5 seconds.) "Data queued on a circuit" -includes all data that we could drop if the circuit were destroyed: -not only the cells on the circuit's cell queue, but also any bytes -queued in buffers associated with streams or half-stream records -attached to the circuit. - -We free non-tunneled directory connections according to a similar -rule, according to the age of their oldest queued data. - -Upon freeing a circuit, a "DESTROY cell" must be sent in both -directions. - -### Reporting low memory. - -We define a "low threshold" equal to 3/4 of MaxMemInQueues. Every -time our memory usage is above the low threshold, we record -ourselves as being "under memory pressure". - -(This is not currently reported.) - - diff --git a/spec/README.md b/spec/README.md new file mode 100644 index 0000000..944f029 --- /dev/null +++ b/spec/README.md @@ -0,0 +1,69 @@ +# Tor specifications + +This repository holds the specifications that describe how Tor works. +They try to present Tor's protocols in sufficient detail to allow +the reader to implement a compatible implementation of Tor without ever +having to read the Tor source code. + +The [proposals](/proposals) directory holds our design proposals. These +include historical documents that have now been merged into the main +specifications. For more information on the proposal process, including an +explanation of how to make new proposals, see +[001-process.txt](/proposals/001-process.txt). + +## What you can find here + +Tor's specification is pretty big, and we've broken it into a bunch of +files. + +* General interest + * [tor-spec.txt](tor-spec.txt) + contains the specification for the core Tor protocol + itself; this is a good place to start reading. + * [cert-spec.txt](cert-spec.txt) describes a certificate format used + in the other parts of the protocol. + * [dir-spec.txt](dir-spec.txt) specifies the operations and formats used to + maintain a view of the network directory. + * [padding-spec.txt](padding-spec.txt) describes a set of padding mechanisms + used to impede traffic analysis. + * [version-spec.txt](version-spec.txt) explains how to parse Tor + version numbers. + * [glossary.txt](glossary.txt) is a glossary of terms used + in the other specifications. +* Client operations + * [address-spec.txt](address-spec.txt) lists a set of special + addresses that Tor handles differently from the regular DNS system. + * [guard-spec.txt](guard-spec.txt) explains the "guard node" algorithm + that Tor clients use to avoid sampling attacks. + * [path-spec.txt](path-spec.txt) explains how clients choose their paths + through the Tor network. + * [socks-extensions](socks-extensions.txt) specifies Tor-specific + extensions to the SOCKS protocol. +* Onion services + * [rend-spec-v2.txt](rend-spec-v2.txt) is the old, deprecated version + of the onion service protocol. + * [rend-spec-v3.txt](rend-spec-v3.txt) is the current version of the + onion service protocol. +* Censorship resistance + * [bridgedb-spec.txt](bridgedb-spec.txt) explains how the `bridgedb` + server gives out bridges to censored clients. + * [gettor-spec.txt](gettor-spec.txt) describes the `gettor` tool, + which is used to download Tor in censored areas. + * [pt-spec.txt](pt-spec.txt) describes the protocol that Tor clients + and relays use to communicate with pluggable transports used for + traffic obfuscation. +* Directory authorities + * [bandwidth-file-spec.txt](bandwidth-file-spec.txt) specifies the + file format used by bandwidth-measuring tools to report their + observations to directory authorities. + * [srv-spec.txt](srv-spec.txt) specifies the protocol that + directory authorities use to securely compute shared random values + for the network. +* Controller protocol + * [control-spec.txt](control-spec.txt) explains the protocol used by + controllers to communicate with a running Tor process. +* Miscellaneous + * [dir-list-spec.txt](dir-list-spec.txt) explains the format used by + tools like the fallback directory scripts to output a list of + Tor directories for inclusion in the Tor source code. + * The [attic](attic) directory has obsolete or historical documents. diff --git a/spec/dos-spec.md b/spec/dos-spec.md new file mode 100644 index 0000000..04470b2 --- /dev/null +++ b/spec/dos-spec.md @@ -0,0 +1,97 @@ +# Denial-of-service prevention mechanisms in Tor + +This document is incomplete; it describes some mechanisms that Tor +uses to avoid different kinds of denial-of-service attacks. + +## Handling low-memory conditions + +(See also `tor-spec.txt`, section 8.1.) + +The Tor protocol requires clients, onion services, relays, and +authorities to store various kind of information in buffers and +caches. But an attacker can use these buffers and queues to queues +to exhaust the memory of the a targeted Tor process, and force the +operating system to kill that process. + +Worse still, the ability to kill targeted Tor instances can be used +to facilitate traffic analysis. (For example, see +[the "Sniper Attack" paper](https://www.freehaven.net/anonbib/#sniper14) +by Jansen, Tschorsch, Johnson, and Scheuermann. + +With this in mind, any Tor implementation—especially one that +runs as a relay or onion service—must take steps to prevent +memory-based denial-of-service attacks. + +### Detecting low memory + +The easiest way to notice you're out of memory would, in theory, be +getting an error when you try to allocate more. Unfortunately, some +systems (e.g. Linux) won't actually give you an "out of memory" +error when you're low on memory. Instead, they overcommit and +promise you memory that they can't actually provide… and then later on, +they might kill processes that actually try to use more memory than +they wish they'd given out. + +So in practice, the mainline Tor implementation uses a different +strategy. It uses a self-imposed "MaxMemInQueues" value as an +upper bound for how much memory it's willing to allocate to certain +kinds of queued usages. This value can either be set by the user, +or derived from a fraction of the total amount of system RAM. + +As of Tor 0.4.7.x, the MaxMemInQueues mechanism tracks the following +kinds of allocation: + * Cells queued on circuits. + * Per-connection read or write buffers. + * On-the-fly compression or decompression state. + * Half-open stream records. + * Cached onion service descriptors (hsdir only). + * Cached DNS resolves (relay only). + * GEOIP-based usage activity statistics. + +Note that directory caches aren't counted, since those are stored on +disk and accessed via mmap. + +### Responding to low memory + +If our allocations exceed MaxMemInQueues, then we take the following +steps to reduce our memory allocation. + +*Freeing from caches*: For each of our onion service descriptor +cache, our DNS cache, and our GEOIP statistics cache, we check +whether they account for greater than 20% of our total allocation. +If they do, we free memory from the offending cache until the total +remaining is no more than 10% of our total allocation. + +When freeing entries from a cache, we aim to free (approximately) +the oldest entries first. + +*Freeing from buffers*: After freeing data from caches, we see +whether allocations are still above 90% of MaxMemInQueues. If they +are, we try to close circuits and connections until we are below 90% +of MaxMemInQueues. + +When deciding to what circuits to free, we sort them based on the +age of the oldest data in their queues, and free the ones with the +oldest data. (For example, a circuit on which a single cell has +been queued for 5 minutes would be freed before a circuit where 100 +cells have been queued for 5 seconds.) "Data queued on a circuit" +includes all data that we could drop if the circuit were destroyed: +not only the cells on the circuit's cell queue, but also any bytes +queued in buffers associated with streams or half-stream records +attached to the circuit. + +We free non-tunneled directory connections according to a similar +rule, according to the age of their oldest queued data. + +Upon freeing a circuit, a "DESTROY cell" must be sent in both +directions. + +### Reporting low memory. + +We define a "low threshold" equal to 3/4 of MaxMemInQueues. Every +time our memory usage is above the low threshold, we record +ourselves as being "under memory pressure". + +(This is not currently reported.) + + -- cgit v1.2.3-54-g00ecf