diff options
| -rw-r--r-- | control-spec.txt | 13 | ||||
| -rw-r--r-- | dir-spec.txt | 40 | ||||
| -rw-r--r-- | glossary.txt | 121 | ||||
| -rw-r--r-- | padding-spec.txt | 291 | ||||
| -rw-r--r-- | proposals/140-consensus-diffs.txt | 50 | ||||
| -rw-r--r-- | proposals/254-padding-negotiation.txt | 36 | ||||
| -rw-r--r-- | proposals/274-rotate-onion-keys-less.txt | 2 | ||||
| -rw-r--r-- | proposals/278-directory-compression-scheme-negotiation.txt | 33 | ||||
| -rw-r--r-- | tor-spec.txt | 25 |
9 files changed, 535 insertions, 76 deletions
diff --git a/control-spec.txt b/control-spec.txt index 52a58ae..808e161 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -2945,10 +2945,10 @@ "FAILED" / "CREATED" HSAddress = 16*Base32Character / "UNKNOWN" AuthType = "NO_AUTH" / "BASIC_AUTH" / "STEALTH_AUTH" / "UNKNOWN" - HsDir = LongName / Fingerprint + HsDir = LongName / Fingerprint / "UNKNOWN" DescriptorID = 32*Base32Character Reason = "BAD_DESC" / "QUERY_REJECTED" / "UPLOAD_REJECTED" / "NOT_FOUND" / - "UNEXPECTED" + "UNEXPECTED" / "QUERY_NO_HSDIR" Replica = 1*DIGIT These events will be triggered when required HiddenService descriptor is @@ -2973,6 +2973,10 @@ - "UPLOAD_REJECTED" - descriptor was rejected by HS directory. - "NOT_FOUND" - HS descriptor with given identifier was not found. - "UNEXPECTED" - nature of failure is unknown. + - "QUERY_NO_HSDIR" - No suitable HSDir were found for the query. + + For "QUERY_NO_HSDIR", the HsDir will be set to "UNKNOWN" which was + introduced in tor 0.3.1.0-alpha. If Action is "CREATED", Tor SHOULD send Replica field as well. The Replica field contains the replica number of the generated descriptor. The Replica @@ -2988,7 +2992,7 @@ HSAddress = 16*Base32Character / "UNKNOWN" DescId = 32*Base32Character - HsDir = LongName + HsDir = LongName / "UNKNOWN" Descriptor = The text of the descriptor formatted as specified in rend-spec.txt section 1.3 or empty string on failure. @@ -3000,6 +3004,9 @@ to "UNKNOWN". The HS_DESC event should be used to get more information on the failed request. + If the fetch fails for the QUERY_NO_HSDIR reason from the HS_DESC event, the + HsDir is set to "UNKNOWN". This was introduced in 0.3.1.0-alpha. + It's expected to receive a reply relatively fast as in it's the time it takes to fetch something over the Tor network. This can be between a couple of seconds up to 60 seconds (not a hard limit). But, in any cases, diff --git a/dir-spec.txt b/dir-spec.txt index 2dcf196..743df55 100644 --- a/dir-spec.txt +++ b/dir-spec.txt @@ -1158,6 +1158,46 @@ Pluggable transports are only relevant to bridges, but these entries can appear in non-bridge relays as well. + "padding-counts" YYYY-MM-DD HH:MM:SS (NSEC s) key=val key=val ... NL + [At most once.] + + YYYY-MM-DD HH:MM:SS defines the end of the included measurement + interval of length NSEC seconds (86400 seconds by default). Counts + are reset to 0 at the end of this interval. + + The keyword list is currently as follows: + + bin-size + - The current rounding value for cell count fields (10000 by + default) + write-drop + - The number of RELAY_DROP cells this relay sent + write-pad + - The number of CELL_PADDING cells this relay sent + write-total + - The total number of cells this relay cent + read-drop + - The number of RELAY_DROP cells this relay received + read-pad + - The number of CELL_PADDING cells this relay received + read-total + - The total number of cells this relay received + enabled-read-pad + - The number of CELL_PADDING cells this relay received on + connections that support padding + enabled-read-total + - The total number of cells this relay received on connections + that support padding + enabled-write-pad + - The total number of cells this relay received on connections + that support padding + enabled-write-total + - The total number of cells sent by this relay on connections + that support padding + max-chanpad-timers + - The maximum number of timers that this relay scheduled for + padding in the previous NSEC interval + "router-sig-ed25519" [As in router descriptors] diff --git a/glossary.txt b/glossary.txt index ba4baeb..e1a0f0a 100644 --- a/glossary.txt +++ b/glossary.txt @@ -3,11 +3,14 @@ The Tor Project -Note: This document aims to specify terms, notations or phrases related -to Tor and The Tor Project. +This document aims to specify terms, notations, and phrases related +to Tor, as used in the Tor specification documents and other documentation. This glossary is not a design document; it is only a reference. +This glossary is a work-in-progress; double-check its definitions before +citing them authoritatively. ;) + 0. Preliminaries The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL @@ -15,24 +18,81 @@ This glossary is not a design document; it is only a reference. "OPTIONAL" in this document are to be interpreted as described in RFC 2119. -0.1 Commonly used Tor configuration terms +1.0 Commonly used Tor configuration terms ORPort - Onion Router Port DirPort - Directory Port -0.2 Tor network components +2.0 Tor network components + + 2.1 Relays, aka OR (onion router) + + [Style guide: prefer the term "Relay"] + + 2.1.1 Specific roles + + Exit relay: The final hop in an exit circuit before traffic leaves + the Tor network to connect to external servers. + + Non-exit relay: Relays that send and receive traffic only to + other Tor relays and Tor clients. + + Entry relay: The first hop in a Tor circuit. Can be either a guard + relay or a bridge, depending on the client's configuration. + + Guard relay: A relay that a client uses as its entry for a longer + period of time. Guard relays are rotated more slowly to prevent + enumeration attacks. + + Bridge: A relay intentionally not listed in the public Tor + consensus, with the purpose of circumventing entities (such as + governments or ISPs) seeking to block clients from using Tor. + Currently, bridges are used only as entry relays. + + Directory cache: A relay that downloads cached directory information + from the directory authorities and serves it to clients on demand. + Any relay will act as a directory cache, if its bandwidth is high enough. + + Rendezvous point: A relay connecting a client to a hidden service. + Each party will builds a three-hop circuit, meeting at the + rendezvous point. + + 2.2 Client, aka OP (onion proxy) - Relay, aka OR (onion router) - - Exit relay - Non-exit relay - Guard relay - Client, aka OP (onion proxy) + [Style: the "OP" and "onion proxy" terms are deprecated.] - Bridge - + 2.3 Authorities: - Circuit: An established path through the network, where cryptographic keys - are negotiated using the ntor protocol with each hop. Circuits can differ - in length depending on their purpose. See also Leaky Pipe Topology. + Directory Authority: Nine total in the Tor network, operated by + trusted individuals. Directory authorities define and serve the + consensus document, defining the "state of the network." This document + contains a "router status" section for every relays currently + in the network. Directory authorities also serve router descriptors, + extra info documents, microdescriptors, and the microdescriptor consensus, + + Bridge Authority: One total. Similar in responsibility to directory + authorities, but for bridges. + + Fallback directory mirror: One of a list of directory caches distributed + with the Tor software. (When a client first connects to the network, and + has no directory information, it asks a fallback directory. From then on, + the client can ask any directory cache that's listed in the directory + information it has.) + + 2.4 Hidden Service: + + A hidden service is a server that will only accept incoming + connections via the hidden service protocol. Connection + initiators will not be able to learn the IP address of the hidden + service, allowing the hidden service to receive incoming connections, + serve content, etc, while preserving its location anonymity. + + 2.5 Circuit: + + An established path through the network, where cryptographic keys + are negotiated using the ntor protocol or TAP (Tor Authentication + Protocol (deprecated) with each hop. Circuits can differ in length + depending on their purpose. See also Leaky Pipe Topology. Origin Circuit - @@ -44,15 +104,38 @@ This glossary is not a design document; it is only a reference. network. For example, a client could connect to a hidden service via an internal circuit. - Stream - Edge connection: + 2.6 Edge connection: - TLS connection: + 2.7 Consensus: The state of the Tor network, published every hour, + decided by a vote from the network's directory authorities. Clients + fetch the consensus from directory authorities, fallback + directories, or directory caches. + 2.8 Descriptor: Each descriptor represents information about one + relay in the Tor network. The descriptor includes the relay's IP + address, public key fingerprint, along with other data. Relays send + descriptors to directory authorities, who will vote and publish a + summary of them in the network consensus. + +3.0 Tor network protocols Link handshake Circuit handshake + Hidden Service Protocol + Directory Protocol + + +4.0 General network definitions + + Leaky Pipe Topology: The ability for the origin of a circuit to address + relay cells to be addressed to any hop in the path of a circuit. In Tor, + the destination hop is determined by using the 'recognized' field of relay + cells. + + Stream: A single application-level connection or request, multiplexed over + a Tor circuit. A 'Stream' can currently carry the contents of a TCP + connection, a DNS request, or a Tor directory request. - Leaky Pipe Topology: The ability for packets to be addressed to any hop - in the path of a circuit. The destination hop is determined by using the - recognized field of relay cells. + Channel: A pairwise connection between two Tor relays, or between a + client and a relay. Circuits are multiplexed over Channels. All + channels are currently implemented as TLS connections. diff --git a/padding-spec.txt b/padding-spec.txt new file mode 100644 index 0000000..3f9b194 --- /dev/null +++ b/padding-spec.txt @@ -0,0 +1,291 @@ + Tor Padding Specification + + Mike Perry + +Note: This is an attempt to specify Tor as currently implemented. Future +versions of Tor will implement improved algorithms. + +This document tries to cover how Tor chooses to use cover traffic to obscure +various traffic patterns from external and internal observers. Other +implementations MAY take other approaches, but implementors should be aware of +the anonymity and load-balancing implications of their choices. + + THIS SPEC ISN'T DONE YET. + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL + NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + RFC 2119. + + +1. Overview + + Tor supports two classes of cover traffic: connection-level padding, and + circuit-level padding. + + Connection-level padding uses the CELL_PADDING cell command for cover + traffic, where as circuit-level padding uses the RELAY_COMMAND_DROP relay + command. CELL_PADDING is single-hop only and can be differentiated from + normal traffic by Tor relays ("internal" observers), but not by entities + monitoring Tor OR connections ("external" observers). + + RELAY_COMMAND_DROP is multi-hop, and is not visible to intermediate Tor + relays, because the relay command field is covered by circuit layer + encryption. Moreover, Tor's 'recognized' field allows RELAY_COMMAND_DROP + padding to be sent to any intermediate node in a circuit (as per Section + 6.1 of tor-spec.txt). + + Currently, only single-hop CELL_PADDING is used by Tor. It is described in + Section 2. At a later date, further sections will be added to this document + to describe various uses of multi-hop circuit-level padding. + + +2. Connection-level padding + +2.1. Background + + Tor clients and relays make use of CELL_PADDING to reduce the resolution of + connection-level metadata retention by ISPs and surveillance infrastructure. + + Such metadata retention is implemented by Internet routers in the form of + Netflow, jFlow, Netstream, or IPFIX records. These records are emitted by + gateway routers in a raw form and then exported (often over plaintext) to a + "collector" that either records them verbatim, or reduces their granularity + further[1]. + + Netflow records and the associated data collection and retention tools are + very configurable, and have many modes of operation, especially when + configured to handle high throughput. However, at ISP scale, per-flow records + are very likely to be employed, since they are the default, and also provide + very high resolution in terms of endpoint activity, second only to full packet + and/or header capture. + + Per-flow records record the endpoint connection 5-tuple, as well as the + total number of bytes sent and received by that 5-tuple during a particular + time period. They can store additional fields as well, but it is primarily + timing and bytecount information that concern us. + + When configured to provide per-flow data, routers emit these raw flow + records periodically for all active connections passing through them + based on two parameters: the "active flow timeout" and the "inactive + flow timeout". + + The "active flow timeout" causes the router to emit a new record + periodically for every active TCP session that continuously sends data. The + default active flow timeout for most routers is 30 minutes, meaning that a + new record is created for every TCP session at least every 30 minutes, no + matter what. This value can be configured from 1 minute to 60 minutes on + major routers. + + The "inactive flow timeout" is used by routers to create a new record if a + TCP session is inactive for some number of seconds. It allows routers to + avoid the need to track a large number of idle connections in memory, and + instead emit a separate record only when there is activity. This value + ranges from 10 seconds to 600 seconds on common routers. It appears as + though no routers support a value lower than 10 seconds. + + For reference, here are default values and ranges (in parenthesis when + known) for common routers, along with citations to their manuals. + + Some routers speak other collection protocols than Netflow, and in the + case of Juniper, use different timeouts for these protocols. Where this + is known to happen, it has been noted. + + Inactive Timeout Active Timeout + Cisco IOS[3] 15s (10-600s) 30min (1-60min) + Cisco Catalyst[4] 5min 32min + Juniper (jFlow)[5] 15s (10-600s) 30min (1-60min) + Juniper (Netflow)[6,7] 60s (10-600s) 30min (1-30min) + H3C (Netstream)[8] 60s (60-600s) 30min (1-60min) + Fortinet[9] 15s 30min + MicroTik[10] 15s 30min + nProbe[14] 30s 120s + Alcatel-Lucent[2] 15s (10-600s) 30min (1-600min) + + The combination of the active and inactive netflow record timeouts allow us + to devise a low-cost padding defense that causes what would otherwise be + split records to "collapse" at the router even before they are exported to + the collector for storage. So long as a connection transmits data before the + "inactive flow timeout" expires, then the router will continue to count the + total bytes on that flow before finally emitting a record at the "active + flow timeout". + + This means that for a minimal amount of padding that prevents the "inactive + flow timeout" from expiring, it is possible to reduce the resolution of raw + per-flow netflow data to the total amount of bytes send and received in a 30 + minute window. This is a vast reduction in resolution for HTTP, IRC, XMPP, + SSH, and other intermittent interactive traffic, especially when all + user traffic in that time period is multiplexed over a single connection + (as it is with Tor). + +2.2. Implementation + + Tor clients currently maintain one TLS connection to their Guard node to + carry actual application traffic, and make up to 3 additional connections to + other nodes to retrieve directory information. + + We pad only the client's connection to the Guard node, and not any other + connection. We treat Bridge node connections to the Tor network as client + connections, and pad them, but otherwise not pad between normal relays. + + Both clients and Guards will maintain a timer for all application (ie: + non-directory) TLS connections. Every time a non-padding packet is sent or + received by either end, that endpoint will sample a timeout value from + between 1.5 seconds and 9.5 seconds using the max(X,X) distribution + described in Section 2.3. The time range is subject to consensus + parameters as specified in Section 2.6. + + If the connection becomes active for any reason before this timer + expires, the timer is reset to a new random value between 1.5 and 9.5 + seconds. If the connection remains inactive until the timer expires, a + single CELL_PADDING cell will be sent on that connection. + + In this way, the connection will only be padded in the event that it is + idle, and will always transmit a packet before the minimum 10 second inactive + timeout. + +2.3. Padding Cell Timeout Distribution Statistics + + It turns out that because the padding is bidirectional, and because both + endpoints are maintaining timers, this creates the situation where the time + before sending a padding packet in either direction is actually + min(client_timeout, server_timeout). + + If client_timeout and server_timeout are uniformly sampled, then the + distribution of min(client_timeout,server_timeout) is no longer uniform, and + the resulting average timeout (Exp[min(X,X)]) is much lower than the + midpoint of the timeout range. + + To compensate for this, instead of sampling each endpoint timeout uniformly, + we instead sample it from max(X,X), where X is uniformly distributed. + + If X is a random variable uniform from 0..R-1 (where R=high-low), then the + random variable Y = max(X,X) has Prob(Y == i) = (2.0*i + 1)/(R*R). + + Then, when both sides apply timeouts sampled from Y, the resulting + bidirectional padding packet rate is now a third random variable: + Z = min(Y,Y). + + The distribution of Z is slightly bell-shaped, but mostly flat around the + mean. It also turns out that Exp[Z] ~= Exp[X]. Here's a table of average + values for each random variable: + + R Exp[X] Exp[Z] Exp[min(X,X)] Exp[Y=max(X,X)] + 2000 999.5 1066 666.2 1332.8 + 3000 1499.5 1599.5 999.5 1999.5 + 5000 2499.5 2666 1666.2 3332.8 + 6000 2999.5 3199.5 1999.5 3999.5 + 7000 3499.5 3732.8 2332.8 4666.2 + 8000 3999.5 4266.2 2666.2 5332.8 + 10000 4999.5 5328 3332.8 6666.2 + 15000 7499.5 7995 4999.5 9999.5 + 20000 9900.5 10661 6666.2 13332.8 + + In this way, we maintain the property that the midpoint of the timeout range + is the expected mean time before a padding packet is sent in either + direction. + +2.4. Maximum overhead bounds + + With the default parameters and the above distribution, we expect a + padded connection to send one padding cell every 5.5 seconds. This + averages to 103 bytes per second full duplex (~52 bytes/sec in each + direction), assuming a 512 byte cell and 55 bytes of TLS+TCP+IP headers. + For a client connection that remains otherwise idle for its expected + ~50 minute lifespan (governed by the circuit available timeout plus a + small additional connection timeout), this is about 154.5KB of overhead + in each direction (309KB total). + + With 2.5M completely idle clients connected simultaneously, 52 bytes per + second amounts to 130MB/second in each direction network-wide, which is + roughly the current amount of Tor directory traffic[11]. Of course, our + 2.5M daily users will neither be connected simultaneously, nor entirely + idle, so we expect the actual overhead to be much lower than this. + +2.5. Reducing or Disabling Padding via Negotiation + + To allow mobile clients to either disable or reduce their padding overhead, + the CELL_PADDING_NEGOTIATE cell (tor-spec.txt section 7.2) may be sent from + clients to relays. This cell is used to instruct relays to cease sending + padding. + + If the client has opted to use reduced padding, it continues to send + padding cells sampled from the range [9000,14000] milliseconds (subject to + consensus parameter alteration as per Section 2.6), still using the + Y=max(X,X) distribution. Since the padding is now unidirectional, the + expected frequency of padding cells is now governed by the Y distribution + above as opposed to Z. For a range of 5000ms, we can see that we expect to + send a padding packet every 9000+3332.8 = 12332.8ms. We also half the + circuit available timeout from ~50min down to ~25min, which causes the + client's OR connections to be closed shortly there after when it is idle, + thus reducing overhead. + + These two changes cause the padding overhead to go from 309KB per one-time-use + Tor connection down to 69KB per one-time-use Tor connection. For continual + usage, the maximum overhead goes from 103 bytes/sec down to 46 bytes/sec. + + If a client opts to completely disable padding, it sends a + CELL_PADDING_NEGOTIATE to instruct the relay not to pad, and then does not + send any further padding itself. + +2.6. Consensus Parameters Governing Behavior + + Connection-level padding is controlled by the following consensus parameters: + + * nf_ito_low + - The low end of the range to send padding when inactive, in ms. + - Default: 1500 + + * nf_ito_high + - The high end of the range to send padding, in ms. + - Default: 9500 + - If nf_ito_low == nf_ito_high == 0, padding will be disabled. + + * nf_ito_low_reduced + - For reduced padding clients: the low end of the range to send padding + when inactive, in ms. + - Default: 9000 + + * nf_ito_high_reduced + - For reduced padding clients: the high end of the range to send padding, + in ms. + - Default: 14000 + + * nf_conntimeout_clients + - The number of seconds to keep circuits opened and available for + clients to use. Note that the actual client timeout is randomized + uniformly from this value to twice this value. This governs client + OR conn lifespan. Reduced padding clients use half the consensus + value. + - Default: 1800 + + * nf_pad_before_usage + - If set to 1, OR connections are padded before the client uses them + for any application traffic. If 0, OR connections are not padded + until application data begins. + - Default: 1 + + * nf_pad_relays + - If set to 1, we also pad inactive relay-to-relay connections + - Default: 0 + + * nf_conntimeout_relays + - The number of seconds that idle relay-to-relay connections are kept + open. + - Default: 3600 + + +1. https://en.wikipedia.org/wiki/NetFlow +2. http://infodoc.alcatel-lucent.com/html/0_add-h-f/93-0073-10-01/7750_SR_OS_Router_Configuration_Guide/Cflowd-CLI.html +3. http://www.cisco.com/en/US/docs/ios/12_3t/netflow/command/reference/nfl_a1gt_ps5207_TSD_Products_Command_Reference_Chapter.html#wp1185203 +4. http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/70974-netflow-catalyst6500.html#opconf +5. https://www.juniper.net/techpubs/software/erx/junose60/swconfig-routing-vol1/html/ip-jflow-stats-config4.html#560916 +6. http://www.jnpr.net/techpubs/en_US/junos15.1/topics/reference/configuration-statement/flow-active-timeout-edit-forwarding-options-po.html +7. http://www.jnpr.net/techpubs/en_US/junos15.1/topics/reference/configuration-statement/flow-active-timeout-edit-forwarding-options-po.html +8. http://www.h3c.com/portal/Technical_Support___Documents/Technical_Documents/Switches/H3C_S9500_Series_Switches/Command/Command/H3C_S9500_CM-Release1648%5Bv1.24%5D-System_Volume/200901/624854_1285_0.htm#_Toc217704193 +9. http://docs-legacy.fortinet.com/fgt/handbook/cli52_html/FortiOS%205.2%20CLI/config_system.23.046.html +10. http://wiki.mikrotik.com/wiki/Manual:IP/Traffic_Flow +11. https://metrics.torproject.org/dirbytes.html +12. http://freehaven.net/anonbib/cache/murdoch-pet2007.pdf +13. https://gitweb.torproject.org/torspec.git/tree/proposals/188-bridge-guards.txt +14. http://www.ntop.org/wp-content/uploads/2013/03/nProbe_UserGuide.pdf diff --git a/proposals/140-consensus-diffs.txt b/proposals/140-consensus-diffs.txt index 8a420df..5a8be3f 100644 --- a/proposals/140-consensus-diffs.txt +++ b/proposals/140-consensus-diffs.txt @@ -16,6 +16,13 @@ Status: Accepted diffs, add parameters, restore diff-only URLs, say what "Digest" means. -nickm + 3-May-2017: Add a notion of "digest-as-signed" vs "full digest", since + otherwise the fact that there are multiple encodings of the same valid + consensus signatures would make clients identify which encodings they + had been given as they asked for diffs. + + 4-May-2017: Remove support for truncated digest prefixes. + 1. Overview. Tor clients and servers need a list of which relays are on the @@ -47,8 +54,18 @@ Status: Accepted 3.0. Preliminaries. - Unless otherwise specified, all hashes in this document are SHA3-256 - hashes, encoded in base64. + Unless otherwise specified, all digests in this document are SHA3-256 + digests, encoded in base64. This document also uses "hash" as + synonymous with "digest". + + A "full digest" of a consensus document covers the entire document, + from the "network-status-version" through the newline after the final + "-----END SIGNATURE-----". + + A "digest as signed" of a consensus document covers the same part that + the signatures cover: the "network-status-version" through the space + immediately after the "directory-signature" keyword on the first + "directory-signature" line. 3.1 Clients @@ -92,17 +109,13 @@ Status: Accepted HTTP/1.0 GET /tor/status-vote/current/consensus{-Flavor}/<FPRLIST>.z X-Or-Diff-From-Consensus: HASH1 HASH2... - where the hashes are the full digests of the consensuses the client + where the hashes are the digests-as-signed of the consensuses the client currently has, and FPRLIST is a list of (abbreviated) fingerprints of authorities the client trusts. Servers will only return a consensus if more than half of the requested - authorities have signed the document, otherwise a 404 error will be sent - back. The fingerprints can be shortened to a length of any multiple of - two, using only the leftmost part of the encoded fingerprint. Tor uses - 3 bytes (6 hex characters) of the fingerprint. (This is just like the - conditional consensus downloads that Tor supports starting with - 0.1.2.1-alpha.) + authorities have signed the document. Otherwise, a 404 error will be sent + back. The advantage of using the same URL that is currently used for consensuses is that the client doesn't need to know whether a server @@ -121,8 +134,8 @@ Status: Accepted I currently lean towards the empty diff.] - Additionally, specific diff for a given consensus hash should be available - a URL of the form: + Additionally, specific diff for a given consensus digest-as-signed + should be available a URL of the form: /tor/status-vote/current/consensus{-Flavor}/diff/<HASH>/<FPRLIST>.z @@ -139,10 +152,10 @@ Status: Accepted to be a full consensus download and would therefore currently start with "network-status-version 3". - Following the network-status-diff line is another header line, starting with - the token "hash" followed by the full digest of the consensus that this diff - applies to and the full digest that the resulting consensus - should have. + Following the network-status-diff line is another header line, + starting with the token "hash" followed by the digest-as-signed of the + consensus that this diff applies to, and the full digest that the + resulting consensus should have. Following the network-status-diff header lines is a diff, or patch, in limited ed format. We choose this format because it is easy to create @@ -158,6 +171,7 @@ Status: Accepted We support the following ed commands, each on a line by itself: - "<n1>d" Delete line n1 - "<n1>,<n2>d" Delete lines n1 through n2, inclusive + - "<n1>,$d" Delete line n1 through the end of the file, inclusive. - "<n1>c" Replace line n1 with the following block - "<n1>,<n2>c" Replace lines n1 through n2, inclusive, with the following block. @@ -172,6 +186,12 @@ Status: Accepted lines are only ever referred to by their position in the original file. + If there are any directory signatures on the original document, the + first command MUST be a "<n1>,$d" form to remove all of the directory + signatures. Using this format ensures that the client will + successfully apply the diff even if they have an unusual encoding for + the signatures. + The "current line" is either the first line of the file, if this is the first command, the last line of a block we added in an append or change command, or the line immediate following a set of lines we just diff --git a/proposals/254-padding-negotiation.txt b/proposals/254-padding-negotiation.txt index e4da004..c3d9ecd 100644 --- a/proposals/254-padding-negotiation.txt +++ b/proposals/254-padding-negotiation.txt @@ -41,31 +41,27 @@ Proposal 251[1]). In that scenario, the primary negotiation mechanism we need is a way for mobile clients to tell their Guards to stop padding, or to pad less -often. The following Trunnel payloads should cover the needed +often. The following Trunnel payload should cover the needed parameters: - const CELL_PADDING_COMMAND_STOP = 1; - const CELL_PADDING_COMMAND_START = 2; + const CHANNELPADDING_COMMAND_STOP = 1; + const CHANNELPADDING_COMMAND_START = 2; - /* This command tells the relay to stop sending any periodic - CELL_PADDING cells. */ - struct cell_padding_stop { - u8 command IN [CELL_PADDING_COMMAND_STOP]; - }; - - /* This command tells the relay to alter its min and max netflow - timeout range values, and send padding at that rate (resuming - if stopped). */ - struct cell_padding_start { - u8 command IN [CELL_PADDING_COMMAND_START]; + /* The start command tells the relay to alter its min and max netflow + timeout range values, and send padding at that rate (resuming + if stopped). The stop command tells the relay to stop sending + link-level padding. */ + struct channelpadding_negotiate { + u8 version IN [0]; + u8 command IN [CHANNELPADDING_COMMAND_START, CHANNELPADDING_COMMAND_STOP]; - /* Min must not be lower than the current consensus parameter - nf_ito_low. */ - u16 ito_low_ms; + /* Min must not be lower than the current consensus parameter + nf_ito_low. Ignored if command is stop. */ + u16 ito_low_ms; - /* Max must not be lower than ito_low_ms */ - u16 ito_high_ms; - }; + /* Max must not be lower than ito_low_ms. Ignored if command is stop. */ + u16 ito_high_ms; + }; More complicated forms of link-level padding can still be specified using the primitives in Section 3, by using "leaky pipe" topology to diff --git a/proposals/274-rotate-onion-keys-less.txt b/proposals/274-rotate-onion-keys-less.txt index 4099759..7c17873 100644 --- a/proposals/274-rotate-onion-keys-less.txt +++ b/proposals/274-rotate-onion-keys-less.txt @@ -41,7 +41,7 @@ Implemented-In: 0.3.1.1-alpha The number of microdescriptors to actually download should decrease by a similar number. - This amount to a significant reduction: currently, by + This amounts to a significant reduction: currently, by back-of-the-envelope estimates, an always-on client that downloads all the directory info in a month downloads about 449MB of compressed consensuses and something around 97 MB of compressed diff --git a/proposals/278-directory-compression-scheme-negotiation.txt b/proposals/278-directory-compression-scheme-negotiation.txt index b62c69a..475e4af 100644 --- a/proposals/278-directory-compression-scheme-negotiation.txt +++ b/proposals/278-directory-compression-scheme-negotiation.txt @@ -13,7 +13,7 @@ Target: N/A using the semantics of the HTTP protocol. Furthermore this proposal also extends Tor's directory protocol with - support for the LZMA2 and Zstandard compression schemes. + support for the LZMA and Zstandard compression schemes. 1. Motivation @@ -42,26 +42,26 @@ Target: N/A most commonly available compression schemes used throughout the network. - - We add support for the LZMA2 compression scheme, which yields + - We add support for the LZMA compression scheme, which yields better compressed size and decompression time at the expensive of higher compression time and higher memory usage. - We add support for the Zstandard compression scheme, which yields - better compression ratio than GZip, but slightly worse than LZMA2, - but with a smaller CPU and memory footprint than LZMA2. + better compression ratio than GZip, but slightly worse than LZMA, + but with a smaller CPU and memory footprint than LZMA. 2. Analysis We investigated the compression ratio, memory usage, memory allocation strategies, and execution time for compression and decompression of - the GZip, BZip2, LZMA2, and Zstandard compression schemes at + the GZip, BZip2, LZMA, and Zstandard compression schemes at compression levels 1 through 9. The data used in this analysis can be found in [1] and the `bench` tool for generating the data can be found in [2]. During the preparation for this proposal Nick have analysed - compressing consensus diffs using both GZip, LZMA2, and Zstandard. The + compressing consensus diffs using both GZip, LZMA, and Zstandard. The result of Nick's analysis can be found in [3]. We must continue to support both "gzip", "deflate", and "identity" @@ -85,7 +85,7 @@ Target: N/A For example: GET / HTTP/1.0 - Accept-Encoding: x-zstd, x-lzma2, gzip, deflate + Accept-Encoding: x-zstd, x-tor-lzma, gzip, deflate When a directory server receives a request with the "Accept-Encoding" header included, to either the ".z" compressed or the uncompressed @@ -106,13 +106,20 @@ Target: N/A Currently supported compression scheme names includes "identity", "gzip", and "deflate". This proposal adds two additional compression - scheme named "x-lzma2" (LZMA2) and "x-zstd" (Zstandard). + scheme named "x-tor-lzma" (LZMA) and "x-zstd" (Zstandard). All compression scheme names are case-insensitive. The "deflate", "gzip", and "identity" compression schemes must be supported by directory servers for backwards compatibility. + We use the name "x-tor-lzma" instead of just "x-lzma" because we + require a defined upper bound of memory usage that is available for + decompression of LZMA compressed data. The upper bound for memory + available for LZMA decompression is defined as 16 MB. This currently + means that will not use the LZMA compression scheme with a "preset" + value higher than 6. + Additionally, when a client, that supports this proposals, makes a request to a directory document with the ".z"-suffix it must send an ordered set of supported compression schemes where the last elements @@ -144,7 +151,7 @@ Target: N/A The detection of compression and decompression bombs are handled in `is_compression_bomb()` in torgzip.c and the same functionality is used both for compression and decompression. These functions must be - extended to support LZMA2 and Zstandard. + extended to support LZMA and Zstandard. 4.2 Detection of Compression Algorithms @@ -152,7 +159,7 @@ Target: N/A decompression handler, when we have received data from another peer, Tor tries to detect the compression scheme in `detect_compression_method()`` in torgzip.c. This function should be - extended to also detect the LZMA2 and Zstandard formats. Possible + extended to also detect the LZMA and Zstandard formats. Possible methods of applying this detection is looking at xz-tools, zstd's CLI, and the libmagic 'compress' module. @@ -175,15 +182,15 @@ Target: N/A Each newly added compression scheme adds to the compression cache of a relay, which increases the memory requirements of a relay. - The LZMA2 compression scheme yields better compression ratio at the + The LZMA compression scheme yields better compression ratio at the expense of higher memory and CPU requirements for compression and slightly higher memory and CPU requirements for decompression. The Zstandard compression scheme yields better compression ratio than GZip does, but does not suffer from the same high CPU and memory - requirements for compression as LZMA2 does. + requirements for compression as LZMA does. - Because of the high requirements for CPU and memory usage for LZMA2 it + Because of the high requirements for CPU and memory usage for LZMA it is possible that we do not support this scheme for all available documents or that we only support it in situations where it is possible to pre-compute and cache the compressed document. diff --git a/tor-spec.txt b/tor-spec.txt index 7ccc90f..927173b 100644 --- a/tor-spec.txt +++ b/tor-spec.txt @@ -428,6 +428,7 @@ see tor-design.pdf. 9 -- RELAY_EARLY (End-to-end data; limited)(See Sec 5.6) 10 -- CREATE2 (Extended CREATE cell) (See Sec 5.1) 11 -- CREATED2 (Extended CREATED cell) (See Sec 5.1) + 12 -- PADDING_NEGOTIATE (Padding negotiation) (See Sec 7.2) Variable-length command values are: 7 -- VERSIONS (Negotiate proto version) (See Sec 4) @@ -539,6 +540,7 @@ see tor-design.pdf. variable-length cells. 3 -- Uses the in-protocol handshake. 4 -- Increases circuit ID width to 4 bytes. + 5 -- Adds support for link padding and negotiation (padding-spec.txt). 4.2. CERTS cells @@ -1535,11 +1537,24 @@ see tor-design.pdf. long-range padding. The contents of a PADDING, VPADDING, or DROP cell SHOULD be chosen randomly, and MUST be ignored. - Currently nodes are not required to do any sort of link padding or - dummy traffic. Because strong attacks exist even with link padding, - and because link padding greatly increases the bandwidth requirements - for running a node, we plan to leave out link padding until this - tradeoff is better understood. + If the link protocol is version 5 or higher, link level padding is + enabled as per padding-spec.txt. On these connections, clients may + negotiate the use of padding with a CELL_PADDING_NEGOTIATE command + whose format is as follows: + Version [1 byte] + Command [1 byte] + ito_low_ms [2 bytes] + ito_high_ms [2 bytes] + + Currently, only version 0 of this cell is defined. In it, the command + field is either 1 (stop padding) or 2 (start padding). For the start + padding command, a pair of timeout values specifying a low and a high + range bounds for randomized padding timeouts may be specified as unsigned + integer values in milliseconds. The ito_low_ms field must not be lower than + the current consensus parameter value for nf_ito_low (default: 1500). + ito_high_ms must be greater than ito_low_ms. + + For more details on padding behavior, see padding-spec.txt. 7.3. Circuit-level flow control |