diff options
Diffstat (limited to 'spec/dir-spec')
27 files changed, 4451 insertions, 0 deletions
diff --git a/spec/dir-spec/accepting-relay-documents.md b/spec/dir-spec/accepting-relay-documents.md new file mode 100644 index 0000000..ffcf7c7 --- /dev/null +++ b/spec/dir-spec/accepting-relay-documents.md @@ -0,0 +1,52 @@ +<a id="dir-spec.txt-3.2"></a> + +# Accepting server descriptor and extra-info document uploads + +When a router posts a signed descriptor to a directory authority, the +authority first checks whether it is well-formed and correctly +self-signed. If it is, the authority next verifies that the nickname +in question is not already assigned to a router with a different +public key. +Finally, the authority MAY check that the router is not blacklisted +because of its key, IP, or another reason. + +An authority also keeps a record of all the Ed25519/RSA1024 +identity key pairs that it has seen before. It rejects any +descriptor that has a known Ed/RSA identity key that it has +already seen accompanied by a different RSA/Ed identity key +in an older descriptor. + +At a future date, authorities will begin rejecting all +descriptors whose RSA key was previously accompanied by an +Ed25519 key, if the descriptor does not list an Ed25519 key. + +At a future date, authorities will begin rejecting all descriptors +that do not list an Ed25519 key. + +If the descriptor passes these tests, and the authority does not already +have a descriptor for a router with this public key, it accepts the +descriptor and remembers it. + +If the authority _does_ have a descriptor with the same public key, the +newly uploaded descriptor is remembered if its publication time is more +recent than the most recent old descriptor for that router, and either: + +```text + - There are non-cosmetic differences between the old descriptor and the + new one. + - Enough time has passed between the descriptors' publication times. + (Currently, 2 hours.) +``` + +Differences between server descriptors are "non-cosmetic" if they would be +sufficient to force an upload as described in section 2.1 above. + +Note that the "cosmetic difference" test only applies to uploaded +descriptors, not to descriptors that the authority downloads from other +authorities. + +When a router posts a signed extra-info document to a directory authority, +the authority again checks it for well-formedness and correct signature, +and checks that its matches the extra-info-digest in some router +descriptor that it believes is currently useful. If so, it accepts it and +stores it and serves it as requested. If not, it drops it. diff --git a/spec/dir-spec/assigning-flags-vote.md b/spec/dir-spec/assigning-flags-vote.md new file mode 100644 index 0000000..7eb3849 --- /dev/null +++ b/spec/dir-spec/assigning-flags-vote.md @@ -0,0 +1,172 @@ +<a id="dir-spec.txt-3.4.2"></a> + +# Assigning flags in a vote + +(This section describes how directory authorities choose which status +flags to apply to routers. Later directory authorities MAY do things +differently, so long as clients keep working well. Clients MUST NOT +depend on the exact behaviors in this section.) + +In the below definitions, a router is considered "active" if it is +running, valid, and not hibernating. + +When we speak of a router's bandwidth in this section, we mean either +its measured bandwidth, or its advertised bandwidth. If a sufficient +threshold (configurable with MinMeasuredBWsForAuthToIgnoreAdvertised, +500 by default) of routers have measured bandwidth values, then the +authority bases flags on _measured_ bandwidths, and treats nodes with +non-measured bandwidths as if their bandwidths were zero. Otherwise, +it uses measured bandwidths for nodes that have them, and advertised +bandwidths for other nodes. + +When computing thresholds based on percentiles of nodes, an authority +only considers nodes that are active, that have not been +omitted as a sybil (see below), and whose bandwidth is at least +4 KB. Nodes that don't meet these criteria do not influence any +threshold calculations (including calculation of stability and uptime +and bandwidth thresholds) and also do not have their Exit status +change. + +"Valid" -- a router is 'Valid' if it is running a version of Tor not +known to be broken, and the directory authority has not blacklisted +it as suspicious. + +```text + "Named" -- + "Unnamed" -- Directory authorities no longer assign these flags. + They were once used to determine whether a relay's nickname was + canonically linked to its public key. +``` + +"Running" -- A router is 'Running' if the authority managed to connect to +it successfully within the last 45 minutes on all its published ORPorts. +Authorities check reachability on: + +```text + * the IPv4 ORPort in the "r" line, and + * the IPv6 ORPort considered for the "a" line, if: + * the router advertises at least one IPv6 ORPort, and + * AuthDirHasIPv6Connectivity 1 is set on the authority. +``` + +A minority of voting authorities that set AuthDirHasIPv6Connectivity will +drop unreachable IPv6 ORPorts from the full consensus. Consensus method 27 +in 0.3.3.x puts IPv6 ORPorts in the microdesc consensus, so that +authorities can drop unreachable IPv6 ORPorts from all consensus flavors. +Consensus method 28 removes IPv6 ORPorts from microdescriptors. + +"Stable" -- A router is 'Stable' if it is active, and either its Weighted +MTBF is at least the median for known active routers or its Weighted MTBF +corresponds to at least 7 days. Routers are never called Stable if they are +running a version of Tor known to drop circuits stupidly. (0.1.1.10-alpha +through 0.1.1.16-rc are stupid this way.) + +To calculate weighted MTBF, compute the weighted mean of the lengths +of all intervals when the router was observed to be up, weighting +intervals by $\\alpha^n$, where $n$ is the amount of time that has +passed since the interval ended, and $\\alpha$ is chosen so that +measurements over approximately one month old no longer influence the +weighted MTBF much. + +\[XXXX what happens when we have less than 4 days of MTBF info.\] + +"Exit" -- A router is called an 'Exit' iff it allows exits to at +least one /8 address space on each of ports 80 and 443. (Up until +Tor version 0.3.2, the flag was assigned if relays exit to at least +two of the ports 80, 443, and 6667.) + +"Fast" -- A router is 'Fast' if it is active, and its bandwidth is either in +the top 7/8ths for known active routers or at least 100KB/s. + +"Guard" -- A router is a possible Guard if all of the following apply: + +```text + - It is Fast, + - It is Stable, + - Its Weighted Fractional Uptime is at least the median for "familiar" + active routers, + - It is "familiar", + - Its bandwidth is at least AuthDirGuardBWGuarantee (if set, 2 MB by + default), OR its bandwidth is among the 25% fastest relays, + - It qualifies for the V2Dir flag as described below (this + constraint was added in 0.3.3.x, because in 0.3.0.x clients + started avoiding guards that didn't also have the V2Dir flag). +``` + +To calculate weighted fractional uptime, compute the fraction +of time that the router is up in any given day, weighting so that +downtime and uptime in the past counts less. + +A node is 'familiar' if 1/8 of all active nodes have appeared more +recently than it, OR it has been around for a few weeks. + +"Authority" -- A router is called an 'Authority' if the authority +generating the network-status document believes it is an authority. + +"V2Dir" -- A router supports the v2 directory protocol or higher if it has +an open directory port OR a tunnelled-dir-server line in its router +descriptor, and it is running a version of the directory +protocol that supports the functionality clients need. (Currently, every +supported version of Tor supports the functionality that clients need, +but some relays might set "DirCache 0" or set really low rate limiting, +making them unqualified to be a directory mirror, i.e. they will omit +the tunnelled-dir-server line from their descriptor.) + +"HSDir" -- A router is a v2 hidden service directory if it stores and +serves v2 hidden service descriptors, has the Stable and Fast flag, and the +authority believes that it's been up for at least 96 hours (or the current +value of MinUptimeHidServDirectoryV2). + +"MiddleOnly" -- An authority should vote for this flag if it believes +that a relay is unsuitable for use except as a middle relay. When +voting for this flag, the authority should also vote against "Exit", +"Guard", "HsDir", and "V2Dir". When voting for this flag, if the +authority votes on the "BadExit" flag, the authority should vote in +favor of "BadExit". (This flag was added in 0.4.7.2-alpha.) + +"NoEdConsensus" -- authorities should not vote on this flag; it is +produced as part of the consensus for consensus method 22 or later. + +"StaleDesc" -- authorities should vote to assign this flag if the +published time on the descriptor is over 18 hours in the past. (This flag +was added in 0.4.0.1-alpha.) + +"Sybil" -- authorities SHOULD NOT accept more than 2 relays on a single IP. +If this happens, the authority _should_ vote for the excess relays, but +should omit the Running or Valid flags and instead should assign the "Sybil" +flag. When there are more than 2 (or AuthDirMaxServersPerAddr) relays to +choose from, authorities should first prefer authorities to non-authorities, +then prefer Running to non-Running, and then prefer high-bandwidth to +low-bandwidth relays. In this comparison, measured bandwidth is used unless +it is not present for a router, in which case advertised bandwidth is used. + +Thus, the network-status vote includes all non-blacklisted, +non-expired, non-superseded descriptors. + +The bandwidth in a "w" line should be taken as the best estimate +of the router's actual capacity that the authority has. For now, +this should be the lesser of the observed bandwidth and bandwidth +rate limit from the server descriptor. It is given in kilobytes +per second, and capped at some arbitrary value (currently 10 MB/s). + +The Measured= keyword on a "w" line vote is currently computed +by multiplying the previous published consensus bandwidth by the +ratio of the measured average node stream capacity to the network +average. If 3 or more authorities provide a Measured= keyword for +a router, the authorities produce a consensus containing a "w" +Bandwidth= keyword equal to the median of the Measured= votes. + +As a special case, if the "w" line in a vote is about a relay with the +Authority flag, it should not include a Measured= keyword. The goal is +to leave such relays marked as Unmeasured, so they can reserve their +attention for authority-specific activities. "w" lines for votes about +authorities may include the bandwidth authority's measurement using +a different keyword, e.g. MeasuredButAuthority=, so it can still be +reported and recorded for posterity. + +The ports listed in a "p" line should be taken as those ports for +which the router's exit policy permits 'most' addresses, ignoring any +accept not for all addresses, ignoring all rejects for private +netblocks. "Most" addresses are permitted if no more than 2^25 +IPv4 addresses (two /8 networks) were blocked. The list is encoded +as described in section 3.8.2. diff --git a/spec/dir-spec/client-operation.md b/spec/dir-spec/client-operation.md new file mode 100644 index 0000000..a59fae4 --- /dev/null +++ b/spec/dir-spec/client-operation.md @@ -0,0 +1,241 @@ +<a id="dir-spec.txt-5"></a> + +# Client operation + +Every Tor that is not a directory server (that is, those that do +not have a DirPort set) implements this section. + +<a id="dir-spec.txt-5.1"></a> + +## Downloading network-status documents { #download-ns } + +Each client maintains a list of directory authorities. Insofar as +possible, clients SHOULD all use the same list. + +```text + [Newer versions of Tor (0.2.8.1-alpha and later): + Each client also maintains a list of default fallback directory mirrors + (fallbacks). Each released version of Tor MAY have a different list, + depending on the mirrors that satisfy the fallback directory criteria at + release time.] +``` + +Clients try to have a live consensus network-status document at all times. +A network-status document is "live" if the time in its valid-after field +has passed, and the time in its valid-until field has not passed. + +When a client has no consensus network-status document, it downloads it +from a randomly chosen fallback directory mirror or authority. Clients +prefer fallbacks to authorities, trying them earlier and more frequently. +In all other cases, the client downloads from caches randomly chosen from +among those believed to be V3 directory servers. (This information comes +from the network-status documents.) + +After receiving any response client MUST discard any network-status +documents that it did not request. + +On failure, the client waits briefly, then tries that network-status +document again from another cache. The client does not build circuits +until it has a live network-status consensus document, and it has +descriptors for a significant proportion of the routers that it believes +are running (this is configurable using torrc options and consensus +parameters). + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. When bootstrap completes, Tor will be ready + to handle an application requesting an exit circuit to services like the + World Wide Web. +``` + +If the consensus does not contain Exits, Tor will only build internal +circuits. In this case, earlier statuses will have included "internal" +as indicated above. When bootstrap completes, Tor will be ready to handle +an application requesting an internal circuit to hidden services at +".onion" addresses. + +If a future consensus contains Exits, exit circuits may become available.\] + +(Note: clients can and should pick caches based on the network-status +information they have: once they have first fetched network-status info +from an authority or fallback, they should not need to go to the authority +directly again, and should only choose the fallback at random, based on its +consensus weight in the current consensus.) + +To avoid swarming the caches whenever a consensus expires, the +clients download new consensuses at a randomly chosen time after the +caches are expected to have a fresh consensus, but before their +consensus will expire. (This time is chosen uniformly at random from +the interval between the time 3/4 into the first interval after the +consensus is no longer fresh, and 7/8 of the time remaining after +that before the consensus is invalid.) + +```text + [For example, if a client has a consensus that became valid at 1:00, + and is fresh until 2:00, and expires at 4:00, that client will fetch + a new consensus at a random time between 2:45 and 3:50, since 3/4 + of the one-hour interval is 45 minutes, and 7/8 of the remaining 75 + minutes is 65 minutes.] +``` + +Clients may choose to download the microdescriptor consensus instead +of the general network status consensus. In that case they should use +the same update strategy as for the normal consensus. They should not +download more than one consensus flavor. + +When a client does not have a live consensus, it will generally use the +most recent consensus it has if that consensus is "reasonably live". A +"reasonably live" consensus is one that expired less than 24 hours ago. + +<a id="dir-spec.txt-5.2"></a> + +## Downloading server descriptors or microdescriptors { #download-desc } + +Clients try to have the best descriptor for each router. A descriptor is +"best" if: + +- It is listed in the consensus network-status document. + +Periodically (currently every 10 seconds) clients check whether there are +any "downloadable" descriptors. A descriptor is downloadable if: + +```text + - It is the "best" descriptor for some router. + - The descriptor was published at least 10 minutes in the past. + (This prevents clients from trying to fetch descriptors that the + mirrors have probably not yet retrieved and cached.) + - The client does not currently have it. + - The client is not currently trying to download it. + - The client would not discard it immediately upon receiving it. + - The client thinks it is running and valid (see section 5.4.1 below). +``` + +If at least 16 known routers have downloadable descriptors, or if +enough time (currently 10 minutes) has passed since the last time the +client tried to download descriptors, it launches requests for all +downloadable descriptors. + +When downloading multiple server descriptors, the client chooses multiple +mirrors so that: + +```text + - At least 3 different mirrors are used, except when this would result + in more than one request for under 4 descriptors. + - No more than 128 descriptors are requested from a single mirror. + - Otherwise, as few mirrors as possible are used. + After choosing mirrors, the client divides the descriptors among them + randomly. +``` + +After receiving any response the client MUST discard any descriptors that +it did not request. + +When a descriptor download fails, the client notes it, and does not +consider the descriptor downloadable again until a certain amount of time +has passed. (Currently 0 seconds for the first failure, 60 seconds for the +second, 5 minutes for the third, 10 minutes for the fourth, and 1 day +thereafter.) Periodically (currently once an hour) clients reset the +failure count. + +Clients retain the most recent descriptor they have downloaded for each +router so long as it is listed in the consensus. If it is not listed, +they keep it so long as it is not too old (currently, ROUTER_MAX_AGE=48 +hours) and no better router descriptor has been downloaded for the same +relay. Caches retain descriptors until they are at least +OLD_ROUTER_DESC_MAX_AGE=5 days old. + +Clients which chose to download the microdescriptor consensus instead +of the general consensus must download the referenced microdescriptors +instead of server descriptors. Clients fetch and cache +microdescriptors preemptively from dir mirrors when starting up, like +they currently fetch descriptors. After bootstrapping, clients only +need to fetch the microdescriptors that have changed. + +When a client gets a new microdescriptor consensus, it looks to see if +there are any microdescriptors it needs to learn, and launches a request +for them. + +Clients maintain a cache of microdescriptors along with metadata like +when it was last referenced by a consensus, and which identity key +it corresponds to. They keep a microdescriptor until it hasn't been +mentioned in any consensus for a week. Future clients might cache them +for longer or shorter times. + +<a id="dir-spec.txt-5.3"></a> + +## Downloading extra-info documents { #download-extra } + +Any client that uses extra-info documents should implement this +section. + +Note that generally, clients don't need extra-info documents. + +Periodically, the Tor instance checks whether it is missing any extra-info +documents: in other words, if it has any server descriptors with an +extra-info-digest field that does not match any of the extra-info +documents currently held. If so, it downloads whatever extra-info +documents are missing. Clients try to download from caches. +We follow the same splitting and back-off rules as in section 5.2. + +<a id="dir-spec.txt-5.5"></a> + +## Retrying failed downloads + +This section applies to caches as well as to clients. + +When a client fails to download a resource (a consensus, a router +descriptor, a microdescriptor, etc) it waits for a certain amount of +time before retrying the download. To determine the amount of time +to wait, clients use a randomized exponential backoff algorithm. +(Specifically, they use a variation of the "decorrelated jitter" +algorithm from +<https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/> .) + +The specific formula used to compute the 'i+1'th delay is: + +```text + Delay_{i+1} = MIN(cap, random_between(lower_bound, upper_bound))) + where upper_bound = MAX(lower_bound+1, Delay_i * 3) + lower_bound = MAX(1, base_delay). +``` + +The value of 'cap' is set to INT_MAX; the value of 'base_delay' +depends on what is being downloaded, whether the client is fully +bootstrapped, how the client is configured, and where it is +downloading from. Current base_delay values are: + +```text + Consensus objects, as a non-bridge cache: + 0 (TestingServerConsensusDownloadInitialDelay) + + Consensus objects, as a client or bridge that has bootstrapped: + 0 (TestingClientConsensusDownloadInitialDelay) + + Consensus objects, as a client or bridge that is bootstrapping, + when connecting to an authority because no "fallback" caches are + known: + 0 (ClientBootstrapConsensusAuthorityOnlyDownloadInitialDelay) + + Consensus objects, as a client or bridge that is bootstrapping, + when "fallback" caches are known but connecting to an authority + anyway: + 6 (ClientBootstrapConsensusAuthorityDownloadInitialDelay) + + Consensus objects, as a client or bridge that is bootstrapping, + when downloading from a "fallback" cache. + 0 (ClientBootstrapConsensusFallbackDownloadInitialDelay) + + Bridge descriptors, as a bridge-using client when at least one bridge + is usable: + 10800 (TestingBridgeDownloadInitialDelay) + + Bridge descriptors, otherwise: + 0 (TestingBridgeBootstrapDownloadInitialDelay) + + Other objects, as cache or authority: + 0 (TestingServerDownloadInitialDelay) + + Other objects, as client: + 0 (TestingClientDownloadInitialDelay) +``` diff --git a/spec/dir-spec/computing-consensus.md b/spec/dir-spec/computing-consensus.md new file mode 100644 index 0000000..329fd45 --- /dev/null +++ b/spec/dir-spec/computing-consensus.md @@ -0,0 +1,683 @@ +# Computing a consensus from a set of votes { #computing-consensus } + +Given a set of votes, authorities compute the contents of the consensus. + +The consensus status, along with as many signatures as the server +currently knows (see section 3.10 below), should be available at + +`http://<hostname>/tor/status-vote/next/consensus.z` + +The contents of the consensus document are as follows: + +The "valid-after", "valid-until", and "fresh-until" times are taken as +the median of the respective values from all the votes. + +The times in the "voting-delay" line are taken as the median of the +VoteSeconds and DistSeconds times in the votes. + +Known-flags is the union of all flags known by any voter. + +Entries are given on the "params" line for every keyword on which a +majority of authorities (total authorities, not just those +participating in this vote) voted on, or if at least three +authorities voted for that parameter. The values given are the +low-median of all votes on that keyword. + +(In consensus methods 7 to 11 inclusive, entries were given on +the "params" line for every keyword on which *any* authority voted, +the value given being the low-median of all votes on that keyword.) + +```text + "client-versions" and "server-versions" are sorted in ascending + order; A version is recommended in the consensus if it is recommended + by more than half of the voting authorities that included a + client-versions or server-versions lines in their votes. +``` + +With consensus methods 19 through 33, a package line is generated for a +given PACKAGENAME/VERSION pair if at least three authorities list such a +package in their votes. (Call these lines the "input" lines for +PACKAGENAME.) The consensus will contain every "package" line that is +listed verbatim by more than half of the authorities listing a line for +the PACKAGENAME/VERSION pair, and no others. + +The authority item groups (dir-source, contact, fingerprint, +vote-digest) are taken from the votes of the voting +authorities. These groups are sorted by the digests of the +authorities identity keys, in ascending order. If the consensus +method is 3 or later, a dir-source line must be included for +every vote with legacy-key entry, using the legacy-key's +fingerprint, the voter's ordinary nickname with the string +"-legacy" appended, and all other fields as from the original +vote's dir-source line. + +```text + A router status entry: + * is included in the result if some router status entry with the same + identity is included by more than half of the authorities (total + authorities, not just those whose votes we have). + (Consensus method earlier than 21) + + * is included according to the rules in section 3.8.0.1 and + 3.8.0.2 below. (Consensus method 22 or later) + + * For any given RSA identity digest, we include at most + one router status entry. + + * For any given Ed25519 identity, we include at most one router + status entry. + + * A router entry has a flag set if that is included by more than half + of the authorities who care about that flag. + + * Two router entries are "the same" if they have the same + (descriptor digest, published time, nickname, IP, ports> tuple. + We choose the tuple for a given router as whichever tuple appears + for that router in the most votes. We break ties first in favor of + the more recently published, then in favor of smaller server + descriptor digest. + + [ + * The Named flag appears if it is included for this routerstatus by + _any_ authority, and if all authorities that list it list the same + nickname. However, if consensus-method 2 or later is in use, and + any authority calls this identity/nickname pair Unnamed, then + this routerstatus does not get the Named flag. + + * If consensus-method 2 or later is in use, the Unnamed flag is + set for a routerstatus if any authorities have voted for a different + identities to be Named with that nickname, or if any authority + lists that nickname/ID pair as Unnamed. + + (With consensus-method 1, Unnamed is set like any other flag.) + + [But note that authorities no longer vote for the Named flag, + and the above two bulletpoints are now irrelevant.] + ] + + * The version is given as whichever version is listed by the most + voters, with ties decided in favor of more recent versions. + + * If consensus-method 4 or later is in use, then routers that + do not have the Running flag are not listed at all. + + * If consensus-method 5 or later is in use, then the "w" line + is generated using a low-median of the bandwidth values from + the votes that included "w" lines for this router. + + * If consensus-method 5 or later is in use, then the "p" line + is taken from the votes that have the same policy summary + for the descriptor we are listing. (They should all be the + same. If they are not, we pick the most commonly listed + one, breaking ties in favor of the lexicographically larger + vote.) The port list is encoded as specified in section 3.8.2. + + * If consensus-method 6 or later is in use and if 3 or more + authorities provide a Measured= keyword in their votes for + a router, the authorities produce a consensus containing a + Bandwidth= keyword equal to the median of the Measured= votes. + + * If consensus-method 7 or later is in use, the params line is + included in the output. + + * If the consensus method is under 11, bad exits are considered as + possible exits when computing bandwidth weights. Otherwise, if + method 11 or later is in use, any router that is determined to get + the BadExit flag doesn't count when we're calculating weights. + + * If consensus method 12 or later is used, only consensus + parameters that more than half of the total number of + authorities voted for are included in the consensus. + + [ As of 0.2.6.1-alpha, authorities no longer advertise or negotiate + any consensus methods lower than 13. ] + + * If consensus method 13 or later is used, microdesc consensuses + omit any router for which no microdesc was agreed upon. + + * If consensus method 14 or later is used, the ns consensus and + microdescriptors may include an "a" line for each router, listing + an IPv6 OR port. + + * If consensus method 15 or later is used, microdescriptors + include "p6" lines including IPv6 exit policies. + + * If consensus method 16 or later is used, ntor-onion-key + are included in microdescriptors + + * If consensus method 17 or later is used, authorities impose a + maximum on the Bandwidth= values that they'll put on a 'w' + line for any router that doesn't have at least 3 measured + bandwidth values in votes. They also add an "Unmeasured=1" + flag to such 'w' lines. + + * If consensus method 18 or later is used, authorities include + "id" lines in microdescriptors. This method adds RSA ids. + + * If consensus method 19 or later is used, authorities may include + "package" lines in consensuses. + + * If consensus method 20 or later is used, authorities may include + GuardFraction information in microdescriptors. + + * If consensus method 21 or later is used, authorities may include + an "id" line for ed25519 identities in microdescriptors. + + [ As of 0.2.8.2-alpha, authorities no longer advertise or negotiate + consensus method 21, because it contains bugs. ] + + * If consensus method 22 or later is used, and the votes do not + produce a majority consensus about a relay's Ed25519 key (see + 3.8.0.1 below), the consensus must include a NoEdConsensus flag on + the "s" line for every relay whose listed Ed key does not reflect + consensus. + + * If consensus method 23 or later is used, authorities include + shared randomness protocol data on their votes and consensus. + + * If consensus-method 24 or later is in use, then routers that + do not have the Valid flag are not listed at all. + + [ As of 0.3.4.1-alpha, authorities no longer advertise or negotiate + any consensus methods lower than 25. ] + + * If consensus-method 25 or later is in use, then we vote + on recommended-protocols and required-protocols lines in the + consensus. We also include protocols lines in routerstatus + entries. + + * If consensus-method 26 or later is in use, then we initialize + bandwidth weights to 1 in our calculations, to avoid + division-by-zero errors on unusual networks. + + * If consensus method 27 or later is used, the microdesc consensus + may include an "a" line for each router, listing an IPv6 OR port. + + [ As of 0.4.3.1-alpha, authorities no longer advertise or negotiate + any consensus methods lower than 28. ] + + * If consensus method 28 or later is used, microdescriptors no longer + include "a" lines. + + * If consensus method 29 or later is used, microdescriptor "family" + lines are canonicalized to improve compression. + + * If consensus method 30 or later is used, the base64 encoded + ntor-onion-key does not include the trailing = sign. + + * If consensus method 31 or later is used, authorities parse the + "bwweightscale" and "maxunmeasuredbw" parameters correctly when + computing votes. + + * If consensus method 32 or later is used, authorities handle the + "MiddleOnly" flag specially when computing a consensus. When the + voters agree to include "MiddleOnly" in a routerstatus, they + automatically remove "Exit", "Guard", "V2Dir", and "HSDir". If + the BadExit flag is included in the consensus, they automatically + add it to the routerstatus. + + * If consensus method 33 or later is used, and the consensus + flavor is "microdesc", then the "Publication" field in the "r" + line is set to "2038-01-01 00:00:00". + + * If consensus method 34 or later is used, the consensus + does not include any "package" lines. +``` + +The signatures at the end of a consensus document are sorted in +ascending order by identity digest. + +All ties in computing medians are broken in favor of the smaller or +earlier item. + +<a id="dir-spec.txt-3.8.0.1"></a> + +## Deciding which Ids to include { #choosing-relay-ids } + +This sorting algorithm is used for consensus-method 22 and later. + +```text + First, consider each listing by tuple of <Ed,Rsa> identities, where 'Ed' + may be "None" if the voter included "id ed25519 none" to indicate that + the authority knows what ed25519 identities are, and thinks that the RSA + key doesn't have one. + + For each such <Ed, RSA> tuple that is listed by more than half of the + total authorities (not just total votes), include it. (It is not + possible for any other <id-Ed, id-RSA'> to have as many votes.) If more + than half of the authorities list a single <Ed,Rsa> pair of this type, we + consider that Ed key to be "consensus"; see description of the + NoEdConsensus flag. + + Log any other id-RSA values corresponding to an id-Ed we included, and any + other id-Ed values corresponding to an id-RSA we included. + + For each <id-RSA> that is not yet included, if it is listed by more than + half of the total authorities, and we do not already have it listed with + some <id-Ed>, include it, but do not consider its Ed identity canonical. +``` + +<a id="dir-spec.txt-3.8.0.2"></a> + +### Deciding which descriptors to include { #choosing-relay-descs } + +Deciding which descriptors to include. + +A tuple belongs to an `<id-RSA, id-Ed>` identity if it is a new tuple that +matches both ID parts, or if it is an old tuple (one with no Ed opinion) +that matches the RSA part. A tuple belongs to an `<id-RSA>` identity if its +RSA identity matches. + +A tuple matches another tuple if all the fields that are present in both +tuples are the same. + +For every included identity, consider the tuples belonging to that +identity. Group them into sets of matching tuples. Include the tuple +that matches the largest set, breaking ties in favor of the most recently +published, and then in favor of the smaller server descriptor digest. + +<a id="dir-spec.txt-3.8.1"></a> + +## Forward compatibility { #consensus-method-list } + +Future versions of Tor will need to include new information in the +consensus documents, but it is important that all authorities (or at least +half) generate and sign the same signed consensus. + +To achieve this, authorities list in their votes their supported methods +for generating consensuses from votes. Later methods will be assigned +higher numbers. Currently specified methods: + +```text + "1" -- The first implemented version. + "2" -- Added support for the Unnamed flag. + "3" -- Added legacy ID key support to aid in authority ID key rollovers + "4" -- No longer list routers that are not running in the consensus + "5" -- adds support for "w" and "p" lines. + "6" -- Prefers measured bandwidth values rather than advertised + "7" -- Provides keyword=integer pairs of consensus parameters + "8" -- Provides microdescriptor summaries + "9" -- Provides weights for selecting flagged routers in paths + "10" -- Fixes edge case bugs in router flag selection weights + "11" -- Don't consider BadExits when calculating bandwidth weights + "12" -- Params are only included if enough auths voted for them + "13" -- Omit router entries with missing microdescriptors. + "14" -- Adds support for "a" lines in ns consensuses and microdescriptors. + "15" -- Adds support for "p6" lines. + "16" -- Adds ntor keys to microdescriptors + "17" -- Adds "Unmeasured=1" flags to "w" lines + "18" -- Adds 'id' to microdescriptors. + "19" -- Adds "package" lines to consensuses + "20" -- Adds GuardFraction information to microdescriptors. + "21" -- Adds Ed25519 keys to microdescriptors. + "22" -- Instantiates Ed25519 voting algorithm correctly. + "23" -- Adds shared randomness protocol data. + "24" -- No longer lists routers that are not Valid in the consensus. + "25" -- Vote on recommended-protocols and required-protocols. + "26" -- Initialize bandwidth weights to 1 to avoid division-by-zero. + "27" -- Adds support for "a" lines in microdescriptor consensuses. + "28" -- Removes "a" lines from microdescriptors. + "29" -- Canonicalizes families in microdescriptors. + "30" -- Removes padding from ntor-onion-key. + "31" -- Uses correct parsing for bwweightscale and maxunmeasuredbw + when computing weights + "32" -- Adds special handling for MiddleOnly flag. + "33" -- Sets "publication" field in microdesc consensus "r" lines + to a meaningless value. + "34" -- Removes "package" lines from consensus. +``` + +Before generating a consensus, an authority must decide which consensus +method to use. To do this, it looks for the highest version number +supported by more than 2/3 of the authorities voting. If it supports this +method, then it uses it. Otherwise, it falls back to the newest consensus +method that it supports (which will probably not result in a sufficiently +signed consensus). + +All authorities MUST support method 25; authorities SHOULD support +more recent methods as well. Authorities SHOULD NOT support or +advertise support for any method before 25. Clients MAY assume that +they will never see a current valid signed consensus for any method +before method 25. + +(The consensuses generated by new methods must be parsable by +implementations that only understand the old methods, and must not cause +those implementations to compromise their anonymity. This is a means for +making changes in the contents of consensus; not for making +backward-incompatible changes in their format.) + +The following methods have incorrect implementations; authorities SHOULD +NOT advertise support for them: + +"21" -- Did not correctly enable support for ed25519 key collation. + +<a id="dir-spec.txt-3.8.2"></a> + +## Encoding port lists + +Whether the summary shows the list of accepted ports or the list of +rejected ports depends on which list is shorter (has a shorter string +representation). In case of ties we choose the list of accepted +ports. As an exception to this rule an allow-all policy is +represented as "accept 1-65535" instead of "reject " and a reject-all +policy is similarly given as "reject 1-65535". + +Summary items are compressed, that is instead of "80-88,89-100" there +only is a single item of "80-100", similarly instead of "20,21" a +summary will say "20-21". + +Port lists are sorted in ascending order. + +The maximum allowed length of a policy summary (including the "accept " +or "reject ") is 1000 characters. If a summary exceeds that length we +use an accept-style summary and list as much of the port list as is +possible within these 1000 bytes. \[XXXX be more specific.\] + +<a id="dir-spec.txt-3.8.3"></a> + +## Computing Bandwidth Weights + +Let weight_scale = 10000, or the value of the "bwweightscale" parameter. +(Before consensus method 31 there was a bug in parsing bwweightscale, so +that if there were any consensus parameters after it alphabetically, it +would always be treated as 10000. A similar bug existed for +"maxunmeasuredbw".) + +Starting with consensus method 26, G, M, E, and D are initialized to 1 and +T to 4. Prior consensus methods initialize them all to 0. With this change, +test tor networks that are small or new are much more likely to produce +bandwidth-weights in their consensus. The extra bandwidth has a negligible +impact on the bandwidth weights in the public tor network. + +Let G be the total bandwidth for Guard-flagged nodes. +Let M be the total bandwidth for non-flagged nodes. +Let E be the total bandwidth for Exit-flagged nodes. +Let D be the total bandwidth for Guard+Exit-flagged nodes. +Let T = G+M+E+D + +Let Wgd be the weight for choosing a Guard+Exit for the guard position. +Let Wmd be the weight for choosing a Guard+Exit for the middle position. +Let Wed be the weight for choosing a Guard+Exit for the exit position. + +Let Wme be the weight for choosing an Exit for the middle position. +Let Wmg be the weight for choosing a Guard for the middle position. + +Let Wgg be the weight for choosing a Guard for the guard position. +Let Wee be the weight for choosing an Exit for the exit position. + +Balanced network conditions then arise from solutions to the following +system of equations: + +Wgg*G + Wgd*D == M + Wmd*D + Wme*E + Wmg*G (guard bw = middle bw) +Wgg*G + Wgd*D == Wee*E + Wed*D (guard bw = exit bw) +Wed*D + Wmd*D + Wgd*D == D (aka: Wed+Wmd+Wdg = weight_scale) +Wmg*G + Wgg*G == G (aka: Wgg = weight_scale-Wmg) +Wme*E + Wee*E == E (aka: Wee = weight_scale-Wme) + +We are short 2 constraints with the above set. The remaining constraints +come from examining different cases of network load. The following +constraints are used in consensus method 10 and above. There are another +incorrect and obsolete set of constraints used for these same cases in +consensus method 9. For those, see dir-spec.txt in Tor 0.2.2.10-alpha +to 0.2.2.16-alpha. + +Case 1: E >= T/3 && G >= T/3 (Neither Exit nor Guard Scarce) + +In this case, the additional two constraints are: Wmg == Wmd, +Wed == 1/3. + +```text + This leads to the solution: + Wgd = weight_scale/3 + Wed = weight_scale/3 + Wmd = weight_scale/3 + Wee = (weight_scale*(E+G+M))/(3*E) + Wme = weight_scale - Wee + Wmg = (weight_scale*(2*G-E-M))/(3*G) + Wgg = weight_scale - Wmg + + Case 2: E < T/3 && G < T/3 (Both are scarce) +``` + +Let R denote the more scarce class (Rare) between Guard vs Exit. +Let S denote the less scarce class. + +Subcase a: R+D \< S + +In this subcase, we simply devote all of D bandwidth to the +scarce class. + +```text + Wgg = Wee = weight_scale + Wmg = Wme = Wmd = 0; + if E < G: + Wed = weight_scale + Wgd = 0 + else: + Wed = 0 + Wgd = weight_scale + + Subcase b: R+D >= S +``` + +In this case, if M \<= T/3, we have enough bandwidth to try to achieve +a balancing condition. + +Add constraints Wgg = weight_scale, Wmd == Wgd to maximize bandwidth in +the guard position while still allowing exits to be used as middle nodes: + +Wee = (weight_scale\*(E - G + M))/E +Wed = (weight_scale\*(D - 2*E + 4*G - 2*M))/(3*D) +Wme = (weight_scale\*(G-M))/E +Wmg = 0 +Wgg = weight_scale +Wmd = (weight_scale - Wed)/2 +Wgd = (weight_scale - Wed)/2 + +If this system ends up with any values out of range (ie negative, or +above weight_scale), use the constraints Wgg == weight_scale and Wee == +weight_scale, since both those positions are scarce: + +```text + Wgg = weight_scale + Wee = weight_scale + Wed = (weight_scale*(D - 2*E + G + M))/(3*D) + Wmd = (weight_Scale*(D - 2*M + G + E))/(3*D) + Wme = 0 + Wmg = 0 + Wgd = weight_scale - Wed - Wmd + + If M > T/3, then the Wmd weight above will become negative. Set it to 0 + in this case: + Wmd = 0 + Wgd = weight_scale - Wed + + Case 3: One of E < T/3 or G < T/3 + + Let S be the scarce class (of E or G). + + Subcase a: (S+D) < T/3: + if G=S: + Wgg = Wgd = weight_scale; + Wmd = Wed = Wmg = 0; + // Minor subcase, if E is more scarce than M, + // keep its bandwidth in place. + if (E < M) Wme = 0; + else Wme = (weight_scale*(E-M))/(2*E); + Wee = weight_scale-Wme; + if E=S: + Wee = Wed = weight_scale; + Wmd = Wgd = Wme = 0; + // Minor subcase, if G is more scarce than M, + // keep its bandwidth in place. + if (G < M) Wmg = 0; + else Wmg = (weight_scale*(G-M))/(2*G); + Wgg = weight_scale-Wmg; + + Subcase b: (S+D) >= T/3 + if G=S: + Add constraints Wgg = weight_scale, Wmd == Wed to maximize bandwidth + in the guard position, while still allowing exits to be + used as middle nodes: + Wgg = weight_scale + Wgd = (weight_scale*(D - 2*G + E + M))/(3*D) + Wmg = 0 + Wee = (weight_scale*(E+M))/(2*E) + Wme = weight_scale - Wee + Wmd = (weight_scale - Wgd)/2 + Wed = (weight_scale - Wgd)/2 + if E=S: + Add constraints Wee == weight_scale, Wmd == Wgd to maximize bandwidth + in the exit position: + Wee = weight_scale; + Wed = (weight_scale*(D - 2*E + G + M))/(3*D); + Wme = 0; + Wgg = (weight_scale*(G+M))/(2*G); + Wmg = weight_scale - Wgg; + Wmd = (weight_scale - Wed)/2; + Wgd = (weight_scale - Wed)/2; +``` + +To ensure consensus, all calculations are performed using integer math +with a fixed precision determined by the bwweightscale consensus +parameter (defaults at 10000, Min: 1, Max: INT32_MAX). (See note above +about parsing bug in bwweightscale before consensus method 31.) + +For future balancing improvements, Tor clients support 11 additional weights +for directory requests and middle weighting. These weights are currently +set at weight_scale, with the exception of the following groups of +assignments: + +Directory requests use middle weights: + +Wbd=Wmd, Wbg=Wmg, Wbe=Wme, Wbm=Wmm + +Handle bridges and strange exit policies: + +Wgm=Wgg, Wem=Wee, Weg=Wed + +<a id="dir-spec.txt-3.9"></a> + +## Computing consensus flavors + +Consensus flavors are variants of the consensus that clients can choose +to download and use instead of the unflavored consensus. The purpose +of a consensus flavor is to remove or replace information in the +unflavored consensus without forcing clients to download information +they would not use anyway. + +Directory authorities can produce and serve an arbitrary number of +flavors of the same consensus. A downside of creating too many new +flavors is that clients will be distinguishable based on which flavor +they download. A new flavor should not be created when adding a field +instead wouldn't be too onerous. + +Examples for consensus flavors include: + +```text + - Publishing hashes of microdescriptors instead of hashes of + full descriptors (see section 3.9.2). + - Including different digests of descriptors, instead of the + perhaps-soon-to-be-totally-broken SHA1. +``` + +Consensus flavors are derived from the unflavored consensus once the +voting process is complete. This is to avoid consensus synchronization +problems. + +Every consensus flavor has a name consisting of a sequence of one +or more alphanumeric characters and dashes. For compatibility, +the original (unflavored) consensus type is called "ns". + +The supported consensus flavors are defined as part of the +authorities' consensus method. + +All consensus flavors have in common that their first line is +"network-status-version" where version is 3 or higher, and the flavor +is a string consisting of alphanumeric characters and dashes: + +"network-status-version" SP version \[SP flavor\] NL + +<a id="dir-spec.txt-3.9.1"></a> + +### ns consensus + +The ns consensus flavor is equivalent to the unflavored consensus. +When the flavor is omitted from the "network-status-version" line, +it should be assumed to be "ns". Some implementations may explicitly +state that the flavor is "ns" when generating consensuses, but should +accept consensuses where the flavor is omitted. + +<a id="dir-spec.txt-3.9.2"></a> + +### Microdescriptor consensus + +The microdescriptor consensus is a consensus flavor that contains +microdescriptor hashes instead of descriptor hashes and that omits +exit-policy summaries which are contained in microdescriptors. The +microdescriptor consensus was designed to contain elements that are +small and frequently changing. Clients use the information in the +microdescriptor consensus to decide which servers to fetch information +about and which servers to fetch information from. + +The microdescriptor consensus is based on the unflavored consensus with +the exceptions as follows: + +"network-status-version" SP version SP "microdesc" NL + +\[At start, exactly once.\] + +The flavor name of a microdescriptor consensus is "microdesc". + +Changes to router status entries are as follows: + +```text + "r" SP nickname SP identity SP publication SP IP SP ORPort + SP DirPort NL + + [At start, exactly once.] + + Similar to "r" lines in section 3.4.1, but without the digest element. + + "a" SP address ":" port NL + + [Any number] + + Identical to the "r" lines in section 3.4.1. +``` + +(Only included when the vote is generated with consensus-method 14 +or later, and the consensus is generated with consensus-method 27 or +later.) + +"p" ... NL + +\[At most once\] + +Not currently generated. + +Exit policy summaries are contained in microdescriptors and +therefore omitted in the microdescriptor consensus. + +"m" SP digest NL + +\[Exactly once.\*\] + +"digest" is the base64 of the SHA256 hash of the router's +microdescriptor with trailing =s omitted. For a given router +descriptor digest and consensus method there should only be a +single microdescriptor digest in the "m" lines of all votes. +If different votes have different microdescriptor digests for +the same descriptor digest and consensus method, at least one +of the authorities is broken. If this happens, the microdesc +consensus should contain whichever microdescriptor digest is +most common. If there is no winner, we break ties in the favor +of the lexically earliest. + +\[\*Before consensus method 13, this field was sometimes erroneously +omitted.\] + +Additionally, a microdescriptor consensus SHOULD use the sha256 digest +algorithm for its signatures. + +<a id="dir-spec.txt-3.10"></a> diff --git a/spec/dir-spec/computing-microdescriptors.md b/spec/dir-spec/computing-microdescriptors.md new file mode 100644 index 0000000..655c66a --- /dev/null +++ b/spec/dir-spec/computing-microdescriptors.md @@ -0,0 +1,161 @@ +<a id="dir-spec.txt-3.3"></a> + +# Computing microdescriptors + +Microdescriptors are a stripped-down version of server descriptors +generated by the directory authorities which may additionally contain +authority-generated information. Microdescriptors contain only the +most relevant parts that clients care about. Microdescriptors are +expected to be relatively static and only change about once per week. +Microdescriptors do not contain any information that clients need to +use to decide which servers to fetch information about, or which +servers to fetch information from. + +Microdescriptors are a straight transform from the server descriptor +and the consensus method. Microdescriptors have no header or footer. +Microdescriptors are identified by the hash of its concatenated +elements without a signature by the router. Microdescriptors do not +contain any version information, because their version is determined +by the consensus method. + +Starting with consensus method 8, microdescriptors contain the +following elements taken from or based on the server descriptor. Order +matters here, because different directory authorities must be able to +transform a given server descriptor and consensus method into the exact +same microdescriptor. + +"onion-key" NL a public key in PEM format + +\[Exactly once, at start\] +\[No extra arguments\] + +The "onion-key" element as specified in section 2.1.1. + +When generating microdescriptors for consensus method 30 or later, +the trailing = sign must be absent. For consensus method 29 or +earlier, the trailing = sign must be present. + +"ntor-onion-key" SP base-64-encoded-key NL + +\[Exactly once\] + +The "ntor-onion-key" element as specified in section 2.1.1. + +(Only included when generating microdescriptors for +consensus-method 16 or later.) + +\[Before Tor 0.4.5.1-alpha, this field was optional.\] + +"a" SP address ":" port NL + +\[Any number\] + +Additional advertised addresses for the OR. + +Present currently only if the OR advertises at least one IPv6 +address; currently, the first address is included and all others are +omitted. Any other IPv4 or IPv6 addresses should be ignored. + +Address and port are as for "or-address" as specified in +section 2.1.1. + +(Only included when generating microdescriptors for +consensus-methods 14 to 27.) + +"family" names NL + +\[At most once\] + +The "family" element as specified in section 2.1.1. + +When generating microdescriptors for consensus method 29 or later, +the following canonicalization algorithm is applied to improve +compression: + +```text + For all entries of the form $hexid=name or $hexid~name, + remove the =name or ~name portion. + + Remove all entries of the form $hexid, where hexid is not + 40 hexadecimal characters long. + + If an entry is a valid nickname, put it into lower case. + + If an entry is a valid $hexid, put it into upper case. + + If there are any entries, add a single $hexid entry for + the relay in question, so that it is a member of its own + family. + + Sort all entries in lexical order. + + Remove duplicate entries. +``` + +(Note that if an entry is not of the form "nickname", "$hexid", +"$hexid=nickname" or "$hexid~nickname", then it will be unchanged: +this is what makes the algorithm forward-compatible.) + +"p" SP ("accept" / "reject") SP PortList NL + +\[At most once.\] + +The exit-policy summary as specified in sections 3.4.1 and 3.8.2. + +\[With microdescriptors, clients don't learn exact exit policies: +clients can only guess whether a relay accepts their request, try the +BEGIN request, and might get end-reason-exit-policy if they guessed +wrong, in which case they'll have to try elsewhere.\] + +\[In consensus methods before 5, this line was omitted.\] + +"p6" SP ("accept" / "reject") SP PortList NL + +\[At most once\] + +The IPv6 exit policy summary as specified in sections 3.4.1 and +3.8.2. A missing "p6" line is equivalent to "p6 reject 1-65535". + +(Only included when generating microdescriptors for +consensus-method 15 or later.) + +"id" SP "rsa1024" SP base64-encoded-identity-digest NL + +\[At most once\] + +The node identity digest (as described in tor-spec.txt), base64 +encoded, without trailing =s. This line is included to prevent +collisions between microdescriptors. + +Implementations SHOULD ignore these lines: they are +added to microdescriptors only to prevent collisions. + +(Only included when generating microdescriptors for +consensus-method 18 or later.) + +"id" SP "ed25519" SP base64-encoded-ed25519-identity NL + +\[At most once\] + +The node's master Ed25519 identity key, base64 encoded, +without trailing =s. + +All implementations MUST ignore this key for any microdescriptor +whose corresponding entry in the consensus includes the +'NoEdConsensus' flag. + +(Only included when generating microdescriptors for +consensus-method 21 or later.) + +"id" SP keytype ... NL + +\[At most once per distinct keytype.\] + +Implementations MUST ignore "id" lines with unrecognized +key-types in place of "rsa1024" or "ed25519" + +(Note that with microdescriptors, clients do not learn the RSA identity of +their routers: they only learn a hash of the RSA identity key. This is +all they need to confirm the actual identity key when doing a TLS +handshake, and all they need to put the identity key digest in their +CREATE cells.) diff --git a/spec/dir-spec/consensus-formats.md b/spec/dir-spec/consensus-formats.md new file mode 100644 index 0000000..7a6cd46 --- /dev/null +++ b/spec/dir-spec/consensus-formats.md @@ -0,0 +1,706 @@ +<a id="dir-spec.txt-3.4.1"></a> + +# Vote and consensus status document formats + +Votes and consensuses are more strictly formatted than other documents +in this specification, since different authorities must be able to +generate exactly the same consensus given the same set of votes. + +The procedure for deciding when to generate vote and consensus status +documents are described in section 1.4 on the voting timeline. + +Status documents contain a preamble, an authority section, a list of +router status entries, and one or more footer signature, in that order. + +Unlike other formats described above, a SP in these documents must be a +single space character (hex 20). + +Some items appear only in votes, and some items appear only in +consensuses. Unless specified, items occur in both. + +The preamble contains the following items. They SHOULD occur in the +order given here: + +"network-status-version" SP version NL + +\[At start, exactly once.\] + +A document format version. For this specification, the version is +"3". + +"vote-status" SP type NL + +\[Exactly once.\] + +The status MUST be "vote" or "consensus", depending on the type of +the document. + +"consensus-methods" SP IntegerList NL + +\[At most once for votes; does not occur in consensuses.\] + +A space-separated list of supported methods for generating +consensuses from votes. See section 3.8.1 for details. Absence of +the line means that only method "1" is supported. + +"consensus-method" SP Integer NL + +\[At most once for consensuses; does not occur in votes.\] +\[No extra arguments\] + +See section 3.8.1 for details. + +(Only included when the vote is generated with consensus-method 2 or +later.) + +"published" SP YYYY-MM-DD SP HH:MM:SS NL + +\[Exactly once for votes; does not occur in consensuses.\] + +The publication time for this status document (if a vote). + +"valid-after" SP YYYY-MM-DD SP HH:MM:SS NL + +\[Exactly once.\] + +The start of the Interval for this vote. Before this time, the +consensus document produced from this vote is not officially in +use. + +(Note that because of propagation delays, clients and relays +may see consensus documents that are up to `DistSeconds` +earlier than this time, and should not warn about them.) + +See section 1.4 for voting timeline information. + +"fresh-until" SP YYYY-MM-DD SP HH:MM:SS NL + +\[Exactly once.\] + +The time at which the next consensus should be produced; before this +time, there is no point in downloading another consensus, since there +won't be a new one. See section 1.4 for voting timeline information. + +"valid-until" SP YYYY-MM-DD SP HH:MM:SS NL + +\[Exactly once.\] + +The end of the Interval for this vote. After this time, all +clients should try to find a more recent consensus. See section 1.4 +for voting timeline information. + +In practice, clients continue to use the consensus for up to 24 hours +after it is no longer valid, if no more recent consensus can be +downloaded. + +"voting-delay" SP VoteSeconds SP DistSeconds NL + +\[Exactly once.\] + +VoteSeconds is the number of seconds that we will allow to collect +votes from all authorities; DistSeconds is the number of seconds +we'll allow to collect signatures from all authorities. See +section 1.4 for voting timeline information. + +"client-versions" SP VersionList NL + +\[At most once.\] + +A comma-separated list of recommended Tor versions for client +usage, in ascending order. The versions are given as defined by +version-spec.txt. If absent, no opinion is held about client +versions. + +"server-versions" SP VersionList NL + +\[At most once.\] + +A comma-separated list of recommended Tor versions for relay +usage, in ascending order. The versions are given as defined by +version-spec.txt. If absent, no opinion is held about server +versions. + +"package" SP PackageName SP Version SP URL SP DIGESTS NL + +\[Any number of times.\] + +For this element: + +```text + PACKAGENAME = NONSPACE + VERSION = NONSPACE + URL = NONSPACE + DIGESTS = DIGEST | DIGESTS SP DIGEST + DIGEST = DIGESTTYPE "=" DIGESTVAL + NONSPACE = one or more non-space printing characters + DIGESTVAL = DIGESTTYPE = one or more non-space printing characters + other than "=". +``` + +Indicates that a package called "package" of version VERSION may be +found at URL, and its digest as computed with DIGESTTYPE is equal to +DIGESTVAL. In consensuses, these lines are sorted lexically by +"PACKAGENAME VERSION" pairs, and DIGESTTYPES must appear in ascending +order. A consensus must not contain the same "PACKAGENAME VERSION" +more than once. If a vote contains the same "PACKAGENAME VERSION" +more than once, all but the last is ignored. + +Included in consensuses only for methods 19-33. Earlier methods +did not include this; method 34 removed it. + +"known-flags" SP FlagList NL + +\[Exactly once.\] + +A space-separated list of all of the flags that this document +might contain. A flag is "known" either because the authority +knows about them and might set them (if in a vote), or because +enough votes were counted for the consensus for an authoritative +opinion to have been formed about their status. + +"flag-thresholds" SP Thresholds NL + +\[At most once for votes; does not occur in consensuses.\] + +```text + A space-separated list of the internal performance thresholds + that the directory authority had at the moment it was forming + a vote. + + The metaformat is: + Thresholds = Threshold | Threshold SP Thresholds + Threshold = ThresholdKey '=' ThresholdVal + ThresholdKey = (KeywordChar | "_") + + ThresholdVal = [0-9]+("."[0-9]+)? "%"? + + Commonly used Thresholds at this point include: + + "stable-uptime" -- Uptime (in seconds) required for a relay + to be marked as stable. + + "stable-mtbf" -- MTBF (in seconds) required for a relay to be + marked as stable. + + "enough-mtbf" -- Whether we have measured enough MTBF to look + at stable-mtbf instead of stable-uptime. + + "fast-speed" -- Bandwidth (in bytes per second) required for + a relay to be marked as fast. + + "guard-wfu" -- WFU (in seconds) required for a relay to be + marked as guard. + + "guard-tk" -- Weighted Time Known (in seconds) required for a + relay to be marked as guard. + + "guard-bw-inc-exits" -- If exits can be guards, then all guards + must have a bandwidth this high. + + "guard-bw-exc-exits" -- If exits can't be guards, then all guards + must have a bandwidth this high. + + "ignoring-advertised-bws" -- 1 if we have enough measured bandwidths + that we'll ignore the advertised bandwidth + claims of routers without measured bandwidth. +``` + +"recommended-client-protocols" SP Entries NL +"recommended-relay-protocols" SP Entries NL +"required-client-protocols" SP Entries NL +"required-relay-protocols" SP Entries NL + +\[At most once for each.\] + +The "proto" element as specified in section 2.1.1. + +To vote on these entries, a protocol/version combination is included +only if it is listed by a majority of the voters. + +These lines should be voted on. A majority of votes is sufficient to +make a protocol un-supported. A supermajority of authorities (2/3) +are needed to make a protocol required. The required protocols +should not be torrc-configurable, but rather should be hardwired in +the Tor code. + +The tor-spec.txt section 9 details how a relay and a client should +behave when they encounter these lines in the consensus. + +"params" SP \[Parameters\] NL + +\[At most once\] + +Parameter ::= Keyword '=' Int32 +Int32 ::= A decimal integer between -2147483648 and 2147483647. +Parameters ::= Parameter | Parameters SP Parameter + +The parameters list, if present, contains a space-separated list of +case-sensitive key-value pairs, sorted in lexical order by their +keyword (as ASCII byte strings). Each parameter has its own meaning. + +(Only included when the vote is generated with consensus-method 7 or +later.) + +See param-spec.txt for a list of parameters and their meanings. + +"shared-rand-previous-value" SP NumReveals SP Value NL + +\[At most once\] + +NumReveals ::= An integer greater or equal to 0. +Value ::= Base64-encoded-data + +The shared_random_value that was generated during the second-to-last +shared randomness protocol run. For example, if this document was +created on the 5th of November, this field carries the shared random +value generated during the protocol run of the 3rd of November. + +See section \[SRCALC\] of srv-spec.txt for instructions on how to compute +this value, and see section \[CONS\] for why we include old shared random +values in votes and consensus. + +Value is the actual shared random value encoded in base64. It will +be exactly 256 bits long. NumReveals is the number of commits used +to generate this SRV. + +"shared-rand-current-value" SP NumReveals SP Value NL + +\[At most once\] + +NumReveals ::= An integer greater or equal to 0. +Value ::= Base64-encoded-data + +The shared_random_value that was generated during the latest shared +randomness protocol run. For example, if this document was created on +the 5th of November, this field carries the shared random value +generated during the protocol run of the 4th of November + +See section \[SRCALC\] of srv-spec.txt for instructions on how to compute +this value given the active commits. + +Value is the actual shared random value encoded in base64. It will +be exactly 256 bits long. NumReveals is the number of commits used to +generate this SRV. + +"bandwidth-file-headers" SP KeyValues NL + +\[At most once for votes; does not occur in consensuses.\] + +KeyValues ::= "" | KeyValue | KeyValues SP KeyValue +KeyValue ::= Keyword '=' Value +Value ::= ArgumentCharValue+ +ArgumentCharValue ::= any printing ASCII character except NL and SP. + +The headers from the bandwidth file used to generate this vote. +The bandwidth file headers are described in bandwidth-file-spec.txt. + +If an authority is not configured with a V3BandwidthsFile, this line +SHOULD NOT appear in its vote. + +If an authority is configured with a V3BandwidthsFile, but parsing +fails, this line SHOULD appear in its vote, but without any headers. + +First-appeared: Tor 0.3.5.1-alpha. + +"bandwidth-file-digest" 1\*(SP algorithm "=" digest) NL + +\[At most once for votes; does not occur in consensuses.\] + +A digest of the bandwidth file used to generate this vote. +"algorithm" is the name of the hash algorithm producing "digest", +which can be "sha256" or another algorithm. "digest" is the +base64 encoding of the hash of the bandwidth file, with trailing =s +omitted. + +If an authority is not configured with a V3BandwidthsFile, this line +SHOULD NOT appear in its vote. + +If an authority is configured with a V3BandwidthsFile, but parsing +fails, this line SHOULD appear in its vote, with the digest(s) of the +unparseable file. + +First-appeared: Tor 0.4.0.4-alpha + +The authority section of a vote contains the following items, followed +in turn by the authority's current key certificate: + +```text + "dir-source" SP nickname SP identity SP address SP IP SP dirport SP + orport NL + + [Exactly once, at start] +``` + +Describes this authority. The nickname is a convenient identifier +for the authority. The identity is an uppercase hex fingerprint of +the authority's current (v3 authority) identity key. The address is +the server's hostname. The IP is the server's current IP address, +and dirport is its current directory port. The orport is the +port at that address where the authority listens for OR +connections. + +"contact" SP string NL + +\[Exactly once\] + +An arbitrary string describing how to contact the directory +server's administrator. Administrators should include at least an +email address and a PGP fingerprint. + +"legacy-dir-key" SP FINGERPRINT NL + +\[At most once\] + +Lists a fingerprint for an obsolete _identity_ key still used +by this authority to keep older clients working. This option +is used to keep key around for a little while in case the +authorities need to migrate many identity keys at once. +(Generally, this would only happen because of a security +vulnerability that affected multiple authorities, like the +Debian OpenSSL RNG bug of May 2008.) + +"shared-rand-participate" NL + +\[At most once\] + +Denotes that the directory authority supports and can participate in the +shared random protocol. + +"shared-rand-commit" SP Version SP AlgName SP Identity SP Commit \[SP Reveal\] NL + +\[Any number of times\] + +Version ::= An integer greater or equal to 0. +AlgName ::= 1\*(ALPHA / DIGIT / "\_" / "-") +Identity ::= 40\* HEXDIG +Commit ::= Base64-encoded-data +Reveal ::= Base64-encoded-data + +Denotes a directory authority commit for the shared randomness +protocol, containing the commitment value and potentially also the +reveal value. See sections \[COMMITREVEAL\] and \[VALIDATEVALUES\] of +srv-spec.txt on how to generate and validate these values. + +Version is the current shared randomness protocol version. AlgName is +the hash algorithm that is used (e.g. "sha3-256") and Identity is the +authority's SHA1 v3 identity fingerprint. Commit is the encoded +commitment value in base64. Reveal is optional and if it's set, it +contains the reveal value in base64. + +If a vote contains multiple commits from the same authority, the +receiver MUST only consider the first commit listed. + +"shared-rand-previous-value" SP NumReveals SP Value NL + +\[At most once\] + +See shared-rand-previous-value description above. + +"shared-rand-current-value" SP NumReveals SP Value NL + +\[At most once\] + +See shared-rand-current-value description above. + +The authority section of a consensus contains groups of the following items, +in the order given, with one group for each authority that contributed to +the consensus, with groups sorted by authority identity digest: + +```text + "dir-source" SP nickname SP identity SP address SP IP SP dirport SP + orport NL + + [Exactly once, at start] + + As in the authority section of a vote. + + "contact" SP string NL + + [Exactly once.] + + As in the authority section of a vote. + + "vote-digest" SP digest NL + + [Exactly once.] +``` + +A digest of the vote from the authority that contributed to this +consensus, as signed (that is, not including the signature). +(Hex, upper-case.) + +For each "legacy-dir-key" in the vote, there is an additional "dir-source" +line containing that legacy key's fingerprint, the authority's nickname +with "-legacy" appended, and all other fields as in the main "dir-source" +line for that authority. These "dir-source" lines do not have +corresponding "contact" or "vote-digest" entries. + +Each router status entry contains the following items. Router status +entries are sorted in ascending order by identity digest. + +```text + "r" SP nickname SP identity SP digest SP publication SP IP SP ORPort + SP DirPort NL + + [At start, exactly once.] +``` + +"Nickname" is the OR's nickname. "Identity" is a hash of its +identity key, encoded in base64, with trailing equals sign(s) +removed. "Digest" is a hash of its most recent descriptor as +signed (that is, not including the signature) by the RSA identity +key (see section 1.3.), encoded in base64. + +"Publication" was once the publication time of the router's most +recent descriptor, in the form YYYY-MM-DD HH:MM:SS, in UTC. Now +it is only used in votes, and may be set to a fixed value in +consensus documents. Implementations SHOULD ignore this value +in non-vote documents. + +"IP" is its current IP address; ORPort is its current OR port, +"DirPort" is its current directory port, or "0" for "none". + +"a" SP address ":" port NL + +\[Any number\] + +The first advertised IPv6 address for the OR, if it is reachable. + +Present only if the OR advertises at least one IPv6 address, and the +authority believes that the first advertised address is reachable. +Any other IPv4 or IPv6 addresses should be ignored. + +Address and port are as for "or-address" as specified in +section 2.1.1. + +(Only included when the vote or consensus is generated with +consensus-method 14 or later.) + +"s" SP Flags NL + +\[Exactly once.\] + +A series of space-separated status flags, in lexical order (as ASCII +byte strings). Currently documented flags are: + +```text + "Authority" if the router is a directory authority. + "BadExit" if the router is believed to be useless as an exit node + (because its ISP censors it, because it is behind a restrictive + proxy, or for some similar reason). + "Exit" if the router is more useful for building + general-purpose exit circuits than for relay circuits. The + path building algorithm uses this flag; see path-spec.txt. + "Fast" if the router is suitable for high-bandwidth circuits. + "Guard" if the router is suitable for use as an entry guard. + "HSDir" if the router is considered a v2 hidden service directory. + "MiddleOnly" if the router is considered unsuitable for + usage other than as a middle relay. Clients do not need + to handle this option, since when it is present, the authorities + will automatically vote against flags that would make the router + usable in other positions. (Since 0.4.7.2-alpha.) + "NoEdConsensus" if any Ed25519 key in the router's descriptor or + microdescriptor does not reflect authority consensus. + "Stable" if the router is suitable for long-lived circuits. + "StaleDesc" if the router should upload a new descriptor because + the old one is too old. + "Running" if the router is currently usable over all its published + ORPorts. (Authorities ignore IPv6 ORPorts unless configured to + check IPv6 reachability.) Relays without this flag are omitted + from the consensus, and current clients (since 0.2.9.4-alpha) + assume that every listed relay has this flag. + "Valid" if the router has been 'validated'. Clients before + 0.2.9.4-alpha would not use routers without this flag by + default. Currently, relays without this flag are omitted + from the consensus, and current (post-0.2.9.4-alpha) clients + assume that every listed relay has this flag. + "V2Dir" if the router implements the v2 directory protocol or + higher. + + "v" SP version NL + + [At most once.] +``` + +The version of the Tor protocol that this relay is running. If +the value begins with "Tor" SP, the rest of the string is a Tor +version number, and the protocol is "The Tor protocol as supported +by the given version of Tor." Otherwise, if the value begins with +some other string, Tor has upgraded to a more sophisticated +protocol versioning system, and the protocol is "a version of the +Tor protocol more recent than any we recognize." + +Directory authorities SHOULD omit version strings they receive from +descriptors if they would cause "v" lines to be over 128 characters +long. + +"pr" SP Entries NL + +\[At most once.\] + +The "proto" family element as specified in section 2.1.1. + +During voting, authorities copy these lines immediately below the "v" +lines. When a descriptor does not contain a "proto" entry, the +authorities should reconstruct it using the approach described below +in section D. They are included in the consensus using the same rules +as currently used for "v" lines, if a sufficiently late consensus +method is in use. + +"w" SP "Bandwidth=" INT \[SP "Measured=" INT\] \[SP "Unmeasured=1"\] NL + +\[At most once.\] + +An estimate of the bandwidth of this relay, in an arbitrary +unit (currently kilobytes per second). Used to weight router +selection. See section 3.4.2 for details on how the value of +Bandwidth is determined in a consensus. + +Additionally, the Measured= keyword is present in votes by +participating bandwidth measurement authorities to indicate +a measured bandwidth currently produced by measuring stream +capacities. It does not occur in consensuses. + +'Bandwidth=' and 'Measured=' values must be between 0 and +2^32 - 1 inclusive. + +The "Unmeasured=1" value is included in consensuses generated +with method 17 or later when the 'Bandwidth=' value is not +based on a threshold of 3 or more measurements for this relay. + +Other weighting keywords may be added later. +Clients MUST ignore keywords they do not recognize. + +"p" SP ("accept" / "reject") SP PortList NL + +\[At most once.\] + +PortList = PortOrRange +PortList = PortList "," PortOrRange +PortOrRange = INT "-" INT / INT + +A list of those ports that this router supports (if 'accept') +or does not support (if 'reject') for exit to "most +addresses". + +"m" SP methods 1\*(SP algorithm "=" digest) NL + +\[Any number, only in votes.\] + +Microdescriptor hashes for all consensus methods that an authority +supports and that use the same microdescriptor format. "methods" +is a comma-separated list of the consensus methods that the +authority believes will produce "digest". "algorithm" is the name +of the hash algorithm producing "digest", which can be "sha256" or +something else, depending on the consensus "methods" supporting +this algorithm. "digest" is the base64 encoding of the hash of +the router's microdescriptor with trailing =s omitted. + +```text + "id" SP "ed25519" SP ed25519-identity NL + "id" SP "ed25519" SP "none" NL + [vote only, at most once] + + "stats" SP [KeyValues] NL + + [At most once. Vote only] +``` + +KeyValue ::= Keyword '=' Number +Number ::= \[0-9\]+("."\[0-9\]+)? +KeyValues ::= KeyValue | KeyValues SP KeyValue + +Line containing various statistics that an authority has computed for +this relay. Each stats is represented as a key + value. Reported keys +are: + +```text + "wfu" - Weighted Fractional Uptime + "tk" - Weighted Time Known + "mtbf" - Mean Time Between Failure (stability) + + (As of tor-0.4.6.1-alpha) +``` + +The footer section is delineated in all votes and consensuses supporting +consensus method 9 and above with the following: + +"directory-footer" NL +\[No extra arguments\] + +It contains two subsections, a bandwidths-weights line and a +directory-signature. (Prior to consensus method 9, footers only contained +directory-signatures without a 'directory-footer' line or +bandwidth-weights.) + +The bandwidths-weights line appears At Most Once for a consensus. It does +not appear in votes. + +"bandwidth-weights" \[SP Weights\] NL + +Weight ::= Keyword '=' Int32 +Int32 ::= A decimal integer between -2147483648 and 2147483647. +Weights ::= Weight | Weights SP Weight + +List of optional weights to apply to router bandwidths during path +selection. They are sorted in lexical order (as ASCII byte strings) and +values are divided by the consensus' "bwweightscale" param. Definition +of our known entries are... + +```text + Wgg - Weight for Guard-flagged nodes in the guard position + Wgm - Weight for non-flagged nodes in the guard Position + Wgd - Weight for Guard+Exit-flagged nodes in the guard Position + + Wmg - Weight for Guard-flagged nodes in the middle Position + Wmm - Weight for non-flagged nodes in the middle Position + Wme - Weight for Exit-flagged nodes in the middle Position + Wmd - Weight for Guard+Exit flagged nodes in the middle Position + + Weg - Weight for Guard flagged nodes in the exit Position + Wem - Weight for non-flagged nodes in the exit Position + Wee - Weight for Exit-flagged nodes in the exit Position + Wed - Weight for Guard+Exit-flagged nodes in the exit Position + + Wgb - Weight for BEGIN_DIR-supporting Guard-flagged nodes + Wmb - Weight for BEGIN_DIR-supporting non-flagged nodes + Web - Weight for BEGIN_DIR-supporting Exit-flagged nodes + Wdb - Weight for BEGIN_DIR-supporting Guard+Exit-flagged nodes + + Wbg - Weight for Guard flagged nodes for BEGIN_DIR requests + Wbm - Weight for non-flagged nodes for BEGIN_DIR requests + Wbe - Weight for Exit-flagged nodes for BEGIN_DIR requests + Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests + + These values are calculated as specified in section 3.8.3. +``` + +The signature contains the following item, which appears Exactly Once +for a vote, and At Least Once for a consensus. + +```text + "directory-signature" [SP Algorithm] SP identity SP signing-key-digest + NL Signature +``` + +This is a signature of the status document, with the initial item +"network-status-version", and the signature item +"directory-signature", using the signing key. (In this case, we take +the hash through the _space_ after directory-signature, not the +newline: this ensures that all authorities sign the same thing.) +"identity" is the hex-encoded digest of the authority identity key of +the signing authority, and "signing-key-digest" is the hex-encoded +digest of the current authority signing key of the signing authority. + +The Algorithm is one of "sha1" or "sha256" if it is present; +implementations MUST ignore directory-signature entries with an +unrecognized Algorithm. "sha1" is the default, if no Algorithm is +given. The algorithm describes how to compute the hash of the +document before signing it. + +"ns"-flavored consensus documents must contain only sha1 signatures. +Votes and microdescriptor documents may contain other signature +types. Note that only one signature from each authority should be +"counted" as meaning that the authority has signed the consensus. + +(Tor clients before 0.2.3.x did not understand the 'algorithm' +field.) diff --git a/spec/dir-spec/consensus-negotiation-timeline.md b/spec/dir-spec/consensus-negotiation-timeline.md new file mode 100644 index 0000000..59f9ced --- /dev/null +++ b/spec/dir-spec/consensus-negotiation-timeline.md @@ -0,0 +1,16 @@ +<a id="dir-spec.txt-A"></a> + +# Consensus-negotiation timeline + +```text + Period begins: this is the Published time. + Everybody sends votes + Reconciliation: everybody tries to fetch missing votes. + consensus may exist at this point. + End of voting period: + everyone swaps signatures. + Now it's okay for caches to download + Now it's okay for clients to download. + + Valid-after/valid-until switchover +``` diff --git a/spec/dir-spec/converting-to-ed25519.md b/spec/dir-spec/converting-to-ed25519.md new file mode 100644 index 0000000..57f9d1d --- /dev/null +++ b/spec/dir-spec/converting-to-ed25519.md @@ -0,0 +1,77 @@ +<a id="dir-spec.txt-C"></a> + +# Converting a curve25519 public key to an ed25519 public key + +Given an X25519 key, that is, an affine point (u,v) on the +Montgomery curve defined by + +bv^2 = u(u^2 + au +1) + +where + +```text + a = 486662 + b = 1 +``` + +and comprised of the compressed form (i.e. consisting of only the +u-coordinate), we can retrieve the y-coordinate of the affine point +(x,y) on the twisted Edwards form of the curve defined by + +-x^2 + y^2 = 1 + d x^2 y^2 + +where + +d = - 121665/121666 + +by computing + +y = (u-1)/(u+1). + +and then we can apply the usual curve25519 twisted Edwards point +decompression algorithm to find _an_ x-coordinate of an affine +twisted Edwards point to check signatures with. Signing keys for +ed25519 are compressed curve points in twisted Edwards form (so a +y-coordinate and the sign of the x-coordinate), and X25519 keys are +compressed curve points in Montgomery form (i.e. a u-coordinate). + +However, note that compressed point in Montgomery form neglects to +encode what the sign of the corresponding twisted Edwards +x-coordinate would be. Thus, we need the sign of the x-coordinate +to do this operation; otherwise, we'll have two possible +x-coordinates that might have correspond to the ed25519 public key. + +To get the sign, the easiest way is to take the corresponding +private key, feed it to the ed25519 public key generation +algorithm, and see what the sign is. + +\[Recomputing the sign bit from the private key every time sounds +rather strange and inefficient to meโฆ โisis\] + +Note that in addition to its coordinates, an expanded Ed25519 private key +also has a 32-byte random value, "prefix", used to compute internal `r` +values in the signature. For security, this prefix value should be +derived deterministically from the curve25519 key. The Tor +implementation derives it as `SHA512(private_key | STR)[0..32]`, where +STR is the nul-terminated string: + +"Derive high part of ed25519 key from curve25519 key\\0" + +On the client side, where there is no access to the curve25519 private +keys, one may use the curve25519 public key's Montgomery u-coordinate to +recover the Montgomery v-coordinate by computing the right-hand side of +the Montgomery curve equation: + +bv^2 = u(u^2 + au +1) + +where + +```text + a = 486662 + b = 1 +``` + +Then, knowing the intended sign of the Edwards x-coordinate, one +may recover said x-coordinate by computing: + +x = (u/v) * sqrt(-a - 2) diff --git a/spec/dir-spec/creating-key-certificates.md b/spec/dir-spec/creating-key-certificates.md new file mode 100644 index 0000000..fc3326f --- /dev/null +++ b/spec/dir-spec/creating-key-certificates.md @@ -0,0 +1,90 @@ +<a id="dir-spec.txt-3.1"></a> + +# Creating key certificates + +Key certificates consist of the following items: + +"dir-key-certificate-version" version NL + +\[At start, exactly once.\] + +Determines the version of the key certificate. MUST be "3" for +the protocol described in this document. Implementations MUST +reject formats they don't understand. + +```text + "dir-address" IPPort NL + [At most once] + + An IP:Port for this authority's directory port. + + "fingerprint" fingerprint NL + + [Exactly once.] +``` + +Hexadecimal encoding without spaces based on the authority's +identity key. + +"dir-identity-key" NL a public key in PEM format + +\[Exactly once.\] +\[No extra arguments\] + +The long-term authority identity key for this authority. This key +SHOULD be at least 2048 bits long; it MUST NOT be shorter than +1024 bits. + +"dir-key-published" YYYY-MM-DD HH:MM:SS NL + +\[Exactly once.\] + +The time (in UTC) when this document and corresponding key were +last generated. + +Implementations SHOULD reject certificates that are published +too far in the future, though they MAY tolerate some clock skew. + +"dir-key-expires" YYYY-MM-DD HH:MM:SS NL + +\[Exactly once.\] + +A time (in UTC) after which this key is no longer valid. + +Implementations SHOULD reject expired certificates, though they +MAY tolerate some clock skew. + +"dir-signing-key" NL a key in PEM format + +\[Exactly once.\] +\[No extra arguments\] + +The directory server's public signing key. This key MUST be at +least 1024 bits, and MAY be longer. + +"dir-key-crosscert" NL CrossSignature NL + +\[Exactly once.\] +\[No extra arguments\] + +CrossSignature is a signature, made using the certificate's signing +key, of the digest of the PKCS1-padded hash of the certificate's +identity key. For backward compatibility with broken versions of the +parser, we wrap the base64-encoded signature in -----BEGIN ID +SIGNATURE---- and -----END ID SIGNATURE----- tags. Implementations +MUST allow the "ID " portion to be omitted, however. + +Implementations MUST verify that the signature is a correct signature +of the hash of the identity key using the signing key. + +"dir-key-certification" NL Signature NL + +\[At end, exactly once.\] +\[No extra arguments\] + +A document signature as documented in section 1.3, using the +initial item "dir-key-certificate-version" and the final item +"dir-key-certification", signed with the authority identity key. + +Authorities MUST generate a new signing key and corresponding +certificate before the key expires. diff --git a/spec/dir-spec/directory-authority-operation-formats.md b/spec/dir-spec/directory-authority-operation-formats.md new file mode 100644 index 0000000..bd487f6 --- /dev/null +++ b/spec/dir-spec/directory-authority-operation-formats.md @@ -0,0 +1,10 @@ +<a id="dir-spec.txt-3"></a> + +# Directory authority operation and formats + +Every authority has two keys used in this protocol: a signing key, and +an authority identity key. (Authorities also have a router identity +key used in their role as a router and by earlier versions of the +directory protocol.) The identity key is used from time to time to +sign new key certificates using new signing keys; it is very sensitive. +The signing key is used to sign key certificates and status documents. diff --git a/spec/dir-spec/directory-cache-operation.md b/spec/dir-spec/directory-cache-operation.md new file mode 100644 index 0000000..14f9a9c --- /dev/null +++ b/spec/dir-spec/directory-cache-operation.md @@ -0,0 +1,167 @@ +<a id="dir-spec.txt-4"></a> + +# Directory cache operation + +All directory caches implement this section, except as noted. + +<a id="dir-spec.txt-4.1"></a> + +## Downloading consensus status documents from directory authorities { #download-ns-from-auth } + +All directory caches try to keep a recent +network-status consensus document to serve to clients. A cache ALWAYS +downloads a network-status consensus if any of the following are true: + +- The cache has no consensus document. +- The cache's consensus document is no longer valid. + +Otherwise, the cache downloads a new consensus document at a randomly +chosen time in the first half-interval after its current consensus +stops being fresh. (This time is chosen at random to avoid swarming +the authorities at the start of each period. The interval size is +inferred from the difference between the valid-after time and the +fresh-until time on the consensus.) + +```text + [For example, if a cache has a consensus that became valid at 1:00, + and is fresh until 2:00, that cache will fetch a new consensus at + a random time between 2:00 and 2:30.] +``` + +Directory caches also fetch consensus flavors from the authorities. +Caches check the correctness of consensus flavors, but do not check +anything about an unrecognized consensus document beyond its digest and +length. Caches serve all consensus flavors from the same locations as +the directory authorities. + +<a id="dir-spec.txt-4.2"></a> + +## Downloading server descriptors from directory authorities { #download-desc-from-auth } + +Periodically (currently, every 10 seconds), directory caches check +whether there are any specific descriptors that they do not have and that +they are not currently trying to download. Caches identify these +descriptors by hash in the recent network-status consensus documents. + +If so, the directory cache launches requests to the authorities for these +descriptors. + +If one of these downloads fails, we do not try to download that descriptor +from the authority that failed to serve it again unless we receive a newer +network-status consensus that lists the same descriptor. + +Directory caches must potentially cache multiple descriptors for each +router. Caches must not discard any descriptor listed by any recent +consensus. If there is enough space to store additional descriptors, +caches SHOULD try to hold those which clients are likely to download the +most. (Currently, this is judged based on the interval for which each +descriptor seemed newest.) + +\[XXXX define recent\] + +<a id="dir-spec.txt-4.3"></a> + +## Downloading microdescriptors from directory authorities { #download-md-from-auth } + +Directory mirrors should fetch, cache, and serve each microdescriptor +from the authorities. + +The microdescriptors with base64 hashes `<D1>`, `<D2>`, `<D3>` are available +at: + +`http://<hostname>/tor/micro/d/<D1>-<D2>-<D3>[.z]` + +`<Dn>` are base64 encoded with trailing =s omitted for size and for +consistency with the microdescriptor consensus format. -s are used +instead of +s to separate items, since the + character is used in +base64 encoding. + +Directory mirrors should check to make sure that the microdescriptors +they're about to serve match the right hashes (either the hashes from +the fetch URL or the hashes from the consensus, respectively). + +(NOTE: Due to squid proxy url limitations at most 92 microdescriptor hashes +can be retrieved in a single request.) + +<a id="dir-spec.txt-4.4"></a> + +## Downloading extra-info documents from directory authorities { #download-ei-from-auth } + +Any cache that chooses to cache extra-info documents should implement this +section. + +Periodically, the Tor instance checks whether it is missing any extra-info +documents: in other words, if it has any server descriptors with an +extra-info-digest field that does not match any of the extra-info +documents currently held. If so, it downloads whatever extra-info +documents are missing. Caches download from authorities. We follow the +same splitting and back-off rules as in section 4.2. + +<a id="dir-spec.txt-4.5"></a> + +## Consensus diffs { #diffs } + +Instead of downloading an entire consensus, clients may download +a "diff" document containing an ed-style diff from a previous +consensus document. Caches (and authorities) make these diffs as +they learn about new consensuses. To do so, they must store a +record of older consensuses. + +(Support for consensus diffs was added in 0.3.1.1-alpha, and is +advertised with the DirCache protocol version "2" or later.) + +<a id="dir-spec.txt-4.5.1"></a> + +### Consensus diff format { #diff-format } + +Consensus diffs are formatted as follows: + +The first line is "network-status-diff-version 1" NL + +The second line is + +"hash" SP FromDigest SP ToDigest NL + +where FromDigest is the hex-encoded SHA3-256 digest of the _signed +part_ of the consensus that the diff should be applied to, and +ToDigest is the hex-encoded SHA3-256 digest of the _entire_ +consensus resulting from applying the diff. (See 3.4.1 for +information on that part of a consensus is signed.) + +The third and subsequent lines encode the diff from FromDigest to +ToDigest in a limited subset of the ed diff format, as specified +in appendix E. + +<a id="dir-spec.txt-4.5.2"></a> + +### Serving and requesting diffs { #diff-requests } + +When downloading the current consensus, a client may include an +HTTP header of the form + +X-Or-Diff-From-Consensus: HASH1, HASH2, ... + +where the HASH values are hex-encoded SHA3-256 digests of the +_signed part_ of one or more consensuses that the client knows +about. + +If a cache knows a consensus diff from one of those consensuses +to the most recent consensus of the requested flavor, it may +send that diff instead of the specified consensus. + +Caches also serve diffs from the URIs: + +```text +/tor/status-vote/current/consensus/diff/<HASH>/<FPRLIST>.z +/tor/status-vote/current/consensus-<FLAVOR>/diff/<HASH>/<FPRLIST>.z +``` + +where FLAVOR is the consensus flavor, defaulting to "ns", and +FPRLIST is +-separated list of recognized authority identity +fingerprints as in appendix B. + +<a id="dir-spec.txt-4.6"></a> + +## Retrying failed downloads { #retry-as-cache } + +See section 5.5 below; it applies to caches as well as clients. diff --git a/spec/dir-spec/downloading-from-other-auths.md b/spec/dir-spec/downloading-from-other-auths.md new file mode 100644 index 0000000..8f474e9 --- /dev/null +++ b/spec/dir-spec/downloading-from-other-auths.md @@ -0,0 +1,54 @@ +# Downloading information from other directory authorities + +## Downloading missing certificates from other directory authorities { #download-missing-certs } + +XXX when to download certificates. + +<a id="dir-spec.txt-3.6"></a> + +## Downloading server descriptors from other directory authorities { #download-missing-descs } + +Periodically (currently, every 10 seconds), directory authorities check +whether there are any specific descriptors that they do not have and that +they are not currently trying to download. +Authorities identify them by hash in vote (if publication date is more +recent than the descriptor we currently have). + +\[XXXX need a way to fetch descriptors ahead of the vote? v2 status docs can +do that for now.\] + +If so, the directory authority launches requests to the authorities for these +descriptors, such that each authority is only asked for descriptors listed +in its most recent vote. If more +than one authority lists the descriptor, we choose which to ask at random. + +If one of these downloads fails, we do not try to download that descriptor +from the authority that failed to serve it again unless we receive a newer +network-status (consensus or vote) from that authority that lists the same +descriptor. + +```text + Directory authorities must potentially cache multiple descriptors for each + router. Authorities must not discard any descriptor listed by any recent + consensus. If there is enough space to store additional descriptors, + authorities SHOULD try to hold those which clients are likely to download the + most. (Currently, this is judged based on the interval for which each + descriptor seemed newest.) +[XXXX define recent] +``` + +Authorities SHOULD NOT download descriptors for routers that they would +immediately reject for reasons listed in section 3.2. + +<a id="dir-spec.txt-3.7"></a> + +## Downloading extra-info documents from other directory authorities { #download-missing-extrainfo } + +Periodically, an authority checks whether it is missing any extra-info +documents: in other words, if it has any server descriptors with an +extra-info-digest field that does not match any of the extra-info +documents currently held. If so, it downloads whatever extra-info +documents are missing. We follow the same splitting and back-off rules +as in section 3.6. + +<a id="dir-spec.txt-3.8"></a> diff --git a/spec/dir-spec/exchanging-detached-signatures.md b/spec/dir-spec/exchanging-detached-signatures.md new file mode 100644 index 0000000..7a8495e --- /dev/null +++ b/spec/dir-spec/exchanging-detached-signatures.md @@ -0,0 +1,77 @@ +# Exchanging detached signatures + +Once an authority has computed and signed a consensus network status, it +should send its detached signature to each other authority in an HTTP POST +request to the URL: + +`http://<hostname>/tor/post/consensus-signature` + +\[XXX Note why we support push-and-then-pull.\] + +All of the detached signatures it knows for consensus status should be +available at: + +`http://<hostname>/tor/status-vote/next/consensus-signatures.z` + +Assuming full connectivity, every authority should compute and sign the +same consensus including any flavors in each period. Therefore, it +isn't necessary to download the consensus or any flavors of it computed +by each authority; instead, the authorities only push/fetch each +others' signatures. A "detached signature" document contains items as +follows: + +"consensus-digest" SP Digest NL + +\[At start, at most once.\] + +The digest of the consensus being signed. + +"valid-after" SP YYYY-MM-DD SP HH:MM:SS NL +"fresh-until" SP YYYY-MM-DD SP HH:MM:SS NL +"valid-until" SP YYYY-MM-DD SP HH:MM:SS NL + +\[As in the consensus\] + +"additional-digest" SP flavor SP algname SP digest NL + +\[Any number.\] + +For each supported consensus flavor, every directory authority +adds one or more "additional-digest" lines. "flavor" is the name +of the consensus flavor, "algname" is the name of the hash +algorithm that is used to generate the digest, and "digest" is the +hex-encoded digest. + +The hash algorithm for the microdescriptor consensus flavor is +defined as SHA256 with algname "sha256". + +```text + "additional-signature" SP flavor SP algname SP identity SP + signing-key-digest NL signature. + + [Any number.] +``` + +For each supported consensus flavor and defined digest algorithm, +every directory authority adds an "additional-signature" line. +"flavor" is the name of the consensus flavor. "algname" is the +name of the algorithm that was used to hash the identity and +signing keys, and to compute the signature. "identity" is the +hex-encoded digest of the authority identity key of the signing +authority, and "signing-key-digest" is the hex-encoded digest of +the current authority signing key of the signing authority. + +The "sha256" signature format is defined as the RSA signature of +the OAEP+-padded SHA256 digest of the item to be signed. When +checking signatures, the signature MUST be treated as valid if the +signature material begins with SHA256(document), so that other +data can get added later. +\[To be honest, I didn't fully understand the previous paragraph +and only copied it from the proposals. Review carefully. -KL\] + +"directory-signature" + +\[As in the consensus; the signature object is the same as in the +consensus document.\] + +<a id="dir-spec.txt-3.11"></a> diff --git a/spec/dir-spec/exchanging-votes.md b/spec/dir-spec/exchanging-votes.md new file mode 100644 index 0000000..924aa67 --- /dev/null +++ b/spec/dir-spec/exchanging-votes.md @@ -0,0 +1,53 @@ +<a id="dir-spec.txt-3.4"></a> + +# Exchanging votes + +Authorities divide time into Intervals. Authority administrators SHOULD +try to all pick the same interval length, and SHOULD pick intervals that +are commonly used divisions of time (e.g., 5 minutes, 15 minutes, 30 +minutes, 60 minutes, 90 minutes). Voting intervals SHOULD be chosen to +divide evenly into a 24-hour day. + +Authorities SHOULD act according to interval and delays in the +latest consensus. Lacking a latest consensus, they SHOULD default to a +30-minute Interval, a 5 minute VotingDelay, and a 5 minute DistDelay. + +Authorities MUST take pains to ensure that their clocks remain accurate +within a few seconds. (Running NTP is usually sufficient.) + +The first voting period of each day begins at 00:00 (midnight) UTC. If +the last period of the day would be truncated by one-half or more, it is +merged with the second-to-last period. + +An authority SHOULD publish its vote immediately at the start of each voting +period (minus VoteSeconds+DistSeconds). It does this by making it +available at + +`http://<hostname>/tor/status-vote/next/authority.z` + +and sending it in an HTTP POST request to each other authority at the URL + +`http://<hostname>/tor/post/vote` + +If, at the start of the voting period, minus DistSeconds, an authority +does not have a current statement from another authority, the first +authority downloads the other's statement. + +Once an authority has a vote from another authority, it makes it available +at + +`http://<hostname>/tor/status-vote/next/<fp>.z` + +where `<fp>` is the fingerprint of the other authority's identity key. +And at + +`http://<hostname>/tor/status-vote/next/d/<d>.z` + +where `<d>` is the digest of the vote document. + +Also, once an authority receives a vote from another authority, it +examines it for new descriptors and fetches them from that authority. +This may be the only way for an authority to hear about relays that didn't +publish their descriptor to all authorities, and, while it's too late +for the authority to include relays in its current vote, it can include +them in its next vote. See section 3.6 below for details. diff --git a/spec/dir-spec/extra-info-document-format.md b/spec/dir-spec/extra-info-document-format.md new file mode 100644 index 0000000..0defe70 --- /dev/null +++ b/spec/dir-spec/extra-info-document-format.md @@ -0,0 +1,591 @@ +<a id="dir-spec.txt-2.1.2"></a> + +# Extra-info document format + +Extra-info documents consist of the following items: + +```text + "extra-info" Nickname Fingerprint NL + [At start, exactly once.] +``` + +Identifies what router this is an extra-info descriptor for. +Fingerprint is encoded in hex (using upper-case letters), with +no spaces. + +```text + "identity-ed25519" + [As in router descriptors] + + "published" YYYY-MM-DD HH:MM:SS NL + + [Exactly once.] +``` + +The time, in UTC, when this document (and its corresponding router +descriptor if any) was generated. It MUST match the published time +in the corresponding server descriptor. + +```text + "read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once.] + "write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once.] +``` + +Declare how much bandwidth the OR has used recently. Usage is divided +into intervals of NSEC seconds. The YYYY-MM-DD HH:MM:SS field +defines the end of the most recent interval. The numbers are the +number of bytes used in the most recent intervals, ordered from +oldest to newest. + +These fields include both IPv4 and IPv6 traffic. + +```text + "ipv6-read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + "ipv6-write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] +``` + +Declare how much bandwidth the OR has used recently, on IPv6 +connections. See "read-history" and "write-history" for full details. + +```text + "geoip-db-digest" Digest NL + [At most once.] +``` + +SHA1 digest of the IPv4 GeoIP database file that is used to +resolve IPv4 addresses to country codes. + +```text + "geoip6-db-digest" Digest NL + [At most once.] +``` + +SHA1 digest of the IPv6 GeoIP database file that is used to +resolve IPv6 addresses to country codes. + +("geoip-start-time" YYYY-MM-DD HH:MM:SS NL) +("geoip-client-origins" CC=NUM,CC=NUM,... NL) + +Only generated by bridge routers (see blocking.pdf), and only +when they have been configured with a geoip database. +Non-bridges SHOULD NOT generate these fields. Contains a list +of mappings from two-letter country codes (CC) to the number +of clients that have connected to that bridge from that +country (approximate, and rounded up to the nearest multiple of 8 +in order to hamper traffic analysis). A country is included +only if it has at least one address. The time in +"geoip-start-time" is the time at which we began collecting geoip +statistics. + +"geoip-start-time" and "geoip-client-origins" have been replaced by +"bridge-stats-end" and "bridge-ips" in 0.2.2.4-alpha. The +reason is that the measurement interval with "geoip-stats" as +determined by subtracting "geoip-start-time" from "published" could +have had a variable length, whereas the measurement interval in +0.2.2.4-alpha and later is set to be exactly 24 hours long. In +order to clearly distinguish the new measurement intervals from +the old ones, the new keywords have been introduced. + +```text + "bridge-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "bridge-stats-end" line, as well as any other "bridge-\*" line, +is only added when the relay has been running as a bridge for at +least 24 hours. + +```text + "bridge-ips" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +unique IP addresses that have connected from that country to the +bridge and which are no known relays, rounded up to the nearest +multiple of 8. + +```text + "bridge-ip-versions" FAM=NUM,FAM=NUM,... NL + [At most once.] +``` + +List of unique IP addresses that have connected to the bridge +per protocol family. + +```text + "bridge-ip-transports" PT=NUM,PT=NUM,... NL + [At most once.] +``` + +List of mappings from pluggable transport names to the number +of unique IP addresses that have connected using that +pluggable transport. Unobfuscated connections are counted +using the reserved pluggable transport name "`<OR>`" (without +quotes). If we received a connection from a transport proxy +but we couldn't figure out the name of the pluggable +transport, we use the reserved pluggable transport name +"`<??>`". + +("`<OR>`" and "`<??>`" are reserved because normal pluggable +transport names MUST match the following regular expression: +"`[a-zA-Z_][a-zA-Z0-9_]*`" ) + +The pluggable transport name list is sorted into lexically +ascending order. + +If no clients have connected to the bridge yet, we only write +"bridge-ip-transports" to the stats file. + +```text + "dirreq-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "dirreq-stats-end" line, as well as any other "dirreq-\*" line, +is only added when the relay has opened its Dir port and after 24 +hours of measuring directory requests. + +```text + "dirreq-v2-ips" CC=NUM,CC=NUM,... NL + [At most once.] + "dirreq-v3-ips" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +unique IP addresses that have connected from that country to +request a v2/v3 network status, rounded up to the nearest multiple +of 8. Only those IP addresses are counted that the directory can +answer with a 200 OK status code. (Note here and below: current Tor +versions, as of 0.2.5.2-alpha, no longer cache or serve v2 +networkstatus documents.) + +```text + "dirreq-v2-reqs" CC=NUM,CC=NUM,... NL + [At most once.] + "dirreq-v3-reqs" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +requests for v2/v3 network statuses from that country, rounded up +to the nearest multiple of 8. Only those requests are counted that +the directory can answer with a 200 OK status code. + +```text + "dirreq-v2-share" NUM% NL + [At most once.] + "dirreq-v3-share" NUM% NL + [At most once.] +``` + +The share of v2/v3 network status requests that the directory +expects to receive from clients based on its advertised bandwidth +compared to the overall network bandwidth capacity. Shares are +formatted in percent with two decimal places. Shares are +calculated as means over the whole 24-hour interval. + +```text + "dirreq-v2-resp" status=NUM,... NL + [At most once.] + "dirreq-v3-resp" status=NUM,... NL + [At most once.] +``` + +List of mappings from response statuses to the number of requests +for v2/v3 network statuses that were answered with that response +status, rounded up to the nearest multiple of 4. Only response +statuses with at least 1 response are reported. New response +statuses can be added at any time. The current list of response +statuses is as follows: + +```text + "ok": a network status request is answered; this number + corresponds to the sum of all requests as reported in + "dirreq-v2-reqs" or "dirreq-v3-reqs", respectively, before + rounding up. + "not-enough-sigs: a version 3 network status is not signed by a + sufficient number of requested authorities. + "unavailable": a requested network status object is unavailable. + "not-found": a requested network status is not found. + "not-modified": a network status has not been modified since the + If-Modified-Since time that is included in the request. + "busy": the directory is busy. + + "dirreq-v2-direct-dl" key=NUM,... NL + [At most once.] + "dirreq-v3-direct-dl" key=NUM,... NL + [At most once.] + "dirreq-v2-tunneled-dl" key=NUM,... NL + [At most once.] + "dirreq-v3-tunneled-dl" key=NUM,... NL + [At most once.] +``` + +List of statistics about possible failures in the download process +of v2/v3 network statuses. Requests are either "direct" +HTTP-encoded requests over the relay's directory port, or +"tunneled" requests using a BEGIN_DIR relay message over the relay's OR +port. The list of possible statistics can change, and statistics +can be left out from reporting. The current list of statistics is +as follows: + +Successful downloads and failures: + +```text + "complete": a client has finished the download successfully. + "timeout": a download did not finish within 10 minutes after + starting to send the response. + "running": a download is still running at the end of the + measurement period for less than 10 minutes after starting to + send the response. + + Download times: + + "min", "max": smallest and largest measured bandwidth in B/s. + "d[1-4,6-9]": 1st to 4th and 6th to 9th decile of measured + bandwidth in B/s. For a given decile i, i/10 of all downloads + had a smaller bandwidth than di, and (10-i)/10 of all downloads + had a larger bandwidth than di. + "q[1,3]": 1st and 3rd quartile of measured bandwidth in B/s. One + fourth of all downloads had a smaller bandwidth than q1, one + fourth of all downloads had a larger bandwidth than q3, and the + remaining half of all downloads had a bandwidth between q1 and + q3. + "md": median of measured bandwidth in B/s. Half of the downloads + had a smaller bandwidth than md, the other half had a larger + bandwidth than md. + + "dirreq-read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + "dirreq-write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] +``` + +Declare how much bandwidth the OR has spent on answering directory +requests. Usage is divided into intervals of NSEC seconds. The +YYYY-MM-DD HH:MM:SS field defines the end of the most recent +interval. The numbers are the number of bytes used in the most +recent intervals, ordered from oldest to newest. + +```text + "entry-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +An "entry-stats-end" line, as well as any other "entry-\*" +line, is first added after the relay has been running for at least +24 hours. + +```text + "entry-ips" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +unique IP addresses that have connected from that country to the +relay and which are no known other relays, rounded up to the +nearest multiple of 8. + +```text + "cell-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "cell-stats-end" line, as well as any other "cell-\*" line, +is first added after the relay has been running for at least 24 +hours. + +```text + "cell-processed-cells" NUM,...,NUM NL + [At most once.] +``` + +Mean number of processed cells per circuit, subdivided into +deciles of circuits by the number of cells they have processed in +descending order from loudest to quietest circuits. + +```text + "cell-queued-cells" NUM,...,NUM NL + [At most once.] +``` + +Mean number of cells contained in queues by circuit decile. These +means are calculated by 1) determining the mean number of cells in +a single circuit between its creation and its termination and 2) +calculating the mean for all circuits in a given decile as +determined in "cell-processed-cells". Numbers have a precision of +two decimal places. + +Note that this statistic can be inaccurate for circuits that had +queued cells at the start or end of the measurement interval. + +```text + "cell-time-in-queue" NUM,...,NUM NL + [At most once.] +``` + +Mean time cells spend in circuit queues in milliseconds. Times are +calculated by 1) determining the mean time cells spend in the +queue of a single circuit and 2) calculating the mean for all +circuits in a given decile as determined in +"cell-processed-cells". + +Note that this statistic can be inaccurate for circuits that had +queued cells at the start or end of the measurement interval. + +```text + "cell-circuits-per-decile" NUM NL + [At most once.] +``` + +Mean number of circuits that are included in any of the deciles, +rounded up to the next integer. + +```text + "conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL + [At most once] +``` + +Number of connections, split into 10-second intervals, that are +used uni-directionally or bi-directionally as observed in the NSEC +seconds (usually 86400 seconds) before YYYY-MM-DD HH:MM:SS. Every +10 seconds, we determine for every connection whether we read and +wrote less than a threshold of 20 KiB (BELOW), read at least 10 +times more than we wrote (READ), wrote at least 10 times more than +we read (WRITE), or read and wrote more than the threshold, but +not 10 times more in either direction (BOTH). After classifying a +connection, read and write counters are reset for the next +10-second interval. + +This measurement includes both IPv4 and IPv6 connections. + +```text + "ipv6-conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL + [At most once] +``` + +Number of IPv6 connections that are used uni-directionally or +bi-directionally. See "conn-bi-direct" for more details. + +```text + "exit-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +An "exit-stats-end" line, as well as any other "exit-\*" line, is +first added after the relay has been running for at least 24 hours +and only if the relay permits exiting (where exiting to a single +port and IP address is sufficient). + +```text + "exit-kibibytes-written" port=N,port=N,... NL + [At most once.] + "exit-kibibytes-read" port=N,port=N,... NL + [At most once.] +``` + +List of mappings from ports to the number of kibibytes that the +relay has written to or read from exit connections to that port, +rounded up to the next full kibibyte. Relays may limit the +number of listed ports and subsume any remaining kibibytes under +port "other". + +```text + "exit-streams-opened" port=N,port=N,... NL + [At most once.] +``` + +List of mappings from ports to the number of opened exit streams +to that port, rounded up to the nearest multiple of 4. Relays may +limit the number of listed ports and subsume any remaining opened +streams under port "other". + +```text + "hidserv-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] + "hidserv-v3-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "hidserv-stats-end" line, as well as any other "hidserv-\*" line, +is first added after the relay has been running for at least 24 +hours. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "hidserv-rend-relayed-cells" SP NUM SP key=val SP key=val ... NL + [At most once.] + "hidserv-rend-v3-relayed-cells" SP NUM SP key=val SP key=val ... NL + [At most once.] +``` + +Approximate number of relay cells seen in either direction on a +circuit after receiving and successfully processing a RENDEZVOUS1 +cell. + +The original measurement value is obfuscated in several steps: +first, it is rounded up to the nearest multiple of 'bin_size' +which is reported in the key=val part of this line; second, a +(possibly negative) noise value is added to the result of the +first step by randomly sampling from a Laplace distribution with +mu = 0 and b = (delta_f / epsilon) with 'delta_f' and 'epsilon' +being reported in the key=val part, too; third, the result of the +previous obfuscation steps is truncated to the next smaller +integer and included as 'NUM'. Note that the overall reported +value can be negative. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "hidserv-dir-onions-seen" SP NUM SP key=val SP key=val ... NL + [At most once.] + "hidserv-dir-v3-onions-seen" SP NUM SP key=val SP key=val ... NL + [At most once.] +``` + +Approximate number of unique hidden-service identities seen in +descriptors published to and accepted by this hidden-service +directory. + +The original measurement value is obfuscated in the same way as +the 'NUM' value reported in "hidserv-rend-relayed-cells", but +possibly with different parameters as reported in the key=val part +of this line. Note that the overall reported value can be +negative. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "transport" transportname address:port [arglist] NL + [Any number.] +``` + +Signals that the router supports the 'transportname' pluggable +transport in IP address 'address' and TCP port 'port'. A single +descriptor MUST not have more than one transport line with the +same 'transportname'. + +Pluggable transports are only relevant to bridges, but these entries +can appear in non-bridge relays as well. + +```text + "padding-counts" YYYY-MM-DD HH:MM:SS (NSEC s) key=NUM key=NUM ... NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). Counts +are reset to 0 at the end of this interval. + +The keyword list is currently as follows: + +```text + bin-size + - The current rounding value for cell count fields (10000 by + default) + write-drop + - The number of RELAY_DROP messages this relay sent + write-pad + - The number of PADDING cells this relay sent + write-total + - The total number of cells this relay cent + read-drop + - The number of RELAY_DROP messages this relay received + read-pad + - The number of PADDING cells this relay received + read-total + - The total number of cells this relay received + enabled-read-pad + - The number of PADDING cells this relay received on + connections that support padding + enabled-read-total + - The total number of cells this relay received on connections + that support padding + enabled-write-pad + - The total number of cells this relay received on connections + that support padding + enabled-write-total + - The total number of cells sent by this relay on connections + that support padding + max-chanpad-timers + - The maximum number of timers that this relay scheduled for + padding in the previous NSEC interval + + "overload-ratelimits" SP version SP YYYY-MM-DD SP HH:MM:SS + SP rate-limit SP burst-limit + SP read-overload-count SP write-overload-count NL + [At most once.] + + Indicates that a bandwidth limit was exhausted for this relay. +``` + +The "rate-limit" and "burst-limit" are the raw values from the +BandwidthRate and BandwidthBurst found in the torrc configuration file. + +The "{read|write}-overload-count" are the counts of how many times the +reported limits of burst/rate were exhausted and thus the maximum +between the read and write count occurrences. To make the counter more +meaningful and to avoid multiple connections saturating the counter +when a relay is overloaded, we only increment it once a minute. + +The 'version' field is set to '1' for now. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "overload-fd-exhausted" SP version YYYY-MM-DD HH:MM:SS NL + [At most once.] +``` + +Indicates that a file descriptor exhaustion was experienced by this +relay. + +The timestamp indicates that the maximum was reached between the +timestamp and the "published" timestamp of the document. + +This overload field should remain in place for 72 hours since last +triggered. If the limits are reached again in this period, the +timestamp is updated, and this 72 hour period restarts. + +The 'version' field is set to '1' for the initial implementation which +detects fd exhaustion only when a socket open fails. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "router-sig-ed25519" + [As in router descriptors] + + "router-signature" NL Signature NL + [At end, exactly once.] + [No extra arguments] +``` + +A document signature as documented in section 1.3, using the +initial item "extra-info" and the final item "router-signature", +signed with the router's identity key. diff --git a/spec/dir-spec/general-use-http-urls.md b/spec/dir-spec/general-use-http-urls.md new file mode 100644 index 0000000..98ea7a5 --- /dev/null +++ b/spec/dir-spec/general-use-http-urls.md @@ -0,0 +1,133 @@ +<a id="dir-spec.txt-B"></a> + +# General-use HTTP URLs + +"Fingerprints" in these URLs are base16-encoded SHA1 hashes. + +The most recent v3 consensus should be available at: + +`http://<hostname>/tor/status-vote/current/consensus.z` + +Similarly, the v3 microdescriptor consensus should be available at: + +`http://<hostname>/tor/status-vote/current/consensus-microdesc.z` + +Starting with Tor version 0.2.1.1-alpha is also available at: + +`http://<hostname>/tor/status-vote/current/consensus/<F1>+<F2>+<F3>.z` + +(NOTE: Due to squid proxy url limitations at most 96 fingerprints can be +retrieved in a single request.) + +Where F1, F2, etc. are authority identity fingerprints the client trusts. +Servers will only return a consensus if more than half of the requested +authorities have signed the document, otherwise a 404 error will be sent +back. The fingerprints can be shortened to a length of any multiple of +two, using only the leftmost part of the encoded fingerprint. Tor uses +3 bytes (6 hex characters) of the fingerprint. + +Clients SHOULD sort the fingerprints in ascending order. Server MUST +accept any order. + +Clients SHOULD use this format when requesting consensus documents from +directory authority servers and from caches running a version of Tor +that is known to support this URL format. + +A concatenated set of all the current key certificates should be available +at: + +`http://<hostname>/tor/keys/all.z` + +The key certificate for this server should be available at: + +`http://<hostname>/tor/keys/authority.z` + +The key certificate for an authority whose authority identity fingerprint +is `<F>` should be available at: + +`http://<hostname>/tor/keys/fp/<F>.z` + +The key certificate whose signing key fingerprint is `<F>` should be +available at: + +`http://<hostname>/tor/keys/sk/<F>.z` + +The key certificate whose identity key fingerprint is `<F>` and whose signing +key fingerprint is `<S>` should be available at: + +`http://<hostname>/tor/keys/fp-sk/<F>-<S>.z` + +(As usual, clients may request multiple certificates using: + +`http://<hostname>/tor/keys/fp-sk/<F1>-<S1>+<F2>-<S2>.z` ) + +\[The above fp-sk format was not supported before Tor 0.2.1.9-alpha.\] + +The most recent descriptor for a server whose identity key has a +fingerprint of `<F>` should be available at: + +`http://<hostname>/tor/server/fp/<F>.z` + +The most recent descriptors for servers with identity fingerprints +`<F1>`, `<F2>`,`<F3>` should be available at: + +`http://<hostname>/tor/server/fp/<F1>+<F2>+<F3>.z` + +(NOTE: Due to squid proxy url limitations at most 96 fingerprints can be +retrieved in a single request. + +Implementations SHOULD NOT download descriptors by identity key +fingerprint. This allows a corrupted server (in collusion with a cache) to +provide a unique descriptor to a client, and thereby partition that client +from the rest of the network.) + +The server descriptor with (descriptor) digest `<D>` (in hex) should be +available at: + +`http://<hostname>/tor/server/d/<D>.z` + +The most recent descriptors with digests `<D1>`, `<D2>`, `<D3>` should be +available at: + +`http://<hostname>/tor/server/d/<D1>+<D2>+<D3>.z` + +The most recent descriptor for this server should be at: + +`http://<hostname>/tor/server/authority.z` + +This is used for authorities, and also if a server is configured +as a bridge. The official Tor implementations (starting at +0.1.1.x) use this resource to test whether a server's own DirPort +is reachable. It is also useful for debugging purposes. + +A concatenated set of the most recent descriptors for all known servers +should be available at: + +`http://<hostname>/tor/server/all.z` + +Extra-info documents are available at the URLS: + +```text + http://<hostname>/tor/extra/d/... + http://<hostname>/tor/extra/fp/... + http://<hostname>/tor/extra/all[.z] + http://<hostname>/tor/extra/authority[.z] +``` + +(These work like the `/tor/server/` URLs: they support fetching extra-info +documents by their digest, by the fingerprint of their servers, +or all at once. When serving by fingerprint, we serve the +extra-info that corresponds to the descriptor we would serve by +that fingerprint. Only directory authorities of version +0.2.0.1-alpha or later are guaranteed to support the first +three classes of URLs. Caches may support them, and MUST +support them if they have advertised "caches-extra-info".) + +For debugging, directories SHOULD expose non-compressed objects at +URLs like the above, but without the final ".z". If the client uses +Accept-Encodings header, it should override the presence or absence +of the ".z" (see section 6.1). + +Clients SHOULD use upper case letters (A-F) when base16-encoding +fingerprints. Servers MUST accept both upper and lower case fingerprints +in requests. diff --git a/spec/dir-spec/index.md b/spec/dir-spec/index.md new file mode 100644 index 0000000..626ded0 --- /dev/null +++ b/spec/dir-spec/index.md @@ -0,0 +1,119 @@ +# Tor directory protocol, version 3 + +<a id="dir-spec.txt-0"></a> + +This directory protocol is used by Tor version 0.2.0.x-alpha and later. +See dir-spec-v1.txt for information on the protocol used up to the +0.1.0.x series, and dir-spec-v2.txt for information on the protocol +used by the 0.1.1.x and 0.1.2.x series. + +This document merges and supersedes the following proposals: + +- 101 Voting on the Tor Directory System +- 103 Splitting identity key from regularly used signing key +- 104 Long and Short Router Descriptors + +XXX timeline +XXX fill in XXXXs + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL +NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in +RFC 2119. + +<a id="dir-spec.txt-0.1"></a> + +## History + +The earliest versions of Onion Routing shipped with a list of known +routers and their keys. When the set of routers changed, users needed to +fetch a new list. + +### The Version 1 Directory protocol { #v1-protocol } + +Early versions of Tor (0.0.2) introduced "Directory authorities": servers +that served signed "directory" documents containing a list of signed +"server descriptors", along with short summary of the status of each +router. Thus, clients could get up-to-date information on the state of +the network automatically, and be certain that the list they were getting +was attested by a trusted directory authority. + +Later versions (0.0.8) added directory caches, which download +directories from the authorities and serve them to clients. Non-caches +fetch from the caches in preference to fetching from the authorities, thus +distributing bandwidth requirements. + +Also added during the version 1 directory protocol were "router status" +documents: short documents that listed only the up/down status of the +routers on the network, rather than a complete list of all the +descriptors. Clients and caches would fetch these documents far more +frequently than they would fetch full directories. + +### The Version 2 Directory Protocol { #v2-protocol } + +During the Tor 0.1.1.x series, Tor revised its handling of directory +documents in order to address two major problems: + +```text + * Directories had grown quite large (over 1MB), and most directory + downloads consisted mainly of server descriptors that clients + already had. + + * Every directory authority was a trust bottleneck: if a single + directory authority lied, it could make clients believe for a time + an arbitrarily distorted view of the Tor network. (Clients + trusted the most recent signed document they downloaded.) Thus, + adding more authorities would make the system less secure, not + more. +``` + +To address these, we extended the directory protocol so that +authorities now published signed "network status" documents. Each +network status listed, for every router in the network: a hash of its +identity key, a hash of its most recent descriptor, and a summary of +what the authority believed about its status. Clients would download +the authorities' network status documents in turn, and believe +statements about routers iff they were attested to by more than half of +the authorities. + +Instead of downloading all server descriptors at once, clients +downloaded only the descriptors that they did not have. Descriptors +were indexed by their digests, in order to prevent malicious caches +from giving different versions of a server descriptor to different +clients. + +Routers began working harder to upload new descriptors only when their +contents were substantially changed. + +<a id="dir-spec.txt-0.2"></a> + +### Goals of the version 3 protocol { #v3-goals } + +Version 3 of the Tor directory protocol tries to solve the following +issues: + +```text + * A great deal of bandwidth used to transmit server descriptors was + used by two fields that are not actually used by Tor routers + (namely read-history and write-history). We save about 60% by + moving them into a separate document that most clients do not + fetch or use. + + * It was possible under certain perverse circumstances for clients + to download an unusual set of network status documents, thus + partitioning themselves from clients who have a more recent and/or + typical set of documents. Even under the best of circumstances, + clients were sensitive to the ages of the network status documents + they downloaded. Therefore, instead of having the clients + correlate multiple network status documents, we have the + authorities collectively vote on a single consensus network status + document. + + * The most sensitive data in the entire network (the identity keys + of the directory authorities) needed to be stored unencrypted so + that the authorities can sign network-status documents on the fly. + Now, the authorities' identity keys are stored offline, and used + to certify medium-term signing keys that can be rotated. +``` + + diff --git a/spec/dir-spec/inferring-missing-proto-lines.md b/spec/dir-spec/inferring-missing-proto-lines.md new file mode 100644 index 0000000..7468a20 --- /dev/null +++ b/spec/dir-spec/inferring-missing-proto-lines.md @@ -0,0 +1,15 @@ +<a id="dir-spec.txt-D"></a> + +# Inferring missing proto lines + +The directory authorities no longer allow versions of Tor before +0.2.4.18-rc. But right now, there is no version of Tor in the consensus +before 0.2.4.19. Therefore, we should disallow versions of Tor earlier +than 0.2.4.19, so that we can have the protocol list for all current Tor +versions include: + +Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1-2 Link=1-4 +LinkAuth=1 Microdesc=1-2 Relay=1-2 + +For Desc, Microdesc and Cons, Tor versions before 0.2.7.stable should be +taken to only support version 1. diff --git a/spec/dir-spec/limited-ed-diff-format.md b/spec/dir-spec/limited-ed-diff-format.md new file mode 100644 index 0000000..f6a2d61 --- /dev/null +++ b/spec/dir-spec/limited-ed-diff-format.md @@ -0,0 +1,39 @@ +<a id="dir-spec.txt-E"></a> + +# Limited ed diff format + +We support the following format for consensus diffs. It's a +subset of the ed diff format, but clients MUST NOT accept other +ed commands. + +We support the following ed commands, each on a line by itself: + +```text + - "<n1>d" Delete line n1 + - "<n1>,<n2>d" Delete lines n1 through n2, inclusive + - "<n1>,$d" Delete line n1 through the end of the file, inclusive. + - "<n1>c" Replace line n1 with the following block + - "<n1>,<n2>c" Replace lines n1 through n2, inclusive, with the + following block. + - "<n1>a" Append the following block after line n1. +``` + +Note that line numbers always apply to the file after all previous +commands have already been applied. Note also that line numbers +are 1-indexed. + +The commands MUST apply to the file from back to front, such that +lines are only ever referred to by their position in the original +file. + +If there are any directory signatures on the original document, the +first command MUST be a "`<n1>,$d`" form to remove all of the directory +signatures. Using this format ensures that the client will +successfully apply the diff even if they have an unusual encoding for +the signatures. + +The replace and append command take blocks. These blocks are simply +appended to the diff after the line with the command. A line with +just a period (".") ends the block (and is not part of the lines +to add). Note that it is impossible to insert a line with just +a single dot. diff --git a/spec/dir-spec/nonterminals-server-descriptors.md b/spec/dir-spec/nonterminals-server-descriptors.md new file mode 100644 index 0000000..e2240ae --- /dev/null +++ b/spec/dir-spec/nonterminals-server-descriptors.md @@ -0,0 +1,32 @@ +<a id="dir-spec.txt-2.1.3"></a> + +# Nonterminals in server descriptors + +```text + nickname ::= between 1 and 19 alphanumeric characters ([A-Za-z0-9]), + case-insensitive. + hexdigest ::= a '$', followed by 40 hexadecimal characters + ([A-Fa-f0-9]). [Represents a relay by the digest of its identity + key.] +``` + +exitpattern ::= addrspec ":" portspec +portspec ::= "\*" | port | port "-" port +port ::= an integer between 1 and 65535, inclusive. + +```text + [Some implementations incorrectly generate ports with value 0. + Implementations SHOULD accept this, and SHOULD NOT generate it. + Connections to port 0 are never permitted.] +``` + +addrspec ::= "\*" | ip4spec | ip6spec +ipv4spec ::= ip4 | ip4 "/" num_ip4_bits | ip4 "/" ip4mask +ip4 ::= an IPv4 address in dotted-quad format +ip4mask ::= an IPv4 mask in dotted-quad format +num_ip4_bits ::= an integer between 0 and 32 +ip6spec ::= ip6 | ip6 "/" num_ip6_bits +ip6 ::= an IPv6 address, surrounded by square brackets. +num_ip6_bits ::= an integer between 0 and 128 + +bool ::= "0" | "1" diff --git a/spec/dir-spec/outline.md b/spec/dir-spec/outline.md new file mode 100644 index 0000000..de969b9 --- /dev/null +++ b/spec/dir-spec/outline.md @@ -0,0 +1,255 @@ +<a id="dir-spec.txt-1"></a> + +# Outline + +There is a small set (say, around 5-10) of semi-trusted directory +authorities. A default list of authorities is shipped with the Tor +software. Users can change this list, but are encouraged not to do so, +in order to avoid partitioning attacks. + +Every authority has a very-secret, long-term "Authority Identity Key". +This is stored encrypted and/or offline, and is used to sign "key +certificate" documents. Every key certificate contains a medium-term +(3-12 months) "authority signing key", that is used by the authority to +sign other directory information. (Note that the authority identity +key is distinct from the router identity key that the authority uses +in its role as an ordinary router.) + +Routers periodically upload signed "routers descriptors" to the +directory authorities describing their keys, capabilities, and other +information. Routers may also upload signed "extra-info documents" +containing information that is not required for the Tor protocol. +Directory authorities serve server descriptors indexed by router +identity, or by hash of the descriptor. + +Routers may act as directory caches to reduce load on the directory +authorities. They announce this in their descriptors. + +Periodically, each directory authority generates a view of +the current descriptors and status for known routers. They send a +signed summary of this view (a "status vote") to the other +authorities. The authorities compute the result of this vote, and sign +a "consensus status" document containing the result of the vote. + +Directory caches download, cache, and re-serve consensus documents. + +Clients, directory caches, and directory authorities all use consensus +documents to find out when their list of routers is out-of-date. +(Directory authorities also use vote statuses.) If it is, they download +any missing server descriptors. Clients download missing descriptors +from caches; caches and authorities download from authorities. +Descriptors are downloaded by the hash of the descriptor, not by the +relay's identity key: this prevents directory servers from attacking +clients by giving them descriptors nobody else uses. + +All directory information is uploaded and downloaded with HTTP. + +<a id="dir-spec.txt-1.1"></a> + +## What's different from version 2? { #changes-since-v2 } + +Clients used to download multiple network status documents, +corresponding roughly to "status votes" above. They would compute the +result of the vote on the client side. + +Authorities used to sign documents using the same private keys they used +for their roles as routers. This forced them to keep these extremely +sensitive keys in memory unencrypted. + +All of the information in extra-info documents used to be kept in the +main descriptors. + +<a id="dir-spec.txt-1.2"></a> + +## Document meta-format { #metaformat } + +Server descriptors, directories, and running-routers documents all obey the +following lightweight extensible information format. + +The highest level object is a Document, which consists of one or more +Items. Every Item begins with a KeywordLine, followed by zero or more +Objects. A KeywordLine begins with a Keyword, optionally followed by +whitespace and more non-newline characters, and ends with a newline. A +Keyword is a sequence of one or more characters in the set \[A-Za-z0-9-\], +but may not start with -. +An Object is a block of encoded data in pseudo-Privacy-Enhanced-Mail (PEM) +style format: that is, lines of encoded data MAY be wrapped by inserting +an ascii linefeed ("LF", also called newline, or "NL" here) character +(cf. RFC 4648 ยง3.1). When line wrapping, implementations MUST wrap lines +at 64 characters. Upon decoding, implementations MUST ignore and discard +all linefeed characters. + +More formally: + +``` +NL = The ascii LF character (hex value 0x0a). +Document ::= (Item | NL)+ +Item ::= KeywordLine Object? +KeywordLine ::= Keyword (WS Argument)*NL +Keyword = KeywordStart KeywordChar* +KeywordStart ::= 'A' ... 'Z' | 'a' ... 'z' | '0' ... '9' +KeywordChar ::= KeywordStart | '-' +Argument := ArgumentChar+ +ArgumentChar ::= any graphical printing ASCII character. +WS = (SP | TAB)+ +Object ::= BeginLine Base64-encoded-data EndLine +BeginLine ::= "-----BEGIN " Keyword (" " Keyword)*"-----" NL +EndLine ::= "-----END " Keyword (" " Keyword)* "-----" NL +``` + +A Keyword may not be `-----BEGIN`. + +The BeginLine and EndLine of an Object must use the same keyword. + +When interpreting a Document, software MUST ignore any KeywordLine that +starts with a keyword it doesn't recognize; future implementations MUST NOT +require current clients to understand any KeywordLine not currently +described. + +Other implementations that want to extend Tor's directory format MAY +introduce their own items. The keywords for extension items SHOULD start +with the characters "x-" or "X-", to guarantee that they will not conflict +with keywords used by future versions of Tor. + +In our document descriptions below, we tag Items with a multiplicity in +brackets. Possible tags are: + +```text + "At start, exactly once": These items MUST occur in every instance of + the document type, and MUST appear exactly once, and MUST be the + first item in their documents. + + "Exactly once": These items MUST occur exactly one time in every + instance of the document type. + + "At end, exactly once": These items MUST occur in every instance of + the document type, and MUST appear exactly once, and MUST be the + last item in their documents. + + "At most once": These items MAY occur zero or one times in any + instance of the document type, but MUST NOT occur more than once. + + "Any number": These items MAY occur zero, one, or more times in any + instance of the document type. + + "Once or more": These items MUST occur at least once in any instance + of the document type, and MAY occur more. +``` + +For forward compatibility, each item MUST allow extra arguments at the +end of the line unless otherwise noted. So if an item's description below +is given as: + +"thing" int int int NL + +then implementations SHOULD accept this string as well: + +"thing 5 9 11 13 16 12" NL + +but not this string: + +"thing 5" NL + +and not this string: + +```text + "thing 5 10 thing" NL + . +``` + +Whenever an item DOES NOT allow extra arguments, we will tag it with +"no extra arguments". + +<a id="dir-spec.txt-1.3"></a> + +## Signing documents { #signing } + +Every signable document below is signed in a similar manner, using a +given "Initial Item", a final "Signature Item", a digest algorithm, and +a signing key. + +The Initial Item must be the first item in the document. + +The Signature Item has the following format: + +`<signature item keyword> [arguments] NL SIGNATURE NL` + +The "SIGNATURE" Object contains a signature (using the signing key) of +the PKCS#1 1.5 padded digest of the entire document, taken from the +beginning of the Initial item, through the newline after the Signature +Item's keyword and its arguments. + +The signature does not include the algorithmIdentifier specified in PKCS #1. + +Unless specified otherwise, the digest algorithm is SHA-1. + +All documents are invalid unless signed with the correct signing key. + +The "Digest" of a document, unless stated otherwise, is its digest *as +signed by this signature scheme*. + +<a id="dir-spec.txt-1.4"></a> + +## Voting timeline + +Every consensus document has a "valid-after" (VA) time, a "fresh-until" +(FU) time and a "valid-until" (VU) time. VA MUST precede FU, which MUST +in turn precede VU. Times are chosen so that every consensus will be +"fresh" until the next consensus becomes valid, and "valid" for a while +after. At least 3 consensuses should be valid at any given time. + +The timeline for a given consensus is as follows: + +VA-DistSeconds-VoteSeconds: The authorities exchange votes. Each authority +uploads their vote to all other authorities. + +VA-DistSeconds-VoteSeconds/2: The authorities try to download any +votes they don't have. + +Authorities SHOULD also reject any votes that other authorities try to +upload after this time. (0.4.4.1-alpha was the first version to reject votes +in this way.) + +Note: Refusing late uploaded votes minimizes the chance of a consensus +split, particular when authorities are under bandwidth pressure. If an +authority is struggling to upload its vote, and finally uploads to a +fraction of authorities after this period, they will compute a consensus +different from the others. By refusing uploaded votes after this time, +we increase the likelihood that most authorities will use the same vote +set. + +Rejecting late uploaded votes does not fix the problem entirely. If +some authorities are able to download a specific vote, but others fail +to do so, then there may still be a consensus split. However, this +change does remove one common cause of consensus splits. + +VA-DistSeconds: The authorities calculate the consensus and exchange +signatures. (This is the earliest point at which anybody can +possibly get a given consensus if they ask for it.) + +VA-DistSeconds/2: The authorities try to download any signatures +they don't have. + +VA: All authorities have a multiply signed consensus. + +```text + VA ... FU: Caches download the consensus. (Note that since caches have + no way of telling what VA and FU are until they have downloaded + the consensus, they assume that the present consensus's VA is + equal to the previous one's FU, and that its FU is one interval after + that.) + + FU: The consensus is no longer the freshest consensus. + + FU ... (the current consensus's VU): Clients download the consensus. + (See note above: clients guess that the next consensus's FU will be + two intervals after the current VA.) +``` + +VU: The consensus is no longer valid; clients should continue to try to +download a new consensus if they have not done so already. + +VU + 24 hours: Clients will no longer use the consensus at all. + +VoteSeconds and DistSeconds MUST each be at least 20 seconds; FU-VA and +VU-FU MUST each be at least 5 minutes. diff --git a/spec/dir-spec/publishing-consensus.md b/spec/dir-spec/publishing-consensus.md new file mode 100644 index 0000000..6f9563c --- /dev/null +++ b/spec/dir-spec/publishing-consensus.md @@ -0,0 +1,38 @@ +# Publishing the signed consensus { #publishing-signed-consensus } + +The voting period ends at the valid-after time. If the consensus has +been signed by a majority of authorities, these documents are made +available at + +`http://<hostname>/tor/status-vote/current/consensus.z` + +and + +`http://<hostname>/tor/status-vote/current/consensus-signatures.z` + +```text + [XXX current/consensus-signatures is not currently implemented, as it + is not used in the voting protocol.] + + [XXX possible future features include support for downloading old + consensuses.] + + The other vote documents are analogously made available under + +http://<hostname>/tor/status-vote/current/authority.z +http://<hostname>/tor/status-vote/current/<fp>.z +http://<hostname>/tor/status-vote/current/d/<d>.z +http://<hostname>/tor/status-vote/current/bandwidth.z +``` + +once the voting period ends, regardless of the number of signatures. + +The authorities serve another consensus of each flavor "F" from the +locations + +```text +/tor/status-vote/(current|next)/consensus-F.z. and +/tor/status-vote/(current|next)/consensus-F/<FP1>+....z. +``` + +The standard URLs for bandwidth list files first-appeared in Tor 0.3.5. diff --git a/spec/dir-spec/router-operation-formats.md b/spec/dir-spec/router-operation-formats.md new file mode 100644 index 0000000..477e92d --- /dev/null +++ b/spec/dir-spec/router-operation-formats.md @@ -0,0 +1,7 @@ +<a id="dir-spec.txt-2"></a> + +# Router operation and formats + +This section describes how relays must behave when publishing their +information to the directory authorities, and the formats that they +use to do so. diff --git a/spec/dir-spec/server-descriptor-format.md b/spec/dir-spec/server-descriptor-format.md new file mode 100644 index 0000000..ddd7fb2 --- /dev/null +++ b/spec/dir-spec/server-descriptor-format.md @@ -0,0 +1,509 @@ +<a id="dir-spec.txt-2.1.1"></a> + +# Server descriptor format + +Server descriptors consist of the following items. + +In lines that take multiple arguments, extra arguments SHOULD be +accepted and ignored. Many of the nonterminals below are defined in +section 2.1.3. + +Note that many versions of Tor will generate an extra newline at the +end of their descriptors. Implementations MUST tolerate one or +more blank lines at the end of a single descriptor or a list of +concatenated descriptors. New implementations SHOULD NOT generate +such blank lines. + +"router" nickname address ORPort SOCKSPort DirPort NL + +\[At start, exactly once.\] + +Indicates the beginning of a server descriptor. "nickname" must be a +valid router nickname as specified in section 2.1.3. "address" must +be an IPv4 +address in dotted-quad format. The last three numbers indicate the +TCP ports at which this OR exposes functionality. ORPort is a port at +which this OR accepts TLS connections for the main OR protocol; +SOCKSPort is deprecated and should always be 0; and DirPort is the +port at which this OR accepts directory-related HTTP connections. If +any port is not supported, the value 0 is given instead of a port +number. (At least one of DirPort and ORPort SHOULD be set; +authorities MAY reject any descriptor with both DirPort and ORPort of +0.) + +```text + "identity-ed25519" NL "-----BEGIN ED25519 CERT-----" NL certificate + "-----END ED25519 CERT-----" NL +``` + +\[Exactly once, in second position in document.\] +\[No extra arguments\] + +The certificate is a base64-encoded Ed25519 certificate (see +cert-spec.txt) with terminating =s removed. When this element +is present, it MUST appear as the first or second element in +the router descriptor. + +The certificate has CERT_TYPE of \[04\]. It must include a +signed-with-ed25519-key extension (see cert-spec.txt, +section 2.2.1), so that we can extract the master identity key. + +\[Before Tor 0.4.5.1-alpha, this field was optional.\] + +"master-key-ed25519" SP MasterKey NL + +\[Exactly once\] + +Contains the base-64 encoded ed25519 master key as a single +argument. If it is present, it MUST match the identity key +in the identity-ed25519 entry. + +\[Before Tor 0.4.5.1-alpha, this field was optional.\] + +"bandwidth" bandwidth-avg bandwidth-burst bandwidth-observed NL + +\[Exactly once\] + +Estimated bandwidth for this router, in bytes per second. The +"average" bandwidth is the volume per second that the OR is willing to +sustain over long periods; the "burst" bandwidth is the volume that +the OR is willing to sustain in very short intervals. The "observed" +value is an estimate of the capacity this relay can handle. The +relay remembers the max bandwidth sustained output over any ten +second period in the past 5 days, and another sustained input. The +"observed" value is the lesser of these two numbers. + +Tor versions released before 2018 only kept bandwidth-observed for one +day. These versions are no longer supported or recommended. + +"platform" string NL + +\[At most once\] + +A human-readable string describing the system on which this OR is +running. This MAY include the operating system, and SHOULD include +the name and version of the software implementing the Tor protocol. + +"published" YYYY-MM-DD HH:MM:SS NL + +\[Exactly once\] + +The time, in UTC, when this descriptor (and its corresponding +extra-info document if any) was generated. + +"fingerprint" fingerprint NL + +\[At most once\] + +A fingerprint (a HASH_LEN-byte of asn1 encoded public key, encoded in +hex, with a single space after every 4 characters) for this router's +identity key. A descriptor is considered invalid (and MUST be +rejected) if the fingerprint line does not match the public key. + +```text + [We didn't start parsing this line until Tor 0.1.0.6-rc; it should + be marked with "opt" until earlier versions of Tor are obsolete.] + + "hibernating" bool NL + + [At most once] +``` + +If the value is 1, then the Tor relay was hibernating when the +descriptor was published, and shouldn't be used to build circuits. + +```text + [We didn't start parsing this line until Tor 0.1.0.6-rc; it should be + marked with "opt" until earlier versions of Tor are obsolete.] + + "uptime" number NL + + [At most once] + + The number of seconds that this OR process has been running. + + "onion-key" NL a public key in PEM format +``` + +\[Exactly once\] +\[No extra arguments\] + +This key is used to encrypt CREATE cells for this OR. The key MUST be +accepted for at least 1 week after any new key is published in a +subsequent descriptor. It MUST be 1024 bits. + +The key encoding is the encoding of the key as a PKCS#1 RSAPublicKey +structure, encoded in base64, and wrapped in "-----BEGIN RSA PUBLIC +KEY-----" and "-----END RSA PUBLIC KEY-----". + +"onion-key-crosscert" NL a RSA signature in PEM format. + +\[Exactly once\] +\[No extra arguments\] + +This element contains an RSA signature, generated using the +onion-key, of the following: + +```text + A SHA1 hash of the RSA identity key, + i.e. RSA key from "signing-key" (see below) [20 bytes] + The Ed25519 identity key, + i.e. Ed25519 key from "master-key-ed25519" [32 bytes] +``` + +If there is no Ed25519 identity key, or if in some future version +there is no RSA identity key, the corresponding field must be +zero-filled. + +Parties verifying this signature MUST allow additional data +beyond the 52 bytes listed above. + +This signature proves that the party creating the descriptor +had control over the secret key corresponding to the +onion-key. + +\[Before Tor 0.4.5.1-alpha, this field was optional whenever +identity-ed25519 was absent.\] + +"ntor-onion-key" base-64-encoded-key + +\[Exactly once\] + +A curve25519 public key used for the ntor circuit extended +handshake. It's the standard encoding of the OR's curve25519 +public key, encoded in base 64. The trailing '=' sign MAY be +omitted from the base64 encoding. The key MUST be accepted +for at least 1 week after any new key is published in a +subsequent descriptor. + +\[Before Tor 0.4.5.1-alpha, this field was optional.\] + +<a id="ntor-onion-key-crosscert"></a> +```text + "ntor-onion-key-crosscert" SP Bit NL + "-----BEGIN ED25519 CERT-----" NL certificate + "-----END ED25519 CERT-----" NL +``` + +\[Exactly once\] +\[No extra arguments\] + +A signature created with the ntor-onion-key, using the +certificate format documented in cert-spec.txt, with type +\[0a\]. The signed key here is the master identity key. + +Bit must be "0" or "1". It indicates the sign of the ed25519 +public key corresponding to the ntor onion key. If Bit is "0", +then implementations MUST guarantee that the x-coordinate of +the resulting ed25519 public key is positive. Otherwise, if +Bit is "1", then the sign of the x-coordinate MUST be negative. + +To compute the ed25519 public key corresponding to a curve25519 +key, and for further explanation on key formats, see appendix C. + +This signature proves that the party creating the descriptor +had control over the secret key corresponding to the +ntor-onion-key. + +\[Before Tor 0.4.5.1-alpha, this field was optional whenever +identity-ed25519 was absent.\] + +"signing-key" NL a public key in PEM format + +\[Exactly once\] +\[No extra arguments\] + +The OR's long-term RSA identity key. It MUST be 1024 bits. + +The encoding is as for "onion-key" above. + +"accept" exitpattern NL +"reject" exitpattern NL + +\[Any number\] + +These lines describe an "exit policy": the rules that an OR follows +when deciding whether to allow a new stream to a given address. The +'exitpattern' syntax is described below. There MUST be at least one +such entry. The rules are considered in order; if no rule matches, +the address will be accepted. For clarity, the last such entry SHOULD +be accept *:* or reject *:*. + +"ipv6-policy" SP ("accept" / "reject") SP PortList NL + +\[At most once.\] + +An exit-policy summary as specified in sections 3.4.1 and 3.8.2, +summarizing +the router's rules for connecting to IPv6 addresses. A missing +"ipv6-policy" line is equivalent to "ipv6-policy reject +1-65535". + +"overload-general" SP version SP YYYY-MM-DD HH:MM:SS NL + +\[At most once.\] + +Indicates that a relay has reached an "overloaded state" which can be +one or many of the following load metrics: + +```text + - Any OOM invocation due to memory pressure + - Any ntor onionskins are dropped + - TCP port exhaustion +``` + +The timestamp is when at least one metrics was detected. It should always +be at the hour and thus, as an example, "2020-01-10 13:00:00" is an +expected timestamp. Because this is a binary state, if the line is +present, we consider that it was hit at the very least once somewhere +between the provided timestamp and the "published" timestamp of the +document which is when the document was generated. + +The overload-general line should remain in place for 72 hours since last +triggered. If the limits are reached again in this period, the timestamp +is updated, and this 72 hour period restarts. + +The 'version' field is set to '1' for now. + +```text + (Introduced in tor-0.4.6.1-alpha, but moved from extra-info to general + descriptor in tor-0.4.6.2-alpha) + + "router-sig-ed25519" SP Signature NL + + [Exactly once.] +``` + +It MUST be the next-to-last element in the descriptor, appearing +immediately before the RSA signature. It MUST contain an Ed25519 +signature of a SHA256 digest of the entire document. This digest is +taken from the first character up to and including the first space +after the "router-sig-ed25519" string. Before computing the digest, +the string "Tor router descriptor signature v1" is prefixed to the +document. + +The signature is encoded in Base64, with terminating =s removed. + +The signing key in the identity-ed25519 certificate MUST +be the one used to sign the document. + +\[Before Tor 0.4.5.1-alpha, this field was optional whenever +identity-ed25519 was absent.\] + +"router-signature" NL Signature NL + +\[At end, exactly once\] +\[No extra arguments\] + +The "SIGNATURE" object contains a signature of the PKCS1-padded +hash of the entire server descriptor, taken from the beginning of the +"router" line, through the newline after the "router-signature" line. +The server descriptor is invalid unless the signature is performed +with the router's identity key. + +"contact" info NL + +\[At most once\] + +Describes a way to contact the relay's administrator, preferably +including an email address and a PGP key fingerprint. + +"bridge-distribution-request" SP Method NL + +\[At most once, bridges only.\] + +The "Method" describes how a Bridge address is distributed by +BridgeDB. Recognized methods are: "none", "any", "https", "email", +"moat". If set to "none", BridgeDB will avoid distributing your bridge +address. If set to "any", BridgeDB will choose how to distribute your +bridge address. Choosing any of the other methods will tell BridgeDB to +distribute your bridge via a specific method: + +```text + - "https" specifies distribution via the web interface at + https://bridges.torproject.org; + - "email" specifies distribution via the email autoresponder at + bridges@torproject.org; + - "moat" specifies distribution via an interactive menu inside Tor + Browser; and + + Potential future "Method" specifiers must be as follows: + Method = (KeywordChar | "_") + +``` + +All bridges SHOULD include this line. Non-bridges MUST NOT include +it. + +BridgeDB SHOULD treat unrecognized Method values as if they were +"none". + +(Default: "any") + +\[This line was introduced in 0.3.2.3-alpha, with a minimal backport +to 0.2.5.16, 0.2.8.17, 0.2.9.14, 0.3.0.13, 0.3.1.9, and later.\] + +"family" names NL + +\[At most once\] + +'Names' is a space-separated list of relay nicknames or +hexdigests. If two ORs list one another in their "family" entries, +then OPs should treat them as a single OR for the purpose of path +selection. + +For example, if node A's descriptor contains "family B", and node B's +descriptor contains "family A", then node A and node B should never +be used on the same circuit. + +```text + "read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once] + "write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once] +``` + +(These fields once appeared in router descriptors, but have +appeared in extra-info descriptors since 0.2.0.x.) + +"eventdns" bool NL + +\[At most once\] + +Declare whether this version of Tor is using the newer enhanced +dns logic. Versions of Tor with this field set to false SHOULD NOT +be used for reverse hostname lookups. + +```text + [This option is obsolete. All Tor current relays should be presumed + to have the evdns backend.] + + "caches-extra-info" NL +``` + +\[At most once.\] +\[No extra arguments\] + +Present only if this router is a directory cache that provides +extra-info documents. + +\[Versions before 0.2.0.1-alpha don't recognize this\] + +"extra-info-digest" SP sha1-digest \[SP sha256-digest\] NL + +\[At most once\] + +"sha1-digest" is a hex-encoded SHA1 digest (using upper-case characters) +of the router's extra-info document, as signed in the router's +extra-info (that is, not including the signature). (If this field is +absent, the router is not uploading a corresponding extra-info +document.) + +"sha256-digest" is a base64-encoded SHA256 digest of the extra-info +document. Unlike the "sha1-digest", this digest is calculated over the +entire document, including the signature. This difference is due to +a long-lived bug in the tor implementation that it would be difficult +to roll out an incremental fix for, not a design choice. Future digest +algorithms specified should not include the signature in the data used +to compute the digest. + +\[Versions before 0.2.7.2-alpha did not include a SHA256 digest.\] +\[Versions before 0.2.0.1-alpha don't recognize this field at all.\] + +"hidden-service-dir" NL + +\[At most once.\] + +Present only if this router stores and serves hidden service +descriptors. This router supports the descriptor versions declared +in the HSDir "proto" entry. If there is no "proto" entry, this +router supports version 2 descriptors. + +```text + "protocols" SP "Link" SP LINK-VERSION-LIST SP "Circuit" SP + CIRCUIT-VERSION-LIST NL + + [At most once.] +``` + +An obsolete list of protocol versions, superseded by the "proto" +entry. This list was never parsed, and has not been emitted +since Tor 0.2.9.4-alpha. New code should neither generate nor +parse this line. + +"allow-single-hop-exits" NL + +\[At most once.\] +\[No extra arguments\] + +Present only if the router allows single-hop circuits to make exit +connections. Most Tor relays do not support this: this is +included for specialized controllers designed to support perspective +access and such. This is obsolete in tor version >= 0.3.1.0-alpha. + +"or-address" SP ADDRESS ":" PORT NL + +\[Any number\] + +ADDRESS = IP6ADDR | IP4ADDR +IPV6ADDR = an ipv6 address, surrounded by square brackets. +IPV4ADDR = an ipv4 address, represented as a dotted quad. +PORT = a number between 1 and 65535 inclusive. + +An alternative for the address and ORPort of the "router" line, but with +two added capabilities: + +```text + * or-address can be either an IPv4 or IPv6 address + * or-address allows for multiple ORPorts and addresses +``` + +A descriptor SHOULD NOT include an or-address line that does nothing but +duplicate the address:port pair from its "router" line. + +The ordering of or-address lines and their PORT entries matter because +Tor MAY accept a limited number of address/port pairs. As of +Tor 0.2.3.x only the first address/port pair is advertised and used. + +"tunnelled-dir-server" NL + +\[At most once.\] +\[No extra arguments\] + +```text + Present if the router accepts "tunneled" directory requests using a + BEGIN_DIR relay message over the router's OR port. + (Added in 0.2.8.1-alpha. Before this, Tor relays accepted + tunneled directory requests only if they had a DirPort open, + or if they were bridges.) + + "proto" SP Entries NL + + [Exactly once.] +``` + +Entries = +Entries = Entry +Entries = Entry SP Entries + +Entry = Keyword "=" Values + +Values = +Values = Value +Values = Value "," Values + +Value = Int +Value = Int "-" Int + +Int = NON_ZERO_DIGIT +Int = Int DIGIT + +Each 'Entry' in the "proto" line indicates that the Tor relay supports +one or more versions of the protocol in question. Entries should be +sorted by keyword. Values should be numerically ascending within each +entry. (This implies that there should be no overlapping ranges.) +Ranges should be represented as compactly as possible. Ints must be no +larger than 63. + +This field was first added in Tor 0.2.9.x. + +\[Before Tor 0.4.5.1-alpha, this field was optional.\] diff --git a/spec/dir-spec/serving-bandwidth-list-files.md b/spec/dir-spec/serving-bandwidth-list-files.md new file mode 100644 index 0000000..6860a6b --- /dev/null +++ b/spec/dir-spec/serving-bandwidth-list-files.md @@ -0,0 +1,30 @@ +<a id="dir-spec.txt-3.4.3"></a> + +# Serving bandwidth list files { #serving-bwlist } + +If an authority has used a bandwidth list file to generate a vote +document it SHOULD make it available at + +`http://<hostname>/tor/status-vote/next/bandwidth.z` + +at the start of each voting period. + +It MUST NOT attempt to send its bandwidth list file in a HTTP POST to +other authorities and it SHOULD NOT make bandwidth list files from other +authorities available. + +If an authority makes this file available, it MUST be the bandwidth file +used to create the vote document available at + +`http://<hostname>/tor/status-vote/next/authority.z` + +To avoid inconsistent reads, authorities SHOULD only read the bandwidth +file once per voting period. Further processing and serving SHOULD use a +cached copy. + +The bandwidth list format is described in bandwidth-file-spec.txt. + +The standard URLs for bandwidth list files first-appeared in +Tor 0.4.0.4-alpha. + +<a id="dir-spec.txt-3.5"></a> diff --git a/spec/dir-spec/standards-compliance.md b/spec/dir-spec/standards-compliance.md new file mode 100644 index 0000000..2d4e525 --- /dev/null +++ b/spec/dir-spec/standards-compliance.md @@ -0,0 +1,81 @@ +<a id="dir-spec.txt-6"></a> + +# Standards compliance + +All clients and servers MUST support HTTP 1.0. Clients and servers MAY +support later versions of HTTP as well. + +<a id="dir-spec.txt-6.1"></a> + +## HTTP headers + +Servers SHOULD set Content-Encoding to the algorithm used to compress the +document(s) being served. Recognized algorithms are: + +```text + - "identity" -- RFC2616 section 3.5 + - "deflate" -- RFC2616 section 3.5 + - "gzip" -- RFC2616 section 3.5 + - "x-zstd" -- The zstandard compression algorithm (www.zstd.net) + - "x-tor-lzma" -- The lzma compression algorithm, with a "preset" + value no higher than 6. +``` + +Clients SHOULD use Accept-Encoding on most directory requests to indicate +which of the above compression algorithms they support. If they omit it +(as Tor clients did before 0.3.1.1-alpha), then the server should serve +only "deflate" or "identity" encoded documents, based on the presence or +absence of the ".z" suffix on the requested URL. + +Note that for anonymous directory requests (that is, requests made over +multi-hop circuits, like those for onion service lookups) implementations +SHOULD NOT advertise any Accept-Encoding values other than deflate. To do +so would be to create a fingerprinting opportunity. + +When receiving multiple documents, clients MUST accept compressed +concatenated documents and concatenated compressed documents as +equivalent. + +Servers MAY set the Content-Length: header. When they do, it should +match the number of compressed bytes that they are sending. + +Servers MAY include an X-Your-Address-Is: header, whose value is the +apparent IP address of the client connecting to them (as a dotted quad). +For directory connections tunneled over a BEGIN_DIR stream, servers SHOULD +report the IP from which the circuit carrying the BEGIN_DIR stream reached +them. + +Servers SHOULD disable caching of multiple network statuses or multiple +server descriptors. Servers MAY enable caching of single descriptors, +single network statuses, the list of all server descriptors, a v1 +directory, or a v1 running routers document. XXX mention times. + +<a id="dir-spec.txt-6.2"></a> + +## HTTP status codes + +Tor delivers the following status codes. Some were chosen without much +thought; other code SHOULD NOT rely on specific status codes yet. + +```text + 200 -- the operation completed successfully + -- the user requested statuses or serverdescs, and none of the ones we + requested were found (0.2.0.4-alpha and earlier). + + 304 -- the client specified an if-modified-since time, and none of the + requested resources have changed since that time. + + 400 -- the request is malformed, or + -- the URL is for a malformed variation of one of the URLs we support, + or + -- the client tried to post to a non-authority, or + -- the authority rejected a malformed posted document, or + + 404 -- the requested document was not found. + -- the user requested statuses or serverdescs, and none of the ones + requested were found (0.2.0.5-alpha and later). + + 503 -- we are declining the request in order to save bandwidth + -- user requested some items that we ordinarily generate or store, + but we do not have any available. +``` diff --git a/spec/dir-spec/uploading-relay-documents.md b/spec/dir-spec/uploading-relay-documents.md new file mode 100644 index 0000000..da34943 --- /dev/null +++ b/spec/dir-spec/uploading-relay-documents.md @@ -0,0 +1,43 @@ +<a id="dir-spec.txt-2.1"></a> + +# Uploading server descriptors and extra-info documents + +ORs SHOULD generate a new server descriptor and a new extra-info +document whenever any of the following events have occurred: + +```text + - A period of time (18 hrs by default) has passed since the last + time a descriptor was generated. + + - A descriptor field other than bandwidth or uptime has changed. + + - Its uptime is less than 24h and bandwidth has changed by a factor of 2 + from the last time a descriptor was generated, and at least a given + interval of time (3 hours by default) has passed since then. + + - Its uptime has been reset (by restarting). + + - It receives a networkstatus consensus in which it is not listed. + + - It receives a networkstatus consensus in which it is listed + with the StaleDesc flag. + + [XXX this list is incomplete; see router_differences_are_cosmetic() + in routerlist.c for others] +``` + +ORs SHOULD NOT publish a new server descriptor or extra-info document +if none of the above events have occurred and not much time has passed +(12 hours by default). + +Tor versions older than 0.3.5.1-alpha ignore uptime when checking for +bandwidth changes. + +After generating a descriptor, ORs upload them to every directory +authority they know, by posting them (in order) to the URL + +http://<hostname:port>/tor/ + +Server descriptors may not exceed 20,000 bytes in length; extra-info +documents may not exceed 50,000 bytes in length. If they do, the +authorities SHOULD reject them. |