From f79272ef1f774b3788b74a3fe4fef75095dfae06 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Fri, 13 Oct 2023 18:00:42 -0400 Subject: Run markdownlint --fix on spec. --- spec/README.md | 1 - spec/SUMMARY.md | 8 ++++ spec/address-spec.md | 7 +++- spec/bandwidth-file-spec-intro.md | 1 - spec/bandwidth-file-spec/definitions.md | 2 +- spec/bandwidth-file-spec/format-details.md | 2 +- spec/bandwidth-file-spec/header-list-format.md | 2 +- spec/bandwidth-file-spec/implementation-details.md | 6 ++- spec/bandwidth-file-spec/relay-line-format.md | 2 +- spec/bandwidth-file-spec/sample-data.md | 13 ++++++- spec/bandwidth-file-spec/scaling-bandwidths.md | 6 ++- spec/bandwidth-file-spec/scope-preliminaries.md | 5 ++- spec/bridgedb-spec.md | 13 ++++++- spec/cert-spec.md | 13 ++++++- spec/control-spec-intro.md | 1 - spec/control-spec/commands.md | 38 ++++++++++++++++++- spec/control-spec/implementation-notes.md | 27 +++++++++---- spec/control-spec/message-format.md | 7 +++- spec/control-spec/protocol-outline.md | 3 +- spec/control-spec/replies.md | 32 +++++++++++++++- spec/control-spec/scope.md | 2 +- spec/dir-list-spec.md | 44 ++++++++++++++++------ spec/dir-spec-intro.md | 1 - ...epting-server-descriptor-extra-info-document.md | 2 +- spec/dir-spec/assigning-flags-vote.md | 4 +- spec/dir-spec/client-operation.md | 17 +++++++-- spec/dir-spec/computing-microdescriptors.md | 2 +- spec/dir-spec/consensus-negotiation-timeline.md | 4 +- ...ting-curve25519-public-key-to-ed25519-public.md | 2 +- spec/dir-spec/creating-key-certificates.md | 2 +- .../directory-authority-operation-formats.md | 2 +- spec/dir-spec/directory-cache-operation.md | 12 +++++- spec/dir-spec/exchanging-votes.md | 2 +- spec/dir-spec/extra-info-document-format.md | 2 +- spec/dir-spec/general-use-http-urls.md | 2 +- spec/dir-spec/inferring-missing-proto-lines.md | 4 +- spec/dir-spec/limited-ed-diff-format.md | 2 +- spec/dir-spec/nonterminals-server-descriptors.md | 2 +- spec/dir-spec/outline.md | 10 +++-- spec/dir-spec/router-operation-formats.md | 2 +- spec/dir-spec/scope-preliminaries.md | 5 ++- spec/dir-spec/server-descriptor-format.md | 2 +- spec/dir-spec/serving-bandwidth-list-files.md | 18 ++++++++- spec/dir-spec/standards-compliance.md | 4 +- ...ding-server-descriptors-extra-info-documents.md | 2 +- .../vote-consensus-status-document-formats.md | 4 +- spec/dos-spec.md | 19 +++++----- spec/ext-orport-spec.md | 15 +++++++- spec/gettor-spec.md | 14 +++++-- spec/glossary.md | 25 +++++++++--- spec/guard-spec-intro.md | 1 - spec/guard-spec/algorithm.md | 26 +++++++++---- spec/guard-spec/appendices.md | 8 +++- ...uit-creation-entry-guard-selection-1000-foot.md | 2 +- spec/guard-spec/introduction-motivation.md | 2 +- spec/guard-spec/state-instances.md | 2 +- .../still-non-addressed-issues-sectiontodo.md | 2 +- spec/padding-spec-intro.md | 1 - spec/padding-spec/acknowledgments.md | 2 +- spec/padding-spec/circuit-level-padding.md | 10 ++++- spec/padding-spec/connection-level-padding.md | 8 +++- spec/padding-spec/overview.md | 2 +- spec/param-spec.md | 13 ++++++- spec/path-spec-intro.md | 1 - spec/path-spec/attaching-streams-to-circuits.md | 2 +- spec/path-spec/building-circuits.md | 2 +- spec/path-spec/cannibalizing-circuits.md | 2 +- ...cting-route-manipulation-by-guard-nodes-path.md | 7 +++- spec/path-spec/general-operation.md | 4 +- spec/path-spec/guard-nodes.md | 5 ++- spec/path-spec/handling-failure.md | 2 +- spec/path-spec/hidden-service-related-circuits.md | 2 +- ...n-to-give-up-timeout-on-circuit-construction.md | 11 +++++- spec/path-spec/old-notes.md | 9 +++-- spec/path-spec/path-selection-constraints.md | 4 +- spec/path-spec/server-descriptor-purposes.md | 2 +- spec/path-spec/when-we-build.md | 9 ++++- spec/pt-spec-intro.md | 1 - spec/pt-spec/acknowledgments.md | 2 +- spec/pt-spec/anonymity-considerations.md | 2 +- spec/pt-spec/architecture-overview.md | 2 +- .../example-client-pluggable-transport-session.md | 2 +- .../example-server-pluggable-transport-session.md | 2 +- spec/pt-spec/introduction.md | 3 +- ...le-transport-client-per-connection-arguments.md | 2 +- ...luggable-transport-configuration-environment.md | 5 ++- spec/pt-spec/pluggable-transport-naming.md | 2 +- spec/pt-spec/pluggable-transport-shutdown.md | 2 +- ...le-transport-to-parent-process-communication.md | 6 +++ spec/pt-spec/references.md | 2 +- spec/pt-spec/specification.md | 2 +- spec/rend-spec-v3-intro.md | 1 - ...deriving-blinded-keys-subcredentials-subcred.md | 17 ++++++++- .../encoding-onion-addresses-onionaddress.md | 2 +- .../encrypting-data-between-client-host.md | 2 +- ...rating-publishing-hidden-service-descriptors.md | 2 +- ...ce-descriptors-encryption-format-hs-desc-enc.md | 12 +++++- ...service-descriptors-outer-wrapper-desc-outer.md | 2 +- ...n-service-directory-format-hidservdir-format.md | 2 +- .../hidden-services-overview-preliminaries.md | 16 +++++--- .../introduction-protocol-intro-protocol.md | 15 +++++++- ...ging-authorized-client-data-client-auth-mgmt.md | 2 +- .../numeric-values-reserved-this-document.md | 2 +- spec/rend-spec-v3/open-questions.md | 4 +- spec/rend-spec-v3/protocol-overview.md | 11 +++++- ...ishing-shared-random-values-pub-sharedrandom.md | 4 +- ...ommendations-for-searching-for-vanity-onions.md | 2 +- spec/rend-spec-v3/rendezvous-protocol.md | 8 +++- spec/rend-spec-v3/reserved-numbers.md | 2 +- spec/rend-spec-v3/selecting-nodes-picknodes.md | 2 +- .../signature-scheme-with-key-blinding-keyblind.md | 4 +- spec/rend-spec-v3/text-vectors.md | 2 +- .../two-methods-for-managing-revision-counters.md | 4 +- spec/socks-extensions.md | 14 +++++-- spec/srv-spec-intro.md | 1 - spec/srv-spec/acknowledgements.md | 2 +- spec/srv-spec/discussion.md | 5 ++- spec/srv-spec/introduction.md | 4 +- spec/srv-spec/overview.md | 10 ++++- spec/srv-spec/protocol.md | 12 +++++- spec/srv-spec/security-analysis.md | 8 +++- spec/srv-spec/specification-spec.md | 9 ++++- spec/tor-spec-intro.md | 1 - .../application-connections-stream-management.md | 2 +- spec/tor-spec/cell-packet-format.md | 2 +- spec/tor-spec/circuit-management.md | 2 +- spec/tor-spec/closing-streams.md | 2 +- spec/tor-spec/connections.md | 4 +- spec/tor-spec/create-created-cells.md | 9 ++++- spec/tor-spec/creating-circuits.md | 3 +- spec/tor-spec/flow-control.md | 7 +++- spec/tor-spec/handling-relay_early-cells.md | 2 +- spec/tor-spec/handling-resource-exhaustion.md | 5 ++- .../negotiating-initializing-connections.md | 11 +++++- spec/tor-spec/opening-streams-transferring-data.md | 4 +- spec/tor-spec/preliminaries.md | 9 ++++- spec/tor-spec/relay-cells.md | 4 +- spec/tor-spec/remote-hostname-lookup.md | 2 +- spec/tor-spec/routing-relay-cells.md | 9 ++++- spec/tor-spec/setting-circuit-keys.md | 4 +- spec/tor-spec/subprotocol-versioning.md | 15 +++++++- spec/tor-spec/system-overview.md | 5 ++- spec/tor-spec/tearing-down-circuits.md | 2 +- spec/version-spec.md | 6 ++- 144 files changed, 669 insertions(+), 217 deletions(-) diff --git a/spec/README.md b/spec/README.md index ffc1061..63aa12f 100644 --- a/spec/README.md +++ b/spec/README.md @@ -65,7 +65,6 @@ need to have mdbook installed to do this.) [`spec/tor-spec-intro.md`]: https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/spec/tor-spec-intro.md?ref_type=heads [`SUMMARY.md`]: https://gitlab.torproject.org/tpo/core/torspec/-/raw/main/spec/SUMMARY.md?ref_type=heads - ---- ## Permalinks diff --git a/spec/SUMMARY.md b/spec/SUMMARY.md index f1288ff..542491b 100644 --- a/spec/SUMMARY.md +++ b/spec/SUMMARY.md @@ -85,10 +85,12 @@ - [Circuit-level padding](./padding-spec/circuit-level-padding.md) - [Acknowledgments](./padding-spec/acknowledgments.md) - [Preventing Denial-Of-Service](./dos-spec.md) + # Additional behaviors for clients - [`Tor's extensions to the SOCKS protocol`](./socks-extensions.md) - [`Special Hostnames in Tor`](./address-spec.md) + # Onion services - [`Tor Rendezvous Specification - Version 3`](./rend-spec-v3-intro.md) @@ -113,6 +115,7 @@ - [Appendix G: Managing authorized client data [CLIENT-AUTH-MGMT]](./rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md) - [Appendix F: Two methods for managing revision counters.](./rend-spec-v3/two-methods-for-managing-revision-counters.md) - [Appendix G: Text vectors](./rend-spec-v3/text-vectors.md) + # Anticensorship tools and protocols - [`BridgeDB specification`](./bridgedb-spec.md) @@ -132,6 +135,7 @@ - [Appendix A: Example Client Pluggable Transport Session](./pt-spec/example-client-pluggable-transport-session.md) - [Appendix B: Example Server Pluggable Transport Session](./pt-spec/example-server-pluggable-transport-session.md) - [`GetTor specification`](./gettor-spec.md) + # For C Tor only - [`The Tor Control Protocol`](./control-spec-intro.md) @@ -142,6 +146,7 @@ - [Replies](./control-spec/replies.md) - [Implementation notes](./control-spec/implementation-notes.md) - [`How Tor Version Numbers Work`](./version-spec.md) + # Less commonly needed file formats - [`Tor Bandwidth File Format`](./bandwidth-file-spec-intro.md) @@ -153,12 +158,15 @@ - [Implementation details](./bandwidth-file-spec/implementation-details.md) - [Sample data](./bandwidth-file-spec/sample-data.md) - [Scaling bandwidths](./bandwidth-file-spec/scaling-bandwidths.md) + # Implementation details - [`Tor Directory List Format`](./dir-list-spec.md) + # Reserved names and numbers - [`Tor network parameters`](./param-spec.md) + # Unfinished - [`Glossary`](./glossary.md) diff --git a/spec/address-spec.md b/spec/address-spec.md index 272e366..0254e24 100644 --- a/spec/address-spec.md +++ b/spec/address-spec.md @@ -11,10 +11,11 @@ Table of Contents ``` + # Overview Most of the time, Tor treats user-specified hostnames as opaque: When -the user connects to www.torproject.org, Tor picks an exit node and uses +the user connects to , Tor picks an exit node and uses that node to connect to "www.torproject.org". Some hostnames, however, can be used to override Tor's default behavior and circuit-building rules. @@ -26,6 +27,7 @@ substituted for certain IP addresses using the MapAddress configuration option or the MAPADDRESS control command. + # .exit ```text @@ -61,6 +63,7 @@ to potential application-level attacks. ``` + # .onion ```text @@ -93,6 +96,7 @@ The "ignored" portion of the address is intended for use in vhosting, and is supported in Tor 0.2.4.10-alpha and later. + # .noconnect SYNTAX: [string].noconnect @@ -105,4 +109,3 @@ using the same instance of Tor that they're controlling. This feature was added in Tor 0.1.2.4-alpha, and taken out in Tor 0.2.2.1-alpha over fears that it provided another avenue for detecting Tor users via application-level web tricks. - diff --git a/spec/bandwidth-file-spec-intro.md b/spec/bandwidth-file-spec-intro.md index 1fd5354..93d812e 100644 --- a/spec/bandwidth-file-spec-intro.md +++ b/spec/bandwidth-file-spec-intro.md @@ -30,4 +30,3 @@ Table of Contents B.3. Quota changes B.4. Torflow aggregation ``` - diff --git a/spec/bandwidth-file-spec/definitions.md b/spec/bandwidth-file-spec/definitions.md index 2749193..0300fb5 100644 --- a/spec/bandwidth-file-spec/definitions.md +++ b/spec/bandwidth-file-spec/definitions.md @@ -1,4 +1,5 @@ + ## Definitions The following nonterminals are defined in Tor directory protocol @@ -52,4 +53,3 @@ Tor 0.2.6.2-alpha and earlier. Parsers MAY ignore longer Lines. Note that directory authorities are only supported on the two most recent stable Tor versions, so we expect that line limits will be removed after Tor 0.4.0 is released in 2019. - diff --git a/spec/bandwidth-file-spec/format-details.md b/spec/bandwidth-file-spec/format-details.md index 247250a..c5de6f3 100644 --- a/spec/bandwidth-file-spec/format-details.md +++ b/spec/bandwidth-file-spec/format-details.md @@ -1,4 +1,5 @@ + # Format details ```text @@ -8,4 +9,3 @@ - Relay Lines (zero or more times), in an arbitrary order. If it does not contain these sections, parsers SHOULD ignore the file. ``` - diff --git a/spec/bandwidth-file-spec/header-list-format.md b/spec/bandwidth-file-spec/header-list-format.md index b751a93..4feae69 100644 --- a/spec/bandwidth-file-spec/header-list-format.md +++ b/spec/bandwidth-file-spec/header-list-format.md @@ -1,4 +1,5 @@ + ## Header List format It consists of a Timestamp line and zero or more HeaderLines. @@ -411,4 +412,3 @@ terminator. Tor 0.4.0.1-alpha and later look for a 5-character terminator, or the first relay bandwidth line. sbws versions 0.1.0 to 1.0.2 used a 4-character terminator, this bug was fixed in 1.0.3. - diff --git a/spec/bandwidth-file-spec/implementation-details.md b/spec/bandwidth-file-spec/implementation-details.md index dc9f8c0..5491d57 100644 --- a/spec/bandwidth-file-spec/implementation-details.md +++ b/spec/bandwidth-file-spec/implementation-details.md @@ -1,7 +1,9 @@ + ## Implementation details + ### Writing bandwidth files atomically To avoid inconsistent reads, implementations SHOULD write bandwidth files @@ -16,11 +18,13 @@ to the configured V3BandwidthsFile path. Torflow does not write bandwidth files atomically. + ### Additional KeyValue pair definitions KeyValue pairs in RelayLines that current implementations generate. + #### Simple Bandwidth Scanner sbws RelayLines contain these keys: @@ -376,6 +380,7 @@ The filtered stream ratio of this relay calculated as explained in B4.3. This KeyValue was added in version 1.7.0 of this specification. + #### Torflow Torflow RelayLines include node_id and bw, and other KeyValue pairs [2]. @@ -391,4 +396,3 @@ References: 4. https://gitweb.torproject.org/torspec.git/tree/version-spec.txt 5. https://semver.org/ ``` - diff --git a/spec/bandwidth-file-spec/relay-line-format.md b/spec/bandwidth-file-spec/relay-line-format.md index 9e1d1b4..13a9ffd 100644 --- a/spec/bandwidth-file-spec/relay-line-format.md +++ b/spec/bandwidth-file-spec/relay-line-format.md @@ -1,4 +1,5 @@ + ## Relay Line format It consists of zero or more RelayLines containing relay ids and @@ -123,4 +124,3 @@ pairs. Additional KeyValue pairs MUST NOT use any keywords specified in the header format. If there are, the parser MAY ignore conflicting keywords. - diff --git a/spec/bandwidth-file-spec/sample-data.md b/spec/bandwidth-file-spec/sample-data.md index 1c21f89..e05ca47 100644 --- a/spec/bandwidth-file-spec/sample-data.md +++ b/spec/bandwidth-file-spec/sample-data.md @@ -1,9 +1,11 @@ + # Sample data The following has not been obtained from any real measurement. + ## Generated by Torflow This an example version 1.0.0 document: @@ -13,6 +15,7 @@ node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test measured_at=1 node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 measured_at=1523911623 updated_at=1523911623 pid_error=3.96703337994 pid_error_sum=3.96703337994 pid_bw=47422125 pid_delta=2.65469736988 circ_fail=0.0 scanner=/filepath + ## Generated by sbws version 0.1.0 1523911758 @@ -24,10 +27,12 @@ file_created=2018-04-16T21:49:18 generator_started=2018-04-16T15:13:25 earliest_bandwidth=2018-04-16T15:13:26 ==== + bw=380 error_circ=0 error_misc=0 error_stream=1 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ nick=Test node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 rtt=380 success=1 time=2018-05-08T16:13:26 bw=189 error_circ=0 error_misc=0 error_stream=0 master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I nick=Test2 node_id=$96C15995F30895689291F455587BD94CA427B6FC rtt=378 success=1 time=2018-05-08T16:13:36 + ## Generated by sbws version 1.0.3 1523911758 @@ -44,11 +49,13 @@ percent_eligible_relays=93 software=sbws software_version=1.0.3 ===== + bw=38000 bw_mean=1127824 bw_median=1180062 desc_bw_avg=1073741824 desc_bw_obs_last=17230879 desc_bw_obs_mean=14732306 error_circ=0 error_misc=0 error_stream=1 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ nick=Test node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 rtt=380 success=1 time=2018-05-08T16:13:26 bw=1 bw_mean=199162 bw_median=185675 desc_bw_avg=409600 desc_bw_obs_last=836165 desc_bw_obs_mean=858030 error_circ=0 error_misc=0 error_stream=0 master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I nick=Test2 node_id=$96C15995F30895689291F455587BD94CA427B6FC rtt=378 success=1 time=2018-05-08T16:13:36 -### When there are not enough eligible measured relays: + +### When there are not enough eligible measured relays 1540496079 version=1.2.0 @@ -66,6 +73,7 @@ software_version=1.0.3 ===== + ## Headers generated by sbws version 1.0.4 1523911758 @@ -86,6 +94,7 @@ software_version=1.0.4 ===== + ## Generated by sbws version 1.1.0 1523911758 @@ -113,6 +122,6 @@ software=sbws software_version=1.1.0 time_to_report_half_network=57273 ===== + bw=1 error_circ=1 error_destination=0 error_misc=0 error_second_relay=0 error_stream=0 master_key_ed25519=J3HQ24kOQWac3L1xlFLp7gY91qkb5NuKxjj1BhDi+m8 nick=snap269 node_id=$DC4D609F95A52614D1E69C752168AF1FCAE0B05F relay_recent_measurement_attempt_count=3 relay_recent_measurements_excluded_error_count=1 relay_recent_measurements_excluded_near_count=3 relay_recent_consensus_count=3 relay_recent_priority_list_count=3 success=3 time=2019-03-16T18:20:57 unmeasured=1 vote=0 bw=1 error_circ=0 error_destination=0 error_misc=0 error_second_relay=0 error_stream=2 master_key_ed25519=h6ZB1E1yBFWIMloUm9IWwjgaPXEpL5cUbuoQDgdSDKg nick=relay node_id=$C4544F9E209A9A9B99591D548B3E2822236C0503 relay_recent_measurement_attempt_count=3 relay_recent_measurements_excluded_error_count=2 relay_recent_measurements_excluded_few_count=1 relay_recent_consensus_count=3 relay_recent_priority_list_count=3 success=1 time=2019-03-17T06:50:58 unmeasured=1 vote=0 - diff --git a/spec/bandwidth-file-spec/scaling-bandwidths.md b/spec/bandwidth-file-spec/scaling-bandwidths.md index 8ae80c9..32fcb83 100644 --- a/spec/bandwidth-file-spec/scaling-bandwidths.md +++ b/spec/bandwidth-file-spec/scaling-bandwidths.md @@ -1,7 +1,9 @@ + # Scaling bandwidths + ## Scaling requirements ```text @@ -18,6 +20,7 @@ torflow and sbws, because their measured bandwidths are similar enough already. + ## A linear scaling method If scaling is required, here is a simple linear bandwidth scaling @@ -47,6 +50,7 @@ Now, the total scaled bandwidth in the upcoming vote is approximately equal to the quota. + ## Quota changes If all generators are using scaling, the quota can be gradually @@ -56,6 +60,7 @@ consensus diffs and compressed consensuses. But if the relay quota is too small, some relays may be over- or under-weighted. + ## Torflow aggregation Torflow implements two methods to compute the bandwidth values from the @@ -125,4 +130,3 @@ r_strm_i = bw_i / bw_avg_strm <> - diff --git a/spec/bandwidth-file-spec/scope-preliminaries.md b/spec/bandwidth-file-spec/scope-preliminaries.md index 969c971..267d2cb 100644 --- a/spec/bandwidth-file-spec/scope-preliminaries.md +++ b/spec/bandwidth-file-spec/scope-preliminaries.md @@ -1,4 +1,5 @@ + # Scope and preliminaries This document describes the format of Tor's Bandwidth File, version @@ -19,6 +20,7 @@ NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and RFC 2119. + ## Acknowledgements The original bandwidth generator (Torflow) and format was @@ -31,6 +33,7 @@ Nick Mathewson (nickm) Iain Learmonth (irl) + ## Outline The Tor directory protocol (dir-spec.txt [3]) sections 3.4.1 @@ -42,6 +45,7 @@ capacities and is produced by bandwidth generators, previously known as bandwidth scanners. + ## Format Versions 1.0.0 - The legacy Bandwidth File format @@ -89,4 +93,3 @@ contains any KeyValue lines after the Timestamp. * 1 for relays with "unmeasured=1", and * the relay's measured and scaled bandwidth when "under_min_report=1". ``` - diff --git a/spec/bridgedb-spec.md b/spec/bridgedb-spec.md index fddd8bd..2080e5d 100644 --- a/spec/bridgedb-spec.md +++ b/spec/bridgedb-spec.md @@ -21,6 +21,7 @@ Table of Contents ``` + # Preliminaries This document specifies how BridgeDB processes bridge descriptor files @@ -33,6 +34,7 @@ specify current behavior as of August 2013, not to specify ideal behavior. + # Importing bridge network statuses and bridge descriptors BridgeDB learns about bridges by parsing bridge network statuses, @@ -48,6 +50,7 @@ files: the operator needs to make sure that these documents have come from a Tor instance that did the validation for us. + ## Parsing bridge network statuses Bridge network status documents contain the information of which bridges @@ -75,6 +78,7 @@ bridges given out should contain at least a given number of bridges with these flags. + ## Parsing bridge descriptors BridgeDB learns about a bridge's most recent IP address and OR port @@ -116,6 +120,7 @@ the bridge network status parsed before, it does not add that bridge to the set of bridges to be given out to bridge users. + ## Parsing extra-info documents BridgeDB learns if a bridge supports a pluggable transport by parsing @@ -142,6 +147,7 @@ fingerprint. Arguments are comma-separated and are of the form k=v,k=v. Bridges that do not have an associated extra-info entry are not invalid. + # Assigning bridges to distributors A "distributor" is a mechanism by which bridges are given (or not @@ -163,6 +169,7 @@ distributors, even if probabilities for assigning bridges to distributors change or distributors are disabled entirely. + # Giving out bridges upon requests Upon receiving a client request, a BridgeDB distributor provides a @@ -179,6 +186,7 @@ transport type, bridge relay flag, or country in which the bridge should not be blocked. + # Selecting bridges to be given out based on IP addresses ```text @@ -282,6 +290,7 @@ total." To do this, BridgeDB combines to the results: ``` + # Selecting bridges to be given out based on email addresses ```text @@ -356,6 +365,7 @@ proceeds as follows: ``` + # Selecting unallocated bridges to be stored in file buckets # Kaner should have a look at this section. -NM @@ -384,6 +394,7 @@ proceeds as follows: ``` + # Displaying Bridge Information After bridges are selected using one of the methods described in @@ -409,6 +420,7 @@ Previously, each line was prepended with the "bridge" keyword, such as > See the commit message for b70347a9c5fd769c6d5d0c0eb5171ace2999a736. + # Writing bridge assignments for statistics BridgeDB can be configured to write bridge assignments to disk for @@ -437,4 +449,3 @@ value to indicate to which IP address area the bridge is returned. The "unallocated" distributor allows the key "bucket" with the file bucket name as value to indicate which file bucket a bridge is assigned to. - diff --git a/spec/cert-spec.md b/spec/cert-spec.md index 36c9c0b..79ee9b0 100644 --- a/spec/cert-spec.md +++ b/spec/cert-spec.md @@ -18,6 +18,7 @@ Table of Contents ``` + # Scope and Preliminaries This document describes a certificate format that Tor uses for @@ -32,6 +33,7 @@ proposal 220, and were first supported by Tor in Tor version 0.2.7.2-alpha. + ## Signing All signatures here, unless otherwise specified, are computed @@ -42,15 +44,18 @@ signed document is prefixed with a personalization string, which will be different in each case. + ## Integer encoding Network byte order (big-endian) is used to encode all integer values in Ed25519 certificates unless explicitly specified otherwise. + # Document formats + ## Ed25519 Certificates When generating a signing key, we also generate a certificate for it. @@ -107,9 +112,11 @@ the certificate up until "SIGNATURE" (that is, signing sizeof(ed25519_cert) - 64 bytes). + ## Basic extensions + ### Signed-with-ed25519-key extension [type 04] In several places, it's desirable to bundle the key signing a @@ -126,6 +133,7 @@ When this extension is present, it MUST match the key used to sign the certificate. + ## RSA->Ed25519 cross-certificate Certificate type [07] (Cross-certification of Ed25519 identity @@ -153,6 +161,7 @@ acts with the authority of the RSA key that signed this certificate." + ## List of certificate types (CERT_TYPE field) The values marked with asterisks are not types corresponding to @@ -194,11 +203,13 @@ certificate type enumeration of in our Ed25519 certificates. ``` + ## List of extension types [04] - signed-with-ed25519-key (section 2.2.1) + ## List of signature prefixes We describe various documents as being signed with a prefix. Here @@ -207,6 +218,7 @@ are those prefixes: "Tor router descriptor signature v1" (see dir-spec.txt) + ## List of certified key types (CERT_KEY_TYPE field) ```text @@ -220,4 +232,3 @@ are those prefixes: "01" for all types of certified key. Implementations SHOULD allow "01" in this position, and infer the actual key type from the CERT_TYPE field.) - diff --git a/spec/control-spec-intro.md b/spec/control-spec-intro.md index 209d40c..08ea28e 100644 --- a/spec/control-spec-intro.md +++ b/spec/control-spec-intro.md @@ -90,4 +90,3 @@ Table of Contents 5.5.4. Phases in Bootstrap Stage 3 5.6 Bootstrap phases reported by older versions of Tor ``` - diff --git a/spec/control-spec/commands.md b/spec/control-spec/commands.md index 254633e..f59a548 100644 --- a/spec/control-spec/commands.md +++ b/spec/control-spec/commands.md @@ -1,9 +1,11 @@ + # Commands All commands are case-insensitive, but most keywords are case-sensitive. + ## SETCONF Change the value of one or more configuration variables. The syntax is: @@ -42,6 +44,7 @@ options with a single SETCONF command (e.g. SETCONF ORPort=443 ORListenAddress=9001). + ## RESETCONF Remove all settings for a given configuration option entirely, assign @@ -54,6 +57,7 @@ its default. The syntax is: Otherwise it behaves like SETCONF above. + ## GETCONF Request the value of zero or more configuration variable(s). @@ -91,6 +95,7 @@ virtual keyword to get all HiddenServiceDir, HiddenServicePort, HiddenServiceVersion, and HiddenserviceAuthorizeClient option settings. + ## SETEVENTS Request the server to inform the client about interesting events. The @@ -100,7 +105,7 @@ syntax is: EventCode = 1*(ALPHA / "_") (see section 4.1.x for event types) -Any events *not* listed in the SETEVENTS line are turned off; thus, sending +Any events _not_ listed in the SETEVENTS line are turned off; thus, sending SETEVENTS with an empty body turns off all event reporting. The server responds with a "250 OK" reply on success, and a "552 @@ -117,6 +122,7 @@ always-on in Tor 0.2.2.1-alpha and later. Each event is described in more detail in Section 4.1. + ## AUTHENTICATE Sent from the client to the server. The syntax is: @@ -163,6 +169,7 @@ case, the controller should just send "AUTHENTICATE" CRLF. connection after an authentication failure.) + ## SAVECONF Sent from the client to the server. The syntax is: @@ -185,6 +192,7 @@ See also the "getinfo config-can-saveconf" command, to tell if the FORCE flag will be required. (Also introduced in 0.3.1.1-alpha.) + ## SIGNAL Sent from the client to the server. The syntax is: @@ -239,6 +247,7 @@ signal that have them. ``` + ## MAPADDRESS Sent from the client to the server. The syntax is: @@ -325,6 +334,7 @@ Example: ``` + ## GETINFO Sent from the client to the server. The syntax is as for GETCONF: @@ -963,6 +973,7 @@ ReplyLine. If a value fits on a single line, the format is: ``` + ## EXTENDCIRCUIT Sent from the client to the server. The format is: @@ -993,6 +1004,7 @@ message body consisting of the CircuitID of the (maybe newly created) circuit. The syntax is "250" SP "EXTENDED" SP CircuitID CRLF. + ## SETCIRCUITPURPOSE Sent from the client to the server. The format is: @@ -1002,6 +1014,7 @@ Sent from the client to the server. The format is: This changes the circuit's purpose. See EXTENDCIRCUIT above for details. + ## SETROUTERPURPOSE Sent from the client to the server. The format is: @@ -1016,6 +1029,7 @@ NOTE: This command was disabled and made obsolete as of Tor historical interest. + ## ATTACHSTREAM Sent from the client to the server. The syntax is: @@ -1057,6 +1071,7 @@ yet, in which case Tor will detach the stream from its current circuit before proceeding with the new attach request.} + ## POSTDESCRIPTOR Sent from the client to the server. The syntax is: @@ -1085,6 +1100,7 @@ whose body explains why the server was not added. If the descriptor is added, Tor replies with "250 OK". + ## REDIRECTSTREAM Sent from the client to the server. The syntax is: @@ -1102,6 +1118,7 @@ a circuit. Tor replies with "250 OK" on success. + ## CLOSESTREAM Sent from the client to the server. The syntax is: @@ -1117,6 +1134,7 @@ Tor replies with "250 OK" on success, or a 512 if there aren't enough arguments, or a 552 if it doesn't recognize the StreamID or reason. + ## CLOSECIRCUIT The syntax is: @@ -1136,12 +1154,14 @@ Tor replies with "250 OK" on success, or a 512 if there aren't enough arguments, or a 552 if it doesn't recognize the CircuitID. + ## QUIT Tells the server to hang up on this controller connection. This command can be used before authenticating. + ## USEFEATURE Adding additional features to the control protocol sometimes will break @@ -1191,6 +1211,7 @@ EXTENDED_EVENTS ``` + ## RESOLVE The syntax is @@ -1209,6 +1230,7 @@ need to listen for ADDRMAP events; see 4.1.7 below. [Added in Tor 0.2.0.3-alpha] + ## PROTOCOLINFO The syntax is: @@ -1285,6 +1307,7 @@ only once!) before AUTHENTICATE.] [PROTOCOLINFO was not supported before Tor 0.2.0.5-alpha.] + ## LOADCONF The syntax is: @@ -1298,6 +1321,7 @@ it had been read from disk. [LOADCONF was added in Tor 0.2.1.1-alpha.] + ## TAKEOWNERSHIP The syntax is: @@ -1347,6 +1371,7 @@ should 'own' that Tor process: ``` + ## AUTHCHALLENGE The syntax is: @@ -1403,6 +1428,7 @@ used (but only once!) before AUTHENTICATE.] [AUTHCHALLENGE was added in Tor 0.2.3.13-alpha.] + ## DROPGUARDS The syntax is: @@ -1417,6 +1443,7 @@ Tor replies with "250 OK" on success. [DROPGUARDS was added in Tor 0.2.5.2-alpha.] + ## HSFETCH The syntax is: @@ -1473,6 +1500,7 @@ Examples are: [HS v3 support added 0.4.1.1-alpha] + ## ADD_ONION The syntax is: @@ -1655,6 +1683,7 @@ Examples: [ClientV3Auth support added 0.4.6.1-alpha] + ## DEL_ONION The syntax is: @@ -1686,6 +1715,7 @@ number of arguments, or a 552 if it doesn't recognize the ServiceID. [HS v3 support added 0.3.3.1-alpha] + ## HSPOST The syntax is: @@ -1728,6 +1758,7 @@ Examples are: ``` + ## ONION_CLIENT_AUTH_ADD The syntax is: @@ -1778,6 +1809,7 @@ On success, "250 OK" is returned. Otherwise, the following error codes exist: ``` + ## ONION_CLIENT_AUTH_REMOVE The syntax is: @@ -1799,6 +1831,7 @@ On success "250 OK" is returned. Otherwise, the following error codes exist: ``` + ## ONION_CLIENT_AUTH_VIEW The syntax is: @@ -1839,6 +1872,7 @@ On success "250 OK" is returned. Otherwise, the following error codes exist: [ONION_CLIENT_AUTH_VIEW was added in Tor 0.4.3.1-alpha] + ## DROPOWNERSHIP The syntax is: @@ -1859,6 +1893,7 @@ ownership. [DROPOWNERSHIP was added in Tor 0.4.0.0-alpha] + ## DROPTIMEOUTS ```text @@ -1873,4 +1908,3 @@ Tor replies with "250 OK" on success. Tor also emits the BUILDTIMEOUT_SET RESET event right after this "250 OK". [DROPTIMEOUTS was added in Tor 0.4.5.0-alpha.] - diff --git a/spec/control-spec/implementation-notes.md b/spec/control-spec/implementation-notes.md index 8340a94..b93f5d6 100644 --- a/spec/control-spec/implementation-notes.md +++ b/spec/control-spec/implementation-notes.md @@ -1,7 +1,9 @@ + # Implementation notes + ## Authentication If the control port is open and no authentication operation is enabled, Tor @@ -50,6 +52,7 @@ This is then encoded in hexadecimal, prefixed by the indicator sequence ``` You can generate the salt of a password by calling + ``` 'tor --hash-password ' ``` @@ -60,7 +63,8 @@ secret that was used to generate the password, either as a quoted string or encoded in hexadecimal. -## Don't let the buffer get too big. + +## Don't let the buffer get too big With old versions of Tor (before 0.2.0.16-alpha), if you ask for lots of events, and 16MB of them queue up on the buffer, the Tor @@ -71,7 +75,8 @@ if you leave huge numbers of events unread, Tor may still run out of memory, so you should still be careful about buffer size. -## Backward compatibility with v0 control protocol. + +## Backward compatibility with v0 control protocol The 'version 0' control protocol was replaced in Tor 0.1.1.x. Support was removed in Tor 0.2.0.x. Every non-obsolete version of Tor now @@ -84,6 +89,7 @@ Tor used to check whether the third octet of the first command is zero. This compatibility was removed in Tor 0.1.2.16 and 0.2.0.4-alpha. + ## Tor config options for use by controllers Tor provides a few special configuration options for use by controllers. @@ -181,7 +187,8 @@ __AllDirActionsPrivate ``` -## Phases from the Bootstrap status event. + +## Phases from the Bootstrap status event ```text [For the bootstrap phases reported by Tor prior to 0.4.0.x, see @@ -201,7 +208,8 @@ Future Tors MAY revisit earlier phases, for example, if the network fails. -### Overview of Bootstrap reporting. + +### Overview of Bootstrap reporting Bootstrap phases can be viewed as belonging to one of three stages: @@ -235,7 +243,8 @@ connections to relays or bridges that it did not connect to in Stage 1. -### Phases in Bootstrap Stage 1. + +### Phases in Bootstrap Stage 1 Phase 0: tag=starting summary="Starting" @@ -311,7 +320,8 @@ established. ``` -### Phases in Bootstrap Stage 2. + +### Phases in Bootstrap Stage 2 ```text Phase 20: @@ -397,7 +407,8 @@ indicate partial steps. ``` -### Phases in Bootstrap Stage 3. + +### Phases in Bootstrap Stage 3 ```text Phase 76: @@ -509,6 +520,7 @@ application connections now. ``` + ## Bootstrap phases reported by older versions of Tor These phases were reported by Tor older than 0.4.0.x. For newer @@ -707,4 +719,3 @@ application requesting an internal circuit to hidden services at ".onion" addresses. If a future consensus contains Exits, exit circuits may become available.] - diff --git a/spec/control-spec/message-format.md b/spec/control-spec/message-format.md index 182f8cc..2e3115b 100644 --- a/spec/control-spec/message-format.md +++ b/spec/control-spec/message-format.md @@ -1,7 +1,9 @@ + # Message format + ## Description format The message formats listed below use ABNF as described in RFC 2234. @@ -23,6 +25,7 @@ accept LF. Tor, however, MUST NOT generate LF instead of CRLF. Controllers SHOULD always send CRLF. + ### Notes on an escaping bug CString = DQUOTE *qcontent DQUOTE @@ -60,6 +63,7 @@ Tor controller; Tor should parse input QuotedStrings from the controller correctly. + ## Commands from controller to Tor ```text @@ -76,6 +80,7 @@ multi-line commands that it does not recognize.) Specific commands and their arguments are described below in section 3. + ## Replies from Tor to the controller ```text @@ -103,6 +108,7 @@ versions of Tor should be prepared to get multi-line AsyncReplies with the final line (usually "650 OK") omitted.] + ## General-use tokens ; CRLF means, "the ASCII Carriage Return character (decimal value 13) @@ -177,4 +183,3 @@ ISOTime2Frac = IsoTime2 [ "." 1*DIGIT ] ; Numbers LeadingDigit = "1" - "9" UInt = LeadingDigit *Digit - diff --git a/spec/control-spec/protocol-outline.md b/spec/control-spec/protocol-outline.md index b0dd9a9..feeef1b 100644 --- a/spec/control-spec/protocol-outline.md +++ b/spec/control-spec/protocol-outline.md @@ -1,4 +1,5 @@ + # Protocol outline TC is a bidirectional message-based protocol. It assumes an underlying @@ -20,6 +21,7 @@ messages to the client indefinitely far into the future. Such Servers respond to messages in the order messages are received. + ## Forward-compatibility This is an evolving protocol; new client and server behavior will be @@ -67,4 +69,3 @@ elements in these places. There are two ways that we do this: must tolerate unexpected values. The only exception would be if there were an explicit statement that no future values will ever be added.) ``` - diff --git a/spec/control-spec/replies.md b/spec/control-spec/replies.md index 5fb71e7..c55635c 100644 --- a/spec/control-spec/replies.md +++ b/spec/control-spec/replies.md @@ -1,4 +1,5 @@ + # Replies Reply codes follow the same 3-character format as used by SMTP, with the @@ -74,6 +75,7 @@ Unless specified to have specific contents, the human-readable messages in error replies should not be relied upon to match those in this document. + ## Asynchronous events These replies can be sent after a corresponding SETEVENTS command has been @@ -138,6 +140,7 @@ controllers have been fixed. At some point this "SHOULD NOT" might become a "MUST NOT". + ### Circuit status changed The syntax is: @@ -297,6 +300,7 @@ The syntax is: ``` + ### Stream status changed The syntax is: @@ -452,6 +456,7 @@ to talk to itself. ``` + ### OR Connection status changed The syntax is: @@ -524,6 +529,7 @@ events. ``` + ### Bandwidth used in the last second The syntax is: @@ -541,6 +547,7 @@ we may also include a breakdown of the connection types that used bandwidth this second (not implemented yet).] + ### Log messages The syntax is: @@ -562,6 +569,7 @@ Control port message trace debug logs are never sent as control port log events, to avoid modifying control output when debugging. + ### New descriptors available This event is generated when new router descriptors (not microdescs or @@ -578,6 +586,7 @@ Syntax: ``` + ### New Address mapping These events are generated when a new address mapping is entered in @@ -620,6 +629,7 @@ StreamId is the global stream identifier of the stream or circuit from which the address was resolved. + ### Descriptors uploaded to us in our role as authoritative dirserver [NOTE: This feature was removed in Tor 0.3.2.1-alpha.] @@ -645,6 +655,7 @@ complaining. The Message field is a human-readable string explaining why we chose the Action. (It doesn't contain newlines.) + ### Our descriptor changed Syntax: @@ -654,6 +665,7 @@ Syntax: [First added in 0.1.2.2-alpha.] + ### Status events Status events (STATUS_GENERAL, STATUS_CLIENT, and STATUS_SERVER) are sent @@ -1098,6 +1110,7 @@ Actions for STATUS_GENERAL events can be as follows: ``` + ### Our set of guard nodes has changed Syntax: @@ -1133,6 +1146,7 @@ The Status values are: ``` + ### Network status has changed Syntax: @@ -1148,6 +1162,7 @@ down in our local status, for example based on connection attempts. [First added in 0.1.2.3-alpha] + ### Bandwidth used on an application stream The syntax is: @@ -1176,6 +1191,7 @@ apply only to streams entering Tor (such as on a SOCKSPort, TransPort, or so on). They are not generated for exiting streams. + ### Per-country client stats The syntax is: @@ -1212,6 +1228,7 @@ algorithm is specified in the description of "geoip-client-origins" in dir-spec.txt. + ### New consensus networkstatus has arrived The syntax is: @@ -1229,6 +1246,7 @@ relay *not* mentioned in this list is implicitly no longer recommended. [First added in 0.2.1.13-alpha] + ### New circuit buildtime has been set The syntax is: @@ -1270,6 +1288,7 @@ this value to the nearest second before using it. [First added in 0.2.2.7-alpha] + ### Signal received The syntax is: @@ -1293,6 +1312,7 @@ generate any event. [First added in 0.2.3.1-alpha] + ### Configuration changed The syntax is: @@ -1310,6 +1330,7 @@ signal). KEYWORD and VALUE specify the configuration option that was changed. Undefined configuration options contain only the KEYWORD. + ### Circuit status changed slightly The syntax is: @@ -1338,6 +1359,7 @@ Other fields are as specified in section 4.1.1 above. [First added in 0.2.3.11-alpha] + ### Pluggable transport launched The syntax is: @@ -1355,6 +1377,7 @@ The syntax is: ``` + ### Bandwidth used on an OR or DIR or EXIT connection The syntax is: @@ -1387,6 +1410,7 @@ These events are only generated if TestingTorNetwork is set. [First added in 0.2.5.2-alpha] + ### Bandwidth used by all streams attached to a circuit The syntax is: @@ -1461,6 +1485,7 @@ were added in Tor 0.3.4.0-alpha] [SS, CWND, RTT, and MIN_RTT were added in Tor 0.4.7.5-alpha] + ### Per-circuit cell stats The syntax is: @@ -1532,6 +1557,7 @@ cell. These events are only generated if TestingTorNetwork is set. [First added in 0.2.5.2-alpha] + ### Token buckets refilled The syntax is: @@ -1580,6 +1606,7 @@ These events are only generated if TestingTorNetwork is set. [First added in 0.2.5.2-alpha] + ### HiddenService descriptors The syntax is: @@ -1653,6 +1680,7 @@ The syntax is: ``` + ### HiddenService descriptors content The syntax is: @@ -1690,6 +1718,7 @@ this event will reply either the descriptor's content or an empty one. [HS v3 support added 0.3.3.1-alpha] + ### Network liveness has changed Syntax: @@ -1705,6 +1734,7 @@ Syntax: ``` + ### Pluggable Transport Logs Syntax: @@ -1733,6 +1763,7 @@ Syntax: ``` + ### Pluggable Transport Status Syntax: @@ -1761,4 +1792,3 @@ Syntax: [PT_STATUS was added in Tor 0.4.0.1-alpha] ``` - diff --git a/spec/control-spec/scope.md b/spec/control-spec/scope.md index 8684e3e..9d5b3d3 100644 --- a/spec/control-spec/scope.md +++ b/spec/control-spec/scope.md @@ -1,4 +1,5 @@ + # Scope This document describes an implementation-specific protocol that is used @@ -19,4 +20,3 @@ versions in the 0.1.1.x series and later.) "OPTIONAL" in this document are to be interpreted as described in RFC 2119. ``` - diff --git a/spec/dir-list-spec.md b/spec/dir-list-spec.md index e6c1e0c..0dbb750 100644 --- a/spec/dir-list-spec.md +++ b/spec/dir-list-spec.md @@ -28,6 +28,7 @@ Table of Contents ``` + # Scope and Preliminaries This document describes the format of Tor's directory lists, which are @@ -58,6 +59,7 @@ Tor 0.2.8.1-alpha through Tor 0.3.1.9. We call this format version 1. Stem and Relay Search have parsers for this legacy format. + ## Format Overview A directory list is a C code fragment containing an array of C string @@ -84,6 +86,7 @@ The order of directory entries and data fields is not significant, except where noted below. + ## Acknowledgements The original fallback directory script and format was created by @@ -97,9 +100,10 @@ This specification was revised after feedback from: ``` + ## Format Versions -The directory list format uses semantic versioning: https://semver.org +The directory list format uses semantic versioning: ```text In particular: @@ -120,6 +124,7 @@ The directory list format uses semantic versioning: https://semver.org ``` + ## Future Plans Tor also has an auth_dirs.inc file, but it is not yet in this format. @@ -135,6 +140,7 @@ We need to write a short proposal, and make some changes to tor and the fallback update script. See #24839 for details. + # Format Details Directory lists contain the following sections: @@ -148,6 +154,7 @@ Directory lists contain the following sections: ``` + ## Nonterminals The following nonterminals are defined in the Onionoo details document @@ -206,12 +213,14 @@ specification in dir-spec.txt: ``` + ## List Header The list header consists of a number of key-value pairs, embedded in C-style comments. + ### List Header Format "/*" SP+ "type=" Keyword SP+ "*/" SP* NL @@ -286,6 +295,7 @@ C-style comments. ``` + ## List Generation The list generation information consists of human-readable prose @@ -300,6 +310,7 @@ Future releases may arbitrarily change the content of this section. Parsers MUST NOT rely on a version increment when the format changes. + ### List Generation Format In general, parsers MUST NOT rely on the format of this section. @@ -316,6 +327,7 @@ There MUST NOT be any section separators in the list generation section, other than the terminating section separator. + ## Directory Entry A directory entry consists of a C string constant, and one or more @@ -329,6 +341,7 @@ these lists after parsing them, and applies the DirAuthorityFallbackRate to their weights.) + ### Directory Entry Format ```text @@ -445,12 +458,14 @@ to their weights.) ``` + # Usage Considerations This section contains recommended library behaviours. It does not affect the format of directory lists. + ## Caching The fallback list typically changes once every 6-12 months. The data in @@ -463,19 +478,20 @@ lists by default every time they are deployed or executed. The latest fallback list can be retrieved from: -https://gitweb.torproject.org/tor.git/plain/src/or/fallback_dirs.inc + Libraries MUST NOT rely on the availability of the server that hosts these lists. The list can also be retrieved using: -git clone https://git.torproject.org/tor.git +git clone If you just want the latest list, you may wish to perform a shallow clone. + ## Retrieving Directory Information Some libraries retrieve directory documents directly from the Tor @@ -503,6 +519,7 @@ schedule. Fallbacks update their own directory information at random intervals, see dir-spec for details. + ## Fallback Reliability The fallback list is typically regenerated when the fallback failure @@ -514,30 +531,33 @@ few fallback queries fail. For example, Tor clients try 3-4 fallbacks before trying an authority. + ## Sample Data A sample version 2.0.0 fallback list is available here: -https://trac.torproject.org/projects/tor/raw-attachment/ticket/22759/fallback_dirs_new_format_version.4.inc + A sample transitional version 2.0.0 fallback list is available here: -https://raw.githubusercontent.com/teor2345/tor/fallback-format-2-v4/src/or/fallback_dirs.inc + + ### Sample Fallback List Header -/* type=fallback */ +/*type=fallback */ /* version=2.0.0 */ -/* ===== */ +/* =====*/ + ### Sample Fallback List Generation -/* Whitelist & blacklist excluded 1326 of 1513 candidates. */ +/*Whitelist & blacklist excluded 1326 of 1513 candidates. */ /* Checked IPv4 DirPorts served a consensus within 15.0s. */ /* -Final Count: 151 (Eligible 187, Target 392 (1963 * 0.20), Max 200) +Final Count: 151 (Eligible 187, Target 392 (1963* 0.20), Max 200) Excluded: 36 (Same Operator 27, Failed/Skipped Download 9, Excess 0) Bandwidth Range: 1.3 - 40.0 MByte/s */ @@ -552,10 +572,11 @@ URL: https:onionoo.torproject.orguptime?first_seen_days=30-&flag=V2Dir&type=rela /* ===== */ + ### Sample Fallback Entries "176.10.104.240:80 orport=443 id=0111BA9B604669E636FFD5B503F382A4B7AD6E80" -/* nickname=foo */ +/*nickname=foo */ /* extrainfo=1 */ /* ===== */ , @@ -563,6 +584,5 @@ URL: https:onionoo.torproject.orguptime?first_seen_days=30-&flag=V2Dir&type=rela " ipv6=[2a01:4f8:162:51e2::2]:9001" /* nickname= */ /* extrainfo=0 */ -/* ===== */ +/* =====*/ , - diff --git a/spec/dir-spec-intro.md b/spec/dir-spec-intro.md index 9a4ef78..71b1090 100644 --- a/spec/dir-spec-intro.md +++ b/spec/dir-spec-intro.md @@ -67,4 +67,3 @@ Table of Contents D. Inferring missing proto lines. E. Limited ed diff format ``` - diff --git a/spec/dir-spec/accepting-server-descriptor-extra-info-document.md b/spec/dir-spec/accepting-server-descriptor-extra-info-document.md index aa69ea9..b507b9c 100644 --- a/spec/dir-spec/accepting-server-descriptor-extra-info-document.md +++ b/spec/dir-spec/accepting-server-descriptor-extra-info-document.md @@ -1,4 +1,5 @@ + ## Accepting server descriptor and extra-info document uploads When a router posts a signed descriptor to a directory authority, the @@ -49,4 +50,3 @@ the authority again checks it for well-formedness and correct signature, and checks that its matches the extra-info-digest in some router descriptor that it believes is currently useful. If so, it accepts it and stores it and serves it as requested. If not, it drops it. - diff --git a/spec/dir-spec/assigning-flags-vote.md b/spec/dir-spec/assigning-flags-vote.md index 309b025..84d024b 100644 --- a/spec/dir-spec/assigning-flags-vote.md +++ b/spec/dir-spec/assigning-flags-vote.md @@ -1,4 +1,5 @@ + ### Assigning flags in a vote (This section describes how directory authorities choose which status @@ -131,7 +132,7 @@ published time on the descriptor is over 18 hours in the past. (This flag was added in 0.4.0.1-alpha.) "Sybil" -- authorities SHOULD NOT accept more than 2 relays on a single IP. -If this happens, the authority *should* vote for the excess relays, but +If this happens, the authority _should_ vote for the excess relays, but should omit the Running or Valid flags and instead should assign the "Sybil" flag. When there are more than 2 (or AuthDirMaxServersPerAddr) relays to choose from, authorities should first prefer authorities to non-authorities, @@ -169,4 +170,3 @@ accept not for all addresses, ignoring all rejects for private netblocks. "Most" addresses are permitted if no more than 2^25 IPv4 addresses (two /8 networks) were blocked. The list is encoded as described in section 3.8.2. - diff --git a/spec/dir-spec/client-operation.md b/spec/dir-spec/client-operation.md index 41942c6..3f19a61 100644 --- a/spec/dir-spec/client-operation.md +++ b/spec/dir-spec/client-operation.md @@ -1,10 +1,12 @@ + # Client operation Every Tor that is not a directory server (that is, those that do not have a DirPort set) implements this section. + ## Downloading network-status documents Each client maintains a list of directory authorities. Insofar as @@ -87,6 +89,7 @@ most recent consensus it has if that consensus is "reasonably live". A "reasonably live" consensus is one that expired less than 24 hours ago. + ## Downloading server descriptors or microdescriptors Clients try to have the best descriptor for each router. A descriptor is @@ -160,6 +163,7 @@ mentioned in any consensus for a week. Future clients might cache them for longer or shorter times. + ## Downloading extra-info documents Any client that uses extra-info documents should implement this @@ -175,6 +179,7 @@ documents are missing. Clients try to download from caches. We follow the same splitting and back-off rules as in section 5.2. + ## Using directory information [XXX This subsection really belongs in path-spec.txt, not here. -KL] @@ -185,7 +190,8 @@ to decide which relays to use and what their keys are likely to be. above.) -### Choosing routers for circuits. + +### Choosing routers for circuits Circuits SHOULD NOT be built until the client has enough directory information: a live consensus network status [XXXX fallback?] and @@ -214,22 +220,26 @@ These flags are used as follows: ``` + ### Managing naming (This section is removed; authorities no longer assign the 'Named' flag.) + ### Software versions An implementation of Tor SHOULD warn when it has fetched a consensus network-status, and it is running a software version not listed. -### Warning about a router's status. + +### Warning about a router's status (This section is removed; authorities no longer assign the 'Named' flag.) + ## Retrying failed downloads This section applies to caches as well as to clients. @@ -240,7 +250,7 @@ time before retrying the download. To determine the amount of time to wait, clients use a randomized exponential backoff algorithm. (Specifically, they use a variation of the "decorrelated jitter" algorithm from -https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ .) + .) The specific formula used to compute the 'i+1'th delay is: @@ -289,4 +299,3 @@ downloading from. Current base_delay values are: Other objects, as client: 0 (TestingClientDownloadInitialDelay) ``` - diff --git a/spec/dir-spec/computing-microdescriptors.md b/spec/dir-spec/computing-microdescriptors.md index 29ddd48..1b88e42 100644 --- a/spec/dir-spec/computing-microdescriptors.md +++ b/spec/dir-spec/computing-microdescriptors.md @@ -1,4 +1,5 @@ + ## Computing microdescriptors Microdescriptors are a stripped-down version of server descriptors @@ -166,4 +167,3 @@ their routers: they only learn a hash of the RSA identity key. This is all they need to confirm the actual identity key when doing a TLS handshake, and all they need to put the identity key digest in their CREATE cells.) - diff --git a/spec/dir-spec/consensus-negotiation-timeline.md b/spec/dir-spec/consensus-negotiation-timeline.md index 9e17724..59f9ced 100644 --- a/spec/dir-spec/consensus-negotiation-timeline.md +++ b/spec/dir-spec/consensus-negotiation-timeline.md @@ -1,5 +1,6 @@ -# Consensus-negotiation timeline. + +# Consensus-negotiation timeline ```text Period begins: this is the Published time. @@ -13,4 +14,3 @@ Valid-after/valid-until switchover ``` - diff --git a/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md b/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md index 471fbdb..a676228 100644 --- a/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md +++ b/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md @@ -1,4 +1,5 @@ + # Converting a curve25519 public key to an ed25519 public key Given an X25519 key, that is, an affine point (u,v) on the @@ -74,4 +75,3 @@ Then, knowing the intended sign of the Edwards x-coordinate, one may recover said x-coordinate by computing: x = (u/v) * sqrt(-a - 2) - diff --git a/spec/dir-spec/creating-key-certificates.md b/spec/dir-spec/creating-key-certificates.md index 4c25867..56b715b 100644 --- a/spec/dir-spec/creating-key-certificates.md +++ b/spec/dir-spec/creating-key-certificates.md @@ -1,4 +1,5 @@ + ## Creating key certificates Key certificates consist of the following items: @@ -87,4 +88,3 @@ initial item "dir-key-certificate-version" and the final item Authorities MUST generate a new signing key and corresponding certificate before the key expires. - diff --git a/spec/dir-spec/directory-authority-operation-formats.md b/spec/dir-spec/directory-authority-operation-formats.md index 9e171cf..bd487f6 100644 --- a/spec/dir-spec/directory-authority-operation-formats.md +++ b/spec/dir-spec/directory-authority-operation-formats.md @@ -1,4 +1,5 @@ + # Directory authority operation and formats Every authority has two keys used in this protocol: a signing key, and @@ -7,4 +8,3 @@ key used in their role as a router and by earlier versions of the directory protocol.) The identity key is used from time to time to sign new key certificates using new signing keys; it is very sensitive. The signing key is used to sign key certificates and status documents. - diff --git a/spec/dir-spec/directory-cache-operation.md b/spec/dir-spec/directory-cache-operation.md index ed14fc2..1aa3f4b 100644 --- a/spec/dir-spec/directory-cache-operation.md +++ b/spec/dir-spec/directory-cache-operation.md @@ -1,9 +1,11 @@ + # Directory cache operation All directory caches implement this section, except as noted. + ## Downloading consensus status documents from directory authorities All directory caches try to keep a recent @@ -33,6 +35,7 @@ length. Caches serve all consensus flavors from the same locations as the directory authorities. + ## Downloading server descriptors from directory authorities Periodically (currently, every 10 seconds), directory caches check @@ -57,6 +60,7 @@ descriptor seemed newest.) [XXXX define recent] + ## Downloading microdescriptors from directory authorities Directory mirrors should fetch, cache, and serve each microdescriptor @@ -80,6 +84,7 @@ the fetch URL or the hashes from the consensus, respectively). can be retrieved in a single request.) + ## Downloading extra-info documents from directory authorities Any cache that chooses to cache extra-info documents should implement this @@ -93,6 +98,7 @@ documents are missing. Caches download from authorities. We follow the same splitting and back-off rules as in section 4.2. + ## Consensus diffs Instead of downloading an entire consensus, clients may download @@ -105,6 +111,7 @@ record of older consensuses. advertised with the DirCache protocol version "2" or later.) + ### Consensus diff format Consensus diffs are formatted as follows: @@ -126,7 +133,8 @@ ToDigest in a limited subset of the ed diff format, as specified in appendix E. -### Serving and requesting diffs. + +### Serving and requesting diffs When downloading the current consensus, a client may include an HTTP header of the form @@ -153,7 +161,7 @@ FPRLIST is +-separated list of recognized authority identity fingerprints as in appendix B. + ## Retrying failed downloads See section 5.5 below; it applies to caches as well as clients. - diff --git a/spec/dir-spec/exchanging-votes.md b/spec/dir-spec/exchanging-votes.md index 4bc6740..aac1d4b 100644 --- a/spec/dir-spec/exchanging-votes.md +++ b/spec/dir-spec/exchanging-votes.md @@ -1,4 +1,5 @@ + ## Exchanging votes Authorities divide time into Intervals. Authority administrators SHOULD @@ -50,4 +51,3 @@ This may be the only way for an authority to hear about relays that didn't publish their descriptor to all authorities, and, while it's too late for the authority to include relays in its current vote, it can include them in its next vote. See section 3.6 below for details. - diff --git a/spec/dir-spec/extra-info-document-format.md b/spec/dir-spec/extra-info-document-format.md index 34df991..a821ecc 100644 --- a/spec/dir-spec/extra-info-document-format.md +++ b/spec/dir-spec/extra-info-document-format.md @@ -1,4 +1,5 @@ + ### Extra-info document format Extra-info documents consist of the following items: @@ -588,4 +589,3 @@ detects fd exhaustion only when a socket open fails. A document signature as documented in section 1.3, using the initial item "extra-info" and the final item "router-signature", signed with the router's identity key. - diff --git a/spec/dir-spec/general-use-http-urls.md b/spec/dir-spec/general-use-http-urls.md index 9cd0a02..4d11372 100644 --- a/spec/dir-spec/general-use-http-urls.md +++ b/spec/dir-spec/general-use-http-urls.md @@ -1,4 +1,5 @@ + # General-use HTTP URLs "Fingerprints" in these URLs are base16-encoded SHA1 hashes. @@ -130,4 +131,3 @@ of the ".z" (see section 6.1). Clients SHOULD use upper case letters (A-F) when base16-encoding fingerprints. Servers MUST accept both upper and lower case fingerprints in requests. - diff --git a/spec/dir-spec/inferring-missing-proto-lines.md b/spec/dir-spec/inferring-missing-proto-lines.md index 0c0d530..7468a20 100644 --- a/spec/dir-spec/inferring-missing-proto-lines.md +++ b/spec/dir-spec/inferring-missing-proto-lines.md @@ -1,5 +1,6 @@ -# Inferring missing proto lines. + +# Inferring missing proto lines The directory authorities no longer allow versions of Tor before 0.2.4.18-rc. But right now, there is no version of Tor in the consensus @@ -12,4 +13,3 @@ LinkAuth=1 Microdesc=1-2 Relay=1-2 For Desc, Microdesc and Cons, Tor versions before 0.2.7.stable should be taken to only support version 1. - diff --git a/spec/dir-spec/limited-ed-diff-format.md b/spec/dir-spec/limited-ed-diff-format.md index 35b5200..53fa334 100644 --- a/spec/dir-spec/limited-ed-diff-format.md +++ b/spec/dir-spec/limited-ed-diff-format.md @@ -1,4 +1,5 @@ + # Limited ed diff format We support the following format for consensus diffs. It's a @@ -36,4 +37,3 @@ appended to the diff after the line with the command. A line with just a period (".") ends the block (and is not part of the lines to add). Note that it is impossible to insert a line with just a single dot. - diff --git a/spec/dir-spec/nonterminals-server-descriptors.md b/spec/dir-spec/nonterminals-server-descriptors.md index 256f759..62298cb 100644 --- a/spec/dir-spec/nonterminals-server-descriptors.md +++ b/spec/dir-spec/nonterminals-server-descriptors.md @@ -1,4 +1,5 @@ + ### Nonterminals in server descriptors ```text @@ -29,4 +30,3 @@ ip6 ::= an IPv6 address, surrounded by square brackets. num_ip6_bits ::= an integer between 0 and 128 bool ::= "0" | "1" - diff --git a/spec/dir-spec/outline.md b/spec/dir-spec/outline.md index 0a5c81d..55deb49 100644 --- a/spec/dir-spec/outline.md +++ b/spec/dir-spec/outline.md @@ -1,4 +1,5 @@ + # Outline There is a small set (say, around 5-10) of semi-trusted directory @@ -44,6 +45,7 @@ clients by giving them descriptors nobody else uses. All directory information is uploaded and downloaded with HTTP. + ## What's different from version 2? Clients used to download multiple network status documents, @@ -58,6 +60,7 @@ All of the information in extra-info documents used to be kept in the main descriptors. + ## Document meta-format Server descriptors, directories, and running-routers documents all obey the @@ -81,7 +84,7 @@ More formally: NL = The ascii LF character (hex value 0x0a). Document ::= (Item | NL)+ Item ::= KeywordLine Object? -KeywordLine ::= Keyword (WS Argument)* NL +KeywordLine ::= Keyword (WS Argument)*NL Keyword = KeywordStart KeywordChar* KeywordStart ::= 'A' ... 'Z' | 'a' ... 'z' | '0' ... '9' KeywordChar ::= KeywordStart | '-' @@ -89,7 +92,7 @@ Argument := ArgumentChar+ ArgumentChar ::= any graphical printing ASCII character. WS = (SP | TAB)+ Object ::= BeginLine Base64-encoded-data EndLine -BeginLine ::= "-----BEGIN " Keyword (" " Keyword)* "-----" NL +BeginLine ::= "-----BEGIN " Keyword (" " Keyword)*"-----" NL EndLine ::= "-----END " Keyword (" " Keyword)* "-----" NL A Keyword may not be "-----BEGIN". @@ -156,6 +159,7 @@ Whenever an item DOES NOT allow extra arguments, we will tag it with "no extra arguments". + ## Signing documents Every signable document below is signed in a similar manner, using a @@ -183,6 +187,7 @@ The "Digest" of a document, unless stated otherwise, is its digest *as signed by this signature scheme*. + ## Voting timeline Every consensus document has a "valid-after" (VA) time, a "fresh-until" @@ -246,4 +251,3 @@ VU + 24 hours: Clients will no longer use the consensus at all. VoteSeconds and DistSeconds MUST each be at least 20 seconds; FU-VA and VU-FU MUST each be at least 5 minutes. - diff --git a/spec/dir-spec/router-operation-formats.md b/spec/dir-spec/router-operation-formats.md index 7757a32..cec9a3c 100644 --- a/spec/dir-spec/router-operation-formats.md +++ b/spec/dir-spec/router-operation-formats.md @@ -1,3 +1,3 @@ -# Router operation and formats +# Router operation and formats diff --git a/spec/dir-spec/scope-preliminaries.md b/spec/dir-spec/scope-preliminaries.md index b513c6d..61d2ff7 100644 --- a/spec/dir-spec/scope-preliminaries.md +++ b/spec/dir-spec/scope-preliminaries.md @@ -1,4 +1,5 @@ + # Scope and preliminaries This directory protocol is used by Tor version 0.2.0.x-alpha and later. @@ -21,6 +22,7 @@ NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and RFC 2119. + ## History The earliest versions of Onion Routing shipped with a list of known @@ -86,6 +88,7 @@ Routers began working harder to upload new descriptors only when their contents were substantially changed. + ## Goals of the version 3 protocol Version 3 of the Tor directory protocol tries to solve the following @@ -116,6 +119,7 @@ issues: ``` + ## Some Remaining questions Things we could solve on a v3 timeframe: @@ -136,4 +140,3 @@ Requiring every client to know about every router won't scale forever. Requiring every directory cache to know every router won't scale forever. - diff --git a/spec/dir-spec/server-descriptor-format.md b/spec/dir-spec/server-descriptor-format.md index 9866076..97b7d09 100644 --- a/spec/dir-spec/server-descriptor-format.md +++ b/spec/dir-spec/server-descriptor-format.md @@ -1,4 +1,5 @@ + ### Server descriptor format Server descriptors consist of the following items. @@ -505,4 +506,3 @@ larger than 63. This field was first added in Tor 0.2.9.x. [Before Tor 0.4.5.1-alpha, this field was optional.] - diff --git a/spec/dir-spec/serving-bandwidth-list-files.md b/spec/dir-spec/serving-bandwidth-list-files.md index 531dc2b..1263e43 100644 --- a/spec/dir-spec/serving-bandwidth-list-files.md +++ b/spec/dir-spec/serving-bandwidth-list-files.md @@ -1,4 +1,5 @@ + ### Serving bandwidth list files If an authority has used a bandwidth list file to generate a vote @@ -27,11 +28,13 @@ The standard URLs for bandwidth list files first-appeared in Tor 0.4.0.4-alpha. + ## Downloading missing certificates from other directory authorities XXX when to download certificates. + ## Downloading server descriptors from other directory authorities Periodically (currently, every 10 seconds), directory authorities check @@ -67,6 +70,7 @@ Authorities SHOULD NOT download descriptors for routers that they would immediately reject for reasons listed in section 3.2. + ## Downloading extra-info documents from other directory authorities Periodically, an authority checks whether it is missing any extra-info @@ -77,6 +81,7 @@ documents are missing. We follow the same splitting and back-off rules as in section 3.6. + ## Computing a consensus from a set of votes Given a set of votes, authorities compute the contents of the consensus. @@ -310,7 +315,8 @@ All ties in computing medians are broken in favor of the smaller or earlier item. -#### Deciding which Ids to include. + +#### Deciding which Ids to include This sorting algorithm is used for consensus-method 22 and later. @@ -336,6 +342,7 @@ This sorting algorithm is used for consensus-method 22 and later. ``` + #### Deciding which descriptors to include Deciding which descriptors to include. @@ -354,6 +361,7 @@ that matches the largest set, breaking ties in favor of the most recently published, and then in favor of the smaller server descriptor digest. + ### Forward compatibility Future versions of Tor will need to include new information in the @@ -428,6 +436,7 @@ NOT advertise support for them: "21" -- Did not correctly enable support for ed25519 key collation. + ### Encoding port lists Whether the summary shows the list of accepted ports or the list of @@ -449,6 +458,7 @@ use an accept-style summary and list as much of the port list as is possible within these 1000 bytes. [XXXX be more specific.] + ### Computing Bandwidth Weights Let weight_scale = 10000, or the value of the "bwweightscale" parameter. @@ -631,6 +641,7 @@ Handle bridges and strange exit policies: Wgm=Wgg, Wem=Wee, Weg=Wed + ## Computing consensus flavors Consensus flavors are variants of the consensus that clients can choose @@ -672,6 +683,7 @@ is a string consisting of alphanumeric characters and dashes: "network-status-version" SP version [SP flavor] NL + ### ns consensus The ns consensus flavor is equivalent to the unflavored consensus. @@ -681,6 +693,7 @@ state that the flavor is "ns" when generating consensuses, but should accept consensuses where the flavor is omitted. + ### Microdescriptor consensus The microdescriptor consensus is a consensus flavor that contains @@ -752,6 +765,7 @@ Additionally, a microdescriptor consensus SHOULD use the sha256 digest algorithm for its signatures. + ## Exchanging detached signatures Once an authority has computed and signed a consensus network status, it @@ -829,6 +843,7 @@ and only copied it from the proposals. Review carefully. -KL] consensus document.] + ## Publishing the signed consensus The voting period ends at the valid-after time. If the consensus has @@ -867,4 +882,3 @@ locations ``` The standard URLs for bandwidth list files first-appeared in Tor 0.3.5. - diff --git a/spec/dir-spec/standards-compliance.md b/spec/dir-spec/standards-compliance.md index f20125e..2d4e525 100644 --- a/spec/dir-spec/standards-compliance.md +++ b/spec/dir-spec/standards-compliance.md @@ -1,10 +1,12 @@ + # Standards compliance All clients and servers MUST support HTTP 1.0. Clients and servers MAY support later versions of HTTP as well. + ## HTTP headers Servers SHOULD set Content-Encoding to the algorithm used to compress the @@ -49,6 +51,7 @@ single network statuses, the list of all server descriptors, a v1 directory, or a v1 running routers document. XXX mention times. + ## HTTP status codes Tor delivers the following status codes. Some were chosen without much @@ -76,4 +79,3 @@ thought; other code SHOULD NOT rely on specific status codes yet. -- user requested some items that we ordinarily generate or store, but we do not have any available. ``` - diff --git a/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md b/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md index f58796e..d354068 100644 --- a/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md +++ b/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md @@ -1,4 +1,5 @@ + ## Uploading server descriptors and extra-info documents ORs SHOULD generate a new server descriptor and a new extra-info @@ -40,4 +41,3 @@ http:///tor/ Server descriptors may not exceed 20,000 bytes in length; extra-info documents may not exceed 50,000 bytes in length. If they do, the authorities SHOULD reject them. - diff --git a/spec/dir-spec/vote-consensus-status-document-formats.md b/spec/dir-spec/vote-consensus-status-document-formats.md index 4ed6db6..379cd8f 100644 --- a/spec/dir-spec/vote-consensus-status-document-formats.md +++ b/spec/dir-spec/vote-consensus-status-document-formats.md @@ -1,4 +1,5 @@ + ### Vote and consensus status document formats Votes and consensuses are more strictly formatted than other documents @@ -370,7 +371,7 @@ shared random protocol. Version ::= An integer greater or equal to 0. AlgName ::= 1*(ALPHA / DIGIT / "_" / "-") -Identity ::= 40 * HEXDIG +Identity ::= 40* HEXDIG Commit ::= Base64-encoded-data Reveal ::= Base64-encoded-data @@ -703,4 +704,3 @@ types. Note that only one signature from each authority should be (Tor clients before 0.2.3.x did not understand the 'algorithm' field.) - diff --git a/spec/dos-spec.md b/spec/dos-spec.md index 04470b2..d3cba22 100644 --- a/spec/dos-spec.md +++ b/spec/dos-spec.md @@ -40,13 +40,14 @@ or derived from a fraction of the total amount of system RAM. As of Tor 0.4.7.x, the MaxMemInQueues mechanism tracks the following kinds of allocation: - * Cells queued on circuits. - * Per-connection read or write buffers. - * On-the-fly compression or decompression state. - * Half-open stream records. - * Cached onion service descriptors (hsdir only). - * Cached DNS resolves (relay only). - * GEOIP-based usage activity statistics. + +* Cells queued on circuits. +* Per-connection read or write buffers. +* On-the-fly compression or decompression state. +* Half-open stream records. +* Cached onion service descriptors (hsdir only). +* Cached DNS resolves (relay only). +* GEOIP-based usage activity statistics. Note that directory caches aren't counted, since those are stored on disk and accessed via mmap. @@ -86,12 +87,10 @@ rule, according to the age of their oldest queued data. Upon freeing a circuit, a "DESTROY cell" must be sent in both directions. -### Reporting low memory. +### Reporting low memory We define a "low threshold" equal to 3/4 of MaxMemInQueues. Every time our memory usage is above the low threshold, we record ourselves as being "under memory pressure". (This is not currently reported.) - - diff --git a/spec/ext-orport-spec.md b/spec/ext-orport-spec.md index d8688c5..75e2ec2 100644 --- a/spec/ext-orport-spec.md +++ b/spec/ext-orport-spec.md @@ -18,6 +18,7 @@ Table of Contents ``` + # Overview This document describes the "Extended ORPort" protocol, a wrapper @@ -33,7 +34,8 @@ This protocol was originally proposed in proposal 196, and extended with authentication in proposal 217. -# Establishing a connection and authenticating. + +# Establishing a connection and authenticating When a client (that is to say, a server-side pluggable transport) connects to an Extended ORPort, the server sends: @@ -68,6 +70,7 @@ If the client sent an AuthType of value 0, or an AuthType that the server does not support, the server MUST close the connection. + ## Authentication type: SAFE_COOKIE We define one authentication type: SAFE_COOKIE. Its AuthType @@ -88,6 +91,7 @@ defined as: where `` is a filesystem path. + ### Cookie-file format The format of the cookie-file is: @@ -108,6 +112,7 @@ present in the cookie file, before proceeding with the authentication protocol. + ### SAFE_COOKIE Protocol specification A client that performs the SAFE_COOKIE handshake begins by sending: @@ -115,6 +120,7 @@ A client that performs the SAFE_COOKIE handshake begins by sending: ClientNonce [32 octets] Where, + + ClientNonce is 32 octets of random data. Then, the server replies with: @@ -161,12 +167,14 @@ Status [1 octet] ``` + # The extended ORPort protocol Once a connection is established and authenticated, the parties communicate with the protocol described here. + ## Protocol The extended server port protocol is as follows: @@ -204,9 +212,11 @@ If the server receives a recognized command that does not parse, it MUST close the connection to the client. + ## Command descriptions + ### USERADDR ```text @@ -223,6 +233,7 @@ transports MUST NOT send them.) The string MUST not be NUL-terminated. + ### TRANSPORT An ASCII string holding the name of the pluggable transport used by @@ -236,6 +247,7 @@ Pluggable transport names are C-identifiers and Tor MUST check them for correctness. + # Security Considerations Extended ORPort or TransportControlPort do _not_ provide link @@ -251,4 +263,3 @@ Extended ORPort to a non-localhost address. Pluggable transport proxies SHOULD issue a warning if they are instructed to connect to a non-localhost Extended ORPort. - diff --git a/spec/gettor-spec.md b/spec/gettor-spec.md index 274d658..6895a1f 100644 --- a/spec/gettor-spec.md +++ b/spec/gettor-spec.md @@ -16,11 +16,13 @@ Table of Contents ``` + # Preface This document describes GetTor and how to properly implementation GetTor. + # Overview GetTor was created to resolve direct and indirect censorship of Tor's @@ -34,6 +36,7 @@ to fetch Tor. We discovered that it is feasible to use alternate transport methods such as SMTP between a non-trusted third party or with IRC and XDCC. + # Implementation Any compliant GetTor implementation will implement at least a single transport @@ -47,11 +50,13 @@ GetTor instance. Security and privacy considerations should be described on a per transport basis. + ## Reference implementation We have implemented[0] a compliant GetTor that supports SMTP as a transport. + # SMTP transport The SMTP transport for GetTor should allow users to send any RFC822 compliant @@ -62,6 +67,7 @@ it should offer the software as a list of packages and their subsequent descriptions. + ## SMTP transport security considerations Any GetTor instance that offers SMTP as a transport should optionally @@ -70,6 +76,7 @@ Optionally GetTor should take an OpenPGP key from the user and encrypt the response with a blinded message. + ## SMTP transport privacy considerations Any GetTor instance that offers SMTP as a transport must at least store the @@ -83,17 +90,18 @@ how GetTor[1] is in use. This must not include any personally identifying information about any of the requester beyond language selection. + # Other transports At this time no other transports have been specified. IRC XDCC is a likely useful system as is XMPP/Jabber with the newest OTR file sharing transport. + # Implementation suggestions It is suggested that any compliant GetTor instance should be written in a so called "safe" language such as Python. -[0] https://gitweb.torproject.org/gettor.git -[1] https://metrics.torproject.org/packages.html - +[0] +[1] diff --git a/spec/glossary.md b/spec/glossary.md index 476e2cd..8ea5363 100644 --- a/spec/glossary.md +++ b/spec/glossary.md @@ -34,6 +34,7 @@ Table of Contents ``` + # Preliminaries The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL @@ -42,20 +43,24 @@ NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and RFC 2119. + ## Commonly used Tor configuration terms ORPort - Onion Router Port DirPort - Directory Port + ## Tor network components + ## Relays, aka OR (onion router) [Style guide: prefer the term "Relay"] + ### Specific roles Exit relay: The final hop in an exit circuit before traffic leaves @@ -85,12 +90,14 @@ Each party builds a three-hop circuit, meeting at the rendezvous point. + ## Client, aka OP (onion proxy) [Style: the "OP" and "onion proxy" terms are deprecated.] -## Authorities: + +## Authorities Directory Authority: Nine total in the Tor network, operated by trusted individuals. Directory authorities define and serve the @@ -109,7 +116,8 @@ the client can ask any directory cache that's listed in the directory information it has.) -## Hidden Service: + +## Hidden Service A hidden service is a server that will only accept incoming connections via the hidden service protocol. Connection @@ -118,7 +126,8 @@ service, allowing the hidden service to receive incoming connections, serve content, etc, while preserving its location anonymity. -## Circuit: + +## Circuit An established path through the network, where cryptographic keys are negotiated using the ntor protocol or TAP (Tor Authentication @@ -136,7 +145,8 @@ network. For example, a client could connect to a hidden service via an internal circuit. -## Edge connection: + +## Edge connection ```text 2.7. Consensus: The state of the Tor network, published every hour, @@ -152,9 +162,11 @@ an internal circuit. ``` + ## Tor network protocols + ## Link handshake The link handshake establishes the TLS connection over which two @@ -163,6 +175,7 @@ authenticates the participants to each other, possibly using Tor cells. + ## Circuit handshake Circuit handshakes establish the hop-by-hop onion encryption @@ -193,12 +206,15 @@ contains the first part of the TAP or ntor key establishment handshake. + ## Hidden Service Protocol + ## Directory Protocol + ## General network definitions Leaky Pipe Topology: The ability for the origin of a circuit to address @@ -213,4 +229,3 @@ connection, a DNS request, or a Tor directory request. Channel: A pairwise connection between two Tor relays, or between a client and a relay. Circuits are multiplexed over Channels. All channels are currently implemented as TLS connections. - diff --git a/spec/guard-spec-intro.md b/spec/guard-spec-intro.md index bfc9e93..e94a4f2 100644 --- a/spec/guard-spec-intro.md +++ b/spec/guard-spec-intro.md @@ -37,4 +37,3 @@ Table of Contents A.4. Controller changes A.5. Persistent state format ``` - diff --git a/spec/guard-spec/algorithm.md b/spec/guard-spec/algorithm.md index 0f04f31..e1d4374 100644 --- a/spec/guard-spec/algorithm.md +++ b/spec/guard-spec/algorithm.md @@ -1,7 +1,9 @@ -# The algorithm. + +# The algorithm + ## The guards listed in the current consensus. [Section:GUARDS] By {set:GUARDS} we mean the set of all guards in the current @@ -14,6 +16,7 @@ We require all guards to have the flags that we potentially need from any guard, so that all guards are usable for all circuits. + ## The Sampled Guard Set. [Section:SAMPLED] We maintain a set, {set:SAMPLED_GUARDS}, that persists across @@ -126,6 +129,7 @@ adversary would need the clients to exhaust all their initial adversary node. + ## The Usable Sample [Section:FILTERED] We maintain another set, {set:FILTERED_GUARDS}, that does not @@ -180,6 +184,7 @@ before the sampling, then our sample would reflect the set of filtering restrictions that we had in the past. + ## The confirmed-guard list. [Section:CONFIRMED] [formerly USED_GUARDS] @@ -241,6 +246,7 @@ committing to a guard before we actually use it for sensitive traffic. + ## The Primary guards [Section:PRIMARY] We keep a run-time non-persistent ordered list of @@ -264,7 +270,7 @@ non-confirmed primary guards. Note that {PRIMARY_GUARDS} do not have to be in {USABLE_FILTERED_GUARDS}: they might be unreachable. -** Rationale ** +**Rationale** These guards are treated differently from other guards. If one of them is usable, then we use it right away. For other guards @@ -273,6 +279,7 @@ first double-check whether perhaps one of the primary guards is usable after all. + ## Retrying guards. [Section:RETRYING] (We run this process as frequently as needed. It can be done once @@ -288,13 +295,14 @@ we decide whether to update its {is_reachable} status to `` based on its {last_tried_connect} time, its {failing_since} time, and the {GUARDS_RETRY_SCHED} schedule. -** Rationale ** +**Rationale** An observation that a guard has been 'unreachable' only lasts for a given amount of time, since we can't infer that it's unreachable now from the fact that it was unreachable a few minutes ago. + ## Selecting guards for circuits. [Section:SELECTING] Every origin circuit is now in one of these states: @@ -379,7 +387,7 @@ restrict the guards that we use for a single circuit. When this happens, we remember the restrictions that applied when choosing the guard for that circuit, since we will need them later (see [UPDATE_WAITING].). -** Rationale ** +**Rationale** We're getting to the core of the algorithm here. Our main goals are to make sure that @@ -401,6 +409,7 @@ sure that it's the best guard we're getting. (see below). [XXX timeout.] + ## When a circuit fails. [Section:ON_FAIL] When a circuit fails in a way that makes us conclude that a guard @@ -422,11 +431,12 @@ is not reachable, we take the following steps: circuits in response to some of these steps; and also see [ON_CONSENSUS].] -** Rationale ** +**Rationale** See [SELECTING] above for rationale. + ## When a circuit succeeds [Section:ON_SUCCESS] When a circuit succeeds in a way that makes us conclude that a @@ -460,11 +470,12 @@ guard _was_ reachable, we take these steps: circuits in response to some of these steps; and see [ON_CONSENSUS].] -** Rationale ** +**Rationale** See [SELECTING] above for rationale. + ## Updating the list of waiting circuits [Section:UPDATE_WAITING] We run this procedure whenever it's possible that a @@ -504,6 +515,7 @@ them after all if the `` circuit goes down before {NONPRIMARY_GUARD_IDLE_TIMEOUT} seconds. + ### Without a list of waiting circuits [Section:NO_CIRCLIST] As an alternative to the section [SECTION:UPDATE_WAITING] above, @@ -546,6 +558,7 @@ circuit is not discarded; instead, it is kept (but not used) until the guard becomes usable or unusable. + ## Whenever we get a new consensus. [Section:ON_CONSENSUS] We update {GUARDS}. @@ -591,4 +604,3 @@ directory fetches, but not for longer circuits. (Also, when we are missing descriptors for our first {NUM_USABLE_PRIMARY_GUARDS} primary guards, we don't build circuits at all until we have fetched them.) - diff --git a/spec/guard-spec/appendices.md b/spec/guard-spec/appendices.md index ea743f7..612ff4c 100644 --- a/spec/guard-spec/appendices.md +++ b/spec/guard-spec/appendices.md @@ -1,13 +1,16 @@ + # Appendices + ## Acknowledgements This research was supported in part by NSF grants CNS-1111539, CNS-1314637, CNS-1526306, CNS-1619454, and CNS-1640548. + ## Parameters with suggested values. [Section:PARAM_VALS] (All suggested values chosen arbitrarily) @@ -76,6 +79,7 @@ CNS-1314637, CNS-1526306, CNS-1619454, and CNS-1640548. ``` + ## Random values [Section:RANDOM] Frequently, we want to randomize the expiration time of something @@ -87,6 +91,7 @@ By RAND(now, INTERVAL) we mean a time between now and INTERVAL in the past, chosen uniformly at random. + ## Why not a sliding scale of primaryness? [Section:CVP] At one meeting, I floated the idea of having "primaryness" be a @@ -141,6 +146,7 @@ need more analysis! Here's why: ``` + ## Controller changes We will add to control-spec.txt a new possible circuit state, GUARD_WAIT, @@ -150,6 +156,7 @@ but we will not use it because a circuit with a better guard might become built too. + ## Persistent state format The persistent state format doesn't need to be part of this @@ -208,4 +215,3 @@ Recognized fields (values of K) are: All dates here are given as a (spaceless) ISO8601 combined date and time in UTC (e.g., 2016-11-29T19:39:31). - diff --git a/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md b/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md index 13cc73e..bbebb04 100644 --- a/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md +++ b/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md @@ -1,4 +1,5 @@ + # Circuit Creation, Entry Guard Selection (1000 foot view) A circuit in Tor is a path through the network connecting a client to @@ -69,4 +70,3 @@ guard if there's a good chance that it will be able to make one. If the circuit fails in a way that makes us conclude that a guard is not reachable, the guard is marked as unreachable, the circuit is closed, and waiting circuits are updated. - diff --git a/spec/guard-spec/introduction-motivation.md b/spec/guard-spec/introduction-motivation.md index f552434..628cf1a 100644 --- a/spec/guard-spec/introduction-motivation.md +++ b/spec/guard-spec/introduction-motivation.md @@ -1,4 +1,5 @@ + # Introduction and motivation Tor uses entry guards to prevent an attacker who controls some @@ -42,4 +43,3 @@ which tries to meet the following goals: - Should maintain the load-balancing offered by the path selection algorithm ``` - diff --git a/spec/guard-spec/state-instances.md b/spec/guard-spec/state-instances.md index 1f292b2..5449648 100644 --- a/spec/guard-spec/state-instances.md +++ b/spec/guard-spec/state-instances.md @@ -1,4 +1,5 @@ + # State instances In the algorithm below, we describe a set of persistent and @@ -46,4 +47,3 @@ A. UseBridges If neither of the above variant-state instances is used, we use a default instance. ``` - diff --git a/spec/guard-spec/still-non-addressed-issues-sectiontodo.md b/spec/guard-spec/still-non-addressed-issues-sectiontodo.md index e565194..4edb761 100644 --- a/spec/guard-spec/still-non-addressed-issues-sectiontodo.md +++ b/spec/guard-spec/still-non-addressed-issues-sectiontodo.md @@ -1,4 +1,5 @@ + # Still non-addressed issues [Section:TODO] Simulate to answer: Will this work in a dystopic world? @@ -26,4 +27,3 @@ Fix all items marked XX or TODO. Suggestion: Treat it as a preference when adding to {CONFIRMED_GUARDS}, but not otherwise. ``` - diff --git a/spec/padding-spec-intro.md b/spec/padding-spec-intro.md index ce9e985..b9172db 100644 --- a/spec/padding-spec-intro.md +++ b/spec/padding-spec-intro.md @@ -37,4 +37,3 @@ Table of Contents 3.4. Circuit padding consensus parameters A. Acknowledgments ``` - diff --git a/spec/padding-spec/acknowledgments.md b/spec/padding-spec/acknowledgments.md index 7213127..5a594da 100644 --- a/spec/padding-spec/acknowledgments.md +++ b/spec/padding-spec/acknowledgments.md @@ -1,4 +1,5 @@ + # Acknowledgments This research was supported in part by NSF grants CNS-1111539, @@ -25,4 +26,3 @@ CNS-1314637, CNS-1526306, CNS-1619454, and CNS-1640548. 18. https://www.usenix.org/node/190967 https://blog.torproject.org/technical-summary-usenix-fingerprinting-paper ``` - diff --git a/spec/padding-spec/circuit-level-padding.md b/spec/padding-spec/circuit-level-padding.md index 5f821ca..575f248 100644 --- a/spec/padding-spec/circuit-level-padding.md +++ b/spec/padding-spec/circuit-level-padding.md @@ -1,4 +1,5 @@ + # Circuit-level padding The circuit padding system in Tor is an extension of the WTF-PAD @@ -21,6 +22,7 @@ operation. For full details on using the circuit padding system to develop future padding defenses, see the research developer documentation[17]. + ## Circuit Padding Negotiation Circuit padding machines are advertised as "Padding" subprotocol versions @@ -93,6 +95,7 @@ If the machine_ctr does not match the current machine instance count on the circuit, the command is ignored. + ## Circuit Padding Machine Message Management Clients MAY send padding cells towards the relay before receiving the @@ -108,6 +111,7 @@ from unexpected relay sources are protocol violations, and clients MAY immediately tear down such circuits to avoid side channel risk. + ## Obfuscating client-side onion service circuit setup The circuit padding currently deployed in Tor attempts to hide client-side @@ -124,6 +128,7 @@ of general circuits. Note that inter-arrival timing is not obfuscated by this defense. + ### Common general circuit construction sequences Most general Tor circuits used to surf the web or download directory @@ -150,6 +155,7 @@ depend on the type of guard used and are not an effective fingerprint for a network/guard-level adversary. + ### Client-side onion service introduction circuit obfuscation Two circuit padding machines work to hide client-side introduction circuits: @@ -192,6 +198,7 @@ the same duration as normal web circuits before they expire (usually 10 minutes). + ### Client-side rendezvous circuit hiding Following a similar argument as for intro circuits, we are aiming for padded @@ -232,6 +239,7 @@ general circuits (their purpose is to surf the web), we can expect that they will look alike. + ### Circuit setup machine overhead For the intro circuit case, we see that the origin-side machine just sends a @@ -243,6 +251,7 @@ For the rend circuit case, this machine is quite light. Both sides send 2 padding cells, for a total of 4 padding cells. + ## Circuit padding consensus parameters The circuit padding system has a handful of consensus parameters that can @@ -278,4 +287,3 @@ at relays and clients. before padding stops being sent on that circuit. - Default: CIRCWINDOW_START_MAX (1000) ``` - diff --git a/spec/padding-spec/connection-level-padding.md b/spec/padding-spec/connection-level-padding.md index 3bb7f12..b0450d7 100644 --- a/spec/padding-spec/connection-level-padding.md +++ b/spec/padding-spec/connection-level-padding.md @@ -1,7 +1,9 @@ + # Connection-level padding + ## Background Tor clients and relays make use of CELL_PADDING to reduce the resolution of @@ -87,6 +89,7 @@ measurement is unidirectional, and so traffic must be sent by both parties in order to prevent record splitting. + ## Implementation Tor clients currently maintain one TLS connection to their Guard node to @@ -129,6 +132,7 @@ queue that they practically can, and if this queue is already nonempty, padding should not be scheduled until after the queue does become empty.) + ## Padding Cell Timeout Distribution Statistics To limit the amount of padding sent, instead of sampling each endpoint @@ -160,6 +164,7 @@ values for each random variable: ``` + ## Maximum overhead bounds With the default parameters and the above distribution, we expect a @@ -178,6 +183,7 @@ roughly the current amount of Tor directory traffic[11]. Of course, our idle, so we expect the actual overhead to be much lower than this. + ## Reducing or Disabling Padding via Negotiation To allow mobile clients to either disable or reduce their padding overhead, @@ -222,6 +228,7 @@ Clients and bridges MUST reject padding negotiation messages from relays, and close the channel if they receive one. + ## Consensus Parameters Governing Behavior Connection-level padding is controlled by the following consensus parameters: @@ -280,4 +287,3 @@ Connection-level padding is controlled by the following consensus parameters: open. - Default: 3600 ``` - diff --git a/spec/padding-spec/overview.md b/spec/padding-spec/overview.md index 76c8865..0abe739 100644 --- a/spec/padding-spec/overview.md +++ b/spec/padding-spec/overview.md @@ -1,4 +1,5 @@ + # Overview Tor supports two classes of cover traffic: connection-level padding, and @@ -25,4 +26,3 @@ connection-level padding. The connection-level padding system regards circuit-level padding as normal data traffic, and hence the connection-level padding system will not add any additional overhead while the circuit-level padding system is actively padding. - diff --git a/spec/param-spec.md b/spec/param-spec.md index 3277609..0d465e1 100644 --- a/spec/param-spec.md +++ b/spec/param-spec.md @@ -19,6 +19,7 @@ Table of Contents X. Obsolete parameters + # Network protocol parameters "circwindow" -- the default package window that circuits should be @@ -71,6 +72,7 @@ Min: 0. Max: 1. Default: 0 First appeared: 0.4.5.1-alpha. + # Performance-tuning parameters "CircuitPriorityHalflifeMsec" -- the halflife parameter used when @@ -120,6 +122,7 @@ Min: 2. Max: 100. Default: 2. First appeared: 0.4.8.2 + # Voting-related parameters "bwweightscale" -- Value that bandwidth-weights are divided by. If not @@ -157,6 +160,7 @@ Min: 1. Max: INT32_MAX. Default: 2/3 of the total number of dirauth. + # Circuit-build-timeout parameters "cbtdisabled", "cbtnummodes", "cbtrecentcount", "cbtmaxtimeouts", @@ -167,6 +171,7 @@ behavior" in path-spec.txt for a series of circuit build time related consensus parameters. + # Directory-related parameters "max-consensus-age-to-cache-for-diff" -- Determines how much @@ -178,6 +183,7 @@ old a consensus can be (in hours) before a client should no longer try to find a diff for it. (min 0, max 8192, default 72) + # Pathbias parameters "pb_mincircs", "pb_noticepct", "pb_warnpct", "pb_extremepct", @@ -186,6 +192,7 @@ try to find a diff for it. (min 0, max 8192, default 72) "pb_extremeusepct", "pb_scaleuse" -- DOCDOC + # Relay behavior "refuseunknownexits" -- if set to one, exit relays look at the previous @@ -269,6 +276,7 @@ Min: 0. Max: 255. Default: 2 First appeared: 0.4.7.5-alpha. + # V3 onion service parameters "hs_intro_min_introduce2", "hs_intro_max_introduce2" -- @@ -327,6 +335,7 @@ Min: 0. Max: INT32_MAX. Default: 25 First appeared: 0.4.2.1-alpha. + # Denial-of-service parameters Denial of Service mitigation parameters. Introduced in 0.3.3.2-alpha: @@ -373,6 +382,7 @@ concurrent connection from a client IP address. rendezvous points for single hop clients. + # Padding-related parameters "circpad_max_circ_queued_cells" -- The circuitpadding module will @@ -406,6 +416,7 @@ First appeared: 0.4.0.3-alpha. "nf_pad_single_onion" -- DOCDOC + # Guard-related parameters (See guard-spec.txt for more information on the vocabulary used here.) @@ -502,6 +513,7 @@ Min: 1. Max: 3650. Default: 20. First appeared: 0.3.0 + # Obsolete parameters "NumDirectoryGuards", "NumEntryGuards" -- Number of guard nodes @@ -532,4 +544,3 @@ they do not. Min: 0, Max: 1. Default: 1. First-appeared: 0.2.4.18-rc Removed in: 0.2.6 - diff --git a/spec/path-spec-intro.md b/spec/path-spec-intro.md index 218aa9b..db16f60 100644 --- a/spec/path-spec-intro.md +++ b/spec/path-spec-intro.md @@ -63,4 +63,3 @@ Tables of Contents X.2. A thing we could do to deal with reachability. X.3. Some stuff that worries me about entry guards. 2006 Jun, Nickm. ``` - diff --git a/spec/path-spec/attaching-streams-to-circuits.md b/spec/path-spec/attaching-streams-to-circuits.md index a603b98..e03414e 100644 --- a/spec/path-spec/attaching-streams-to-circuits.md +++ b/spec/path-spec/attaching-streams-to-circuits.md @@ -1,4 +1,5 @@ + # Attaching streams to circuits When a circuit that might support a request is built, Tor tries to attach @@ -16,4 +17,3 @@ XXX Timeouts and when Tor auto-retries. * What stream-end-reasons are appropriate for retrying. If no reply to BEGIN/RESOLVE, then the stream will timeout and fail. - diff --git a/spec/path-spec/building-circuits.md b/spec/path-spec/building-circuits.md index 2382a88..af0b5cc 100644 --- a/spec/path-spec/building-circuits.md +++ b/spec/path-spec/building-circuits.md @@ -1,3 +1,3 @@ -# Building circuits +# Building circuits diff --git a/spec/path-spec/cannibalizing-circuits.md b/spec/path-spec/cannibalizing-circuits.md index 8d07489..2d15175 100644 --- a/spec/path-spec/cannibalizing-circuits.md +++ b/spec/path-spec/cannibalizing-circuits.md @@ -1,4 +1,5 @@ + ## Cannibalizing circuits If we need a circuit and have a clean one already established, in @@ -12,4 +13,3 @@ from scratch on demand. We can also cannibalize clean circuits when the client asks to exit at a given node -- either via the ".exit" notation or because the destination is running at the same location as an exit node. - diff --git a/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md b/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md index 1191efc..c7ff50c 100644 --- a/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md +++ b/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md @@ -1,4 +1,5 @@ + # Detecting route manipulation by Guard nodes (Path Bias) The Path Bias defense is designed to defend against a type of route @@ -33,6 +34,7 @@ restrict the defense to being informational only at this stage (see section 7.5). + ## Measuring path construction success rates Clients maintain two counts for each of their guards: a count of the @@ -49,6 +51,7 @@ If a circuit closes prematurely after construction but before being requested to close by the client, this is counted as a failure. + ## Measuring path usage success rates Clients maintain two usage counts for each of their guards: a count @@ -84,6 +87,7 @@ Prematurely closed circuits are not probed, and are counted as usage failures. + ## Scaling success counts To provide a moving average of recent Guard activity while @@ -99,6 +103,7 @@ currently open circuits are subtracted from the usage counts before scaling, and added back after scaling. + ## Parametrization The following consensus parameters tune various aspects of the @@ -183,6 +188,7 @@ defense. ``` + ## Known barriers to enforcement Due to intermittent CPU overload at relays, the normal rate of @@ -190,4 +196,3 @@ successful circuit completion is highly variable. The Guard-dropping version of the defense is unlikely to be deployed until the ntor circuit handshake is enabled, or the nature of CPU overload induced failure is better understood. - diff --git a/spec/path-spec/general-operation.md b/spec/path-spec/general-operation.md index 7a1c281..e1632af 100644 --- a/spec/path-spec/general-operation.md +++ b/spec/path-spec/general-operation.md @@ -1,4 +1,5 @@ + # General operation Tor begins building circuits as soon as it has enough directory @@ -44,6 +45,7 @@ ATTACHSTREAM commands). Paths constructed through these means may violate some constraints given below. + ## Terminology A "path" is an ordered sequence of nodes, not yet built as a circuit. @@ -74,6 +76,7 @@ is unknown (usually its target IP), but we believe the path probably supports the request according to the rules given below. + ## A relay's bandwidth Old versions of Tor did not report bandwidths in network status @@ -91,4 +94,3 @@ value. For more recent versions of Tor, we take the bandwidth value declared in the consensus, and fall back to the clipped advertised bandwidth only if the consensus does not have bandwidths listed. - diff --git a/spec/path-spec/guard-nodes.md b/spec/path-spec/guard-nodes.md index 6b7bc8b..77f7fdf 100644 --- a/spec/path-spec/guard-nodes.md +++ b/spec/path-spec/guard-nodes.md @@ -1,4 +1,5 @@ + # Guard nodes We use Guard nodes (also called "helper nodes" in the research @@ -7,6 +8,7 @@ our Guard selection algorithm -- which has grown rather complex -- see guard-spec.txt. + ## How consensus bandwidth weights factor into entry guard selection When weighting a list of routers for choosing an entry guard, the following @@ -29,7 +31,7 @@ If a router has been marked as both an entry guard and an exit, then we prefer to use it more, with our preference for doing so (roughly) linearly increasing w.r.t. the router's non-guard bandwidth and bandwidth weight (calculated without taking the guard flag into account). From proposal -#236: +# 236: | | Let Wpf denote the weight from the 'bandwidth-weights' line a @@ -40,4 +42,3 @@ increasing w.r.t. the router's non-guard bandwidth and bandwidth weight | choose N proportionally to F*Wpf*B + (1-F)*Wpn*B. where F is the weight as calculated using the above parameters. - diff --git a/spec/path-spec/handling-failure.md b/spec/path-spec/handling-failure.md index d7bb80f..a05fc58 100644 --- a/spec/path-spec/handling-failure.md +++ b/spec/path-spec/handling-failure.md @@ -1,4 +1,5 @@ + ## Handling failure If an attempt to extend a circuit fails (either because the first create @@ -14,4 +15,3 @@ a fresh descriptor for it. Excessive amounts of either type of failure can indicate an attack on anonymity. See section 7 for how excessive failure is handled. - diff --git a/spec/path-spec/hidden-service-related-circuits.md b/spec/path-spec/hidden-service-related-circuits.md index e0e78e3..fc6e30d 100644 --- a/spec/path-spec/hidden-service-related-circuits.md +++ b/spec/path-spec/hidden-service-related-circuits.md @@ -1,5 +1,5 @@ + # Hidden-service related circuits XXX Tracking expected hidden service use (client-side and hidserv-side) - diff --git a/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md b/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md index 75f8641..06f6914 100644 --- a/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md +++ b/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md @@ -1,10 +1,12 @@ + ## Learning when to give up ("timeout") on circuit construction Since version 0.2.2.8-alpha, Tor clients attempt to learn when to give up on circuits based on network conditions. + ### Distribution choice Based on studies of build times, we found that the distribution of @@ -16,6 +18,7 @@ in the accuracy of the tail, clients approximate the tail of the multi-modal distribution with a single Pareto curve. + ### How much data to record From our observations, the minimum number of circuit build times for a @@ -44,6 +47,7 @@ choose a different persistence mechanism than this histogram, but be aware that build time binning is still needed for parameter estimation. + ### Parameter estimation Once 'cbtmincircs' build times are recorded, Tor clients update the @@ -53,7 +57,7 @@ too many circuits timing out). Tor clients calculate the parameters for a Pareto distribution fitting the data using the maximum likelihood estimator. For derivation, see: -https://en.wikipedia.org/wiki/Pareto_distribution#Estimation_of_parameters + Because build times are not a true Pareto distribution, we alter how Xm is computed. In a max likelihood estimator, the mode of the distribution is @@ -117,6 +121,7 @@ but at least 60 seconds: ``` + ### Calculating timeouts thresholds for circuits of different lengths The timeout_ms and close_ms estimates above are good only for 3-hop @@ -139,6 +144,7 @@ the client should add X to Actions(N) for every round-trip communication required with the Xth hop. + ### How to record timeouts Pareto estimators begin to lose their accuracy if the tail is omitted. @@ -159,6 +165,7 @@ should be ignored, as this typically means one of the relays in the path is offline. + ### Detecting Changing Network Conditions Tor clients attempt to detect both network connectivity loss and drastic @@ -179,6 +186,7 @@ recent 'cbrrecentcount') are not stored as persistent state. On reload, we start with a new, empty state. + ### Consensus parameters governing behavior Clients that implement circuit build timeout learning should obey the @@ -290,4 +298,3 @@ the listed default values should be used instead. Effect: This is the maximum number of circuits that can be open at at the same time during the circuit build time learning phase. ``` - diff --git a/spec/path-spec/old-notes.md b/spec/path-spec/old-notes.md index 10623c1..34734fb 100644 --- a/spec/path-spec/old-notes.md +++ b/spec/path-spec/old-notes.md @@ -1,7 +1,9 @@ + # Old notes + ## Do we actually do this? ```text @@ -28,7 +30,8 @@ How to deal with network down. remove it. -NM] -## A thing we could do to deal with reachability. + +## A thing we could do to deal with reachability And as a bonus, it leads to an answer to Nick's attack ("If I pick my helper nodes all on 18.0.0.0:*, then I move, you'll know where I @@ -39,7 +42,8 @@ likely (though not certain) that some of the originals will become useful. Is that smart or just complex? -## Some stuff that worries me about entry guards. 2006 Jun, Nickm. + +## Some stuff that worries me about entry guards. 2006 Jun, Nickm It is unlikely for two users to have the same set of entry guards. Observing a user is sufficient to learn its entry guards. So, as we move @@ -57,4 +61,3 @@ our location (IP? subnet?) changes, we have two bad options. We could [Do we do any of this now? If not, this should move into 099-misc or 098-todo. -NM] - diff --git a/spec/path-spec/path-selection-constraints.md b/spec/path-spec/path-selection-constraints.md index f590b36..694a052 100644 --- a/spec/path-spec/path-selection-constraints.md +++ b/spec/path-spec/path-selection-constraints.md @@ -1,4 +1,5 @@ + ## Path selection and constraints We choose the path for each new circuit before we build it. We choose the @@ -86,6 +87,7 @@ mind. Each kind of request puts certain constraints on paths: ``` + ### Choosing an exit If we know what IP address we want to connect to or resolve, we can @@ -104,6 +106,7 @@ flagged as "BadExit" by more than half of the authorities who advertise themselves as listing bad exits. + ### User configuration Users can alter the default behavior for path selection with configuration @@ -127,4 +130,3 @@ options. in the path. They also allow the guard node to be chosen as the RP, IP, and HSDIR, and as the hop before those positions. ``` - diff --git a/spec/path-spec/server-descriptor-purposes.md b/spec/path-spec/server-descriptor-purposes.md index 6e3a71e..3a7dca0 100644 --- a/spec/path-spec/server-descriptor-purposes.md +++ b/spec/path-spec/server-descriptor-purposes.md @@ -1,4 +1,5 @@ + # Server descriptor purposes There are currently three "purposes" supported for server descriptors: @@ -16,4 +17,3 @@ Bridge-purpose descriptors are for routers that are used as bridges. See doc/design-paper/blocking.pdf for more design explanation, or proposal 125 for specific details. Currently bridge descriptors are used in place of normal entry guards, for Tor clients that have UseBridges enabled. - diff --git a/spec/path-spec/when-we-build.md b/spec/path-spec/when-we-build.md index 4320c58..ca9b704 100644 --- a/spec/path-spec/when-we-build.md +++ b/spec/path-spec/when-we-build.md @@ -1,7 +1,9 @@ + ## When we build + ### We don't build circuits until we have enough directory info There's a class of possible attacks where our directory servers @@ -56,6 +58,7 @@ If the consensus lists zero exit-flagged relays, Tor instead uses the fraction of middle relays. + ### Clients build circuits preemptively When running as a client, Tor tries to maintain at least a certain @@ -90,6 +93,7 @@ The Tor client SHOULD NOT store its list of predicted requests to a persistent medium. + ### Clients build circuits on demand Additionally, when a client request exists that no circuit (built or @@ -109,6 +113,7 @@ In some cases we can reuse an already established circuit if it's clean; see Section 2.3 (cannibalizing circuits) for details. + ### Relays build circuits for testing reachability and bandwidth Tor relays test reachability of their ORPort once they have @@ -131,11 +136,13 @@ established a circuit, but they use an ordinary exit circuit for this purpose. + ### Hidden-service circuits See section 4 below. + ### Rate limiting of failed circuits If we fail to build a circuit N times in a X second period (see Section @@ -144,6 +151,7 @@ have elapsed. XXXX + ### When to tear down circuits Clients should tear down circuits (in general) only when those circuits @@ -158,4 +166,3 @@ stream-less circuits only under one of the following conditions: - The circuit is dirty (has had a stream attached), and it has been dirty for at least MaxCircuitDirtiness. ``` - diff --git a/spec/pt-spec-intro.md b/spec/pt-spec-intro.md index 016effa..1b4555e 100644 --- a/spec/pt-spec-intro.md +++ b/spec/pt-spec-intro.md @@ -35,4 +35,3 @@ Table of Contents Appendix A. Example Client Pluggable Transport Session Appendix B. Example Server Pluggable Transport Session ``` - diff --git a/spec/pt-spec/acknowledgments.md b/spec/pt-spec/acknowledgments.md index 8c12c83..92d373a 100644 --- a/spec/pt-spec/acknowledgments.md +++ b/spec/pt-spec/acknowledgments.md @@ -1,6 +1,6 @@ + # Acknowledgments This specification draws heavily from prior versions done by Jacob Appelbaum, Nick Mathewson, and George Kadianakis. - diff --git a/spec/pt-spec/anonymity-considerations.md b/spec/pt-spec/anonymity-considerations.md index 80b3826..ce3a54e 100644 --- a/spec/pt-spec/anonymity-considerations.md +++ b/spec/pt-spec/anonymity-considerations.md @@ -1,4 +1,5 @@ + # Anonymity Considerations When designing and implementing a Pluggable Transport, care @@ -20,4 +21,3 @@ Additionally, certain obfuscation mechanisms rely on information such as the server IP address/port being confidential, so clients also need to take care to preserve server side information confidential when applicable. - diff --git a/spec/pt-spec/architecture-overview.md b/spec/pt-spec/architecture-overview.md index 44718b2..64a3eca 100644 --- a/spec/pt-spec/architecture-overview.md +++ b/spec/pt-spec/architecture-overview.md @@ -1,4 +1,5 @@ + # Architecture Overview ```text @@ -34,4 +35,3 @@ Each invocation of a PT MUST be either a client OR a server. All PT client forward proxies MUST support either SOCKS 4 or SOCKS 5, and SHOULD prefer SOCKS 5 over SOCKS 4. - diff --git a/spec/pt-spec/example-client-pluggable-transport-session.md b/spec/pt-spec/example-client-pluggable-transport-session.md index 017bcaf..2f20f5d 100644 --- a/spec/pt-spec/example-client-pluggable-transport-session.md +++ b/spec/pt-spec/example-client-pluggable-transport-session.md @@ -1,4 +1,5 @@ + # Appendix A: Example Client Pluggable Transport Session Environment variables: @@ -16,4 +17,3 @@ PROXY DONE CMETHOD obfs3 socks5 127.0.0.1:32525 CMETHOD obfs4 socks5 127.0.0.1:37347 CMETHODS DONE - diff --git a/spec/pt-spec/example-server-pluggable-transport-session.md b/spec/pt-spec/example-server-pluggable-transport-session.md index 15a9ab1..3fc3962 100644 --- a/spec/pt-spec/example-server-pluggable-transport-session.md +++ b/spec/pt-spec/example-server-pluggable-transport-session.md @@ -1,4 +1,5 @@ + # Appendix B: Example Server Pluggable Transport Session Environment variables: @@ -15,4 +16,3 @@ VERSION 1 SMETHOD obfs3 198.51.100.1:1984 SMETHOD obfs4 198.51.100.1:43734 ARGS:cert=HszPy3vWfjsESCEOo9ZBkRv6zQ/1mGHzc8arF0y2SpwFr3WhsMu8rK0zyaoyERfbz3ddFw,iat-mode=0 SMETHODS DONE - diff --git a/spec/pt-spec/introduction.md b/spec/pt-spec/introduction.md index fae42bc..6db3ea5 100644 --- a/spec/pt-spec/introduction.md +++ b/spec/pt-spec/introduction.md @@ -1,4 +1,5 @@ + # Introduction This specification describes a way to decouple protocol-level @@ -17,10 +18,10 @@ in this document will be able to use all spec compliant Pluggable Transports. + ## Requirements Notation The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. - diff --git a/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md b/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md index cce6748..869fb59 100644 --- a/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md +++ b/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md @@ -1,4 +1,5 @@ + ## Pluggable Transport Client Per-Connection Arguments Certain PT transport protocols require that the client provides @@ -35,4 +36,3 @@ SOCKS protocol version. If the encoded argument list is less than 255 bytes in length, the "PLEN" field must be set to "1" and the "PASSWD" field must contain a single NUL character. - diff --git a/spec/pt-spec/pluggable-transport-configuration-environment.md b/spec/pt-spec/pluggable-transport-configuration-environment.md index 3a201f8..0fd9571 100644 --- a/spec/pt-spec/pluggable-transport-configuration-environment.md +++ b/spec/pt-spec/pluggable-transport-configuration-environment.md @@ -1,4 +1,5 @@ + ## Pluggable Transport Configuration Environment Variables All Pluggable Transport proxy instances are configured by their @@ -10,6 +11,7 @@ indicate any relations to Tor, except for the origins of this specification. + ### Common Environment Variables When launching either a client or server Pluggable Transport proxy, @@ -99,6 +101,7 @@ Example:: TOR_PT_OUTBOUND_BIND_ADDRESS_V6=[2001:db8::4] + ### Pluggable Transport Client Environment Variables Client-side Pluggable Transport forward proxies are configured @@ -139,6 +142,7 @@ Examples: ``` + ### Pluggable Transport Server Environment Variables Server-side Pluggable Transport reverse proxies are configured @@ -252,4 +256,3 @@ incoming traffic, "TOR_PT_AUTH_COOKIE_FILE" MUST be omitted. Example: TOR_PT_AUTH_COOKIE_FILE=/var/lib/tor/extended_orport_auth_cookie - diff --git a/spec/pt-spec/pluggable-transport-naming.md b/spec/pt-spec/pluggable-transport-naming.md index 50ebcb3..339f89c 100644 --- a/spec/pt-spec/pluggable-transport-naming.md +++ b/spec/pt-spec/pluggable-transport-naming.md @@ -1,4 +1,5 @@ + ## Pluggable Transport Naming Pluggable Transport names serve as unique identifiers, and every @@ -10,4 +11,3 @@ ASCII letters, numbers or underscores. No length limit is imposted. PT names MUST satisfy the regular expression "[a-zA-Z_][a-zA-Z0-9_]*". - diff --git a/spec/pt-spec/pluggable-transport-shutdown.md b/spec/pt-spec/pluggable-transport-shutdown.md index b2c0fcc..cd826d9 100644 --- a/spec/pt-spec/pluggable-transport-shutdown.md +++ b/spec/pt-spec/pluggable-transport-shutdown.md @@ -1,4 +1,5 @@ + ## Pluggable Transport Shutdown The recommended way for Pluggable Transport using applications and @@ -29,4 +30,3 @@ Pluggable Transports to handle graceful shutdown is as follows. terminated (eg: via detecting that its parent process ID has changed on U*IX systems), and gracefully terminate. ``` - diff --git a/spec/pt-spec/pluggable-transport-to-parent-process-communication.md b/spec/pt-spec/pluggable-transport-to-parent-process-communication.md index b7f8175..985bcb5 100644 --- a/spec/pt-spec/pluggable-transport-to-parent-process-communication.md +++ b/spec/pt-spec/pluggable-transport-to-parent-process-communication.md @@ -1,4 +1,5 @@ + ## Pluggable Transport To Parent Process Communication All Pluggable Transport Proxies communicate to the parent process @@ -19,6 +20,7 @@ The parent process MUST ignore lines received from PT proxies with unknown keywords. + ### Common Messages When a PT proxy first starts up, it must determine which version @@ -93,6 +95,7 @@ Example: ENV-ERROR No TOR_PT_AUTH_COOKIE_FILE when TOR_PT_EXTENDED_SERVER_PORT set + ### Pluggable Transport Client Messages After negotiating the Pluggable Transport Specification version, @@ -174,6 +177,7 @@ Notes: ``` + ### Pluggable Transport Server Messages PT server reverse proxies iterate over the requested transports @@ -240,6 +244,7 @@ Upon sending the "SMETHODS DONE" message, the PT proxy initialization is complete. + ### Pluggable Transport Log Message This message is for a client or server PT to be able to signal back to the @@ -270,6 +275,7 @@ Example: `LOG SEVERITY=debug MESSAGE="Connected to bridge A"` + ### Pluggable Transport Status Message This message is for a client or server PT to be able to signal back to the diff --git a/spec/pt-spec/references.md b/spec/pt-spec/references.md index 6956c00..e94d41d 100644 --- a/spec/pt-spec/references.md +++ b/spec/pt-spec/references.md @@ -1,4 +1,5 @@ + # References ```text @@ -19,4 +20,3 @@ [RFC1929] Leech, M., "Username/Password Authentication for SOCKS V5", RFC 1929, March 1996. ``` - diff --git a/spec/pt-spec/specification.md b/spec/pt-spec/specification.md index 0c10377..7a8822b 100644 --- a/spec/pt-spec/specification.md +++ b/spec/pt-spec/specification.md @@ -1,4 +1,5 @@ + # Specification Pluggable Transport proxies follow the following workflow @@ -49,4 +50,3 @@ throughout their lifespan. 7) Upon being signaled to terminate by the parent process (3.4), the PT Proxy gracefully shuts down. ``` - diff --git a/spec/rend-spec-v3-intro.md b/spec/rend-spec-v3-intro.md index 5881eaa..78efe6a 100644 --- a/spec/rend-spec-v3-intro.md +++ b/spec/rend-spec-v3-intro.md @@ -95,4 +95,3 @@ Change history: 2015-05-26: Fix two typos. ``` - diff --git a/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md b/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md index b98f1ae..2a1cb72 100644 --- a/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md +++ b/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md @@ -1,4 +1,5 @@ + ## Deriving blinded keys and subcredentials [SUBCRED] In each time period (see [TIME-PERIODS] for a definition of time @@ -56,6 +57,7 @@ Which Tor servers hosts a hidden service depends on: ``` + ### Dividing time into periods [TIME-PERIODS] To prevent a single set of hidden service directory from becoming a @@ -87,6 +89,7 @@ after the epoch, at 2016-04-12 12:00 UTC, and ended at 16904*1440*60 + (12*60*60) seconds after the epoch, at 2016-04-13 12:00 UTC. + ### When to publish a hidden service descriptor [WHEN-HSDESC] Hidden services periodically publish their descriptor to the responsible @@ -100,11 +103,12 @@ descriptor again to the responsible HSDirs for that time period. [TODO: Control republish period using a consensus parameter?] + #### Overlapping descriptors Hidden services need to upload multiple descriptors so that they can be reachable to clients with older or newer consensuses than them. Services -need to upload their descriptors to the HSDirs _before_ the beginning of +need to upload their descriptors to the HSDirs *before* the beginning of each upcoming time period, so that they are readily available for clients to fetch them. Furthermore, services should keep uploading their old descriptor even after the end of a time period, so that they can be reachable by @@ -120,6 +124,7 @@ achieved. [TODO: What to do when we run multiple hidden services in a single host?] + ### Where to publish a hidden service descriptor [WHERE-HSDESC] This section specifies how the HSDir hash ring is formed at any given @@ -184,6 +189,7 @@ Again, nodes from lower-numbered replicas are disregarded when choosing the spread for a replica. + ### Using time periods and SRVs to fetch/upload HS descriptors [FETCHUPLOADDESC] Hidden services and clients need to make correct use of time periods (TP) @@ -221,6 +227,7 @@ Let's start with an illustration of the system: ``` + #### Client behavior for fetching descriptors [CLIENTFETCH] And here is how clients use TPs and SRVs to fetch descriptors: @@ -250,6 +257,7 @@ SRV#1 for fetching descriptors. Also, if a client (C2) is at 01:00 right after SRV#2, it will still use TP#1 and SRV#1. + #### Service behavior for uploading descriptors [SERVICEUPLOAD] As discussed above, services maintain two active descriptors at any time. We @@ -263,6 +271,7 @@ Services like clients also employ a different logic for picking SRV and TP values based on their position in the graph above. Here is the logic: + ##### First descriptor upload logic [FIRSTDESCUPLOAD] Here is the service logic for uploading its first descriptor: @@ -298,6 +307,7 @@ Consider that the service is at 01:00 right after SRV#2: it will upload its first descriptor using TP#1 and SRV#1. + ##### Second descriptor upload logic [SECONDDESCUPLOAD] Here is the service logic for uploading its second descriptor: @@ -331,6 +341,7 @@ Consider that the service is at 01:00 right after SRV#2: it will upload its second descriptor using TP#2 and SRV#2. + #### Directory behavior for handling descriptor uploads [DIRUPLOAD] Upon receiving a hidden service descriptor publish request, directories MUST @@ -356,6 +367,7 @@ service (required for decrypting the first layer of encryption), or the necessary client credentials (for decrypting the second layer). + ### Expiring hidden service descriptors [EXPIRE-DESC] Hidden services set their descriptor's "descriptor-lifetime" field to 180 @@ -368,6 +380,7 @@ as descriptors including those intro points are valid (even if that's after the time period has changed). + ### URLs for anonymous uploading and downloading Hidden service descriptors conforming to this specification are uploaded @@ -381,6 +394,7 @@ These requests must be made anonymously, on circuits not used for anything else. + ### Client-side validation of onion addresses When a Tor client receives a prop224 onion address from the user, it @@ -402,4 +416,3 @@ only occur malevolently and never naturally) is to extract the ed25519 public key from the onion address and multiply it by the ed25519 group order and ensure that the result is the ed25519 identity element. For more details, please see [TORSION-REFS]. - diff --git a/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md b/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md index 674f584..7919b73 100644 --- a/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md +++ b/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md @@ -1,4 +1,5 @@ + # Encoding onion addresses [ONIONADDRESS] The onion address of a hidden service includes its identity public key, a @@ -24,4 +25,3 @@ encoded as shown below: For more information about this encoding, please see our discussion thread at [ONIONADDRESS-REFS]. - diff --git a/spec/rend-spec-v3/encrypting-data-between-client-host.md b/spec/rend-spec-v3/encrypting-data-between-client-host.md index 41cb1fa..f3ce6f7 100644 --- a/spec/rend-spec-v3/encrypting-data-between-client-host.md +++ b/spec/rend-spec-v3/encrypting-data-between-client-host.md @@ -1,4 +1,5 @@ + # Encrypting data between client and host A successfully completed handshake, as embedded in the @@ -9,4 +10,3 @@ Tor relay encryption protocol, applying encryption with these keys before other encryption, and decrypting with these keys before other decryption. The client encrypts with Kf and decrypts with Kb; the service host does the opposite. - diff --git a/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md b/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md index 403ef31..a540bec 100644 --- a/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md +++ b/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md @@ -1,8 +1,8 @@ + # Generating and publishing hidden service descriptors [HSDIR] Hidden service descriptors follow the same metaformat as other Tor directory objects. They are published anonymously to Tor servers with the HSDir flag, HSDir=2 protocol version and tor version >= 0.3.0.8 (because a bug was fixed in this version). - diff --git a/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md b/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md index 8c6982b..719d4fa 100644 --- a/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md +++ b/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md @@ -1,4 +1,5 @@ + ## Hidden service descriptors: encryption format [HS-DESC-ENC] Hidden service descriptors are protected by two layers of encryption. @@ -10,6 +11,7 @@ second layer of encryption is only useful when client authorization is enabled and protects against entities that do not possess valid client credentials. + ### First layer of encryption [HS-DESC-FIRST-LAYER] The first layer of HS descriptor encryption is designed to protect @@ -17,6 +19,7 @@ descriptor confidentiality against entities who don't know the public identity key of the hidden service. + #### First layer encryption logic The encryption keys and format for the first layer of encryption are @@ -39,6 +42,7 @@ Before encryption the plaintext is padded with NUL bytes to the nearest multiple of 10k bytes. + #### First layer plaintext format After clients decrypt the first layer of encryption, they need to parse the @@ -133,6 +137,7 @@ Here are all the supported fields: ``` + #### Client behavior [FIRST-LAYER-CLIENT-BEHAVIOR] ```text @@ -154,6 +159,7 @@ Here are all the supported fields: ``` + #### Hiding client authorization data ```text @@ -176,6 +182,7 @@ Here are all the supported fields: ``` + ### Second layer of encryption [HS-DESC-SECOND-LAYER] The second layer of descriptor encryption is designed to protect descriptor @@ -188,6 +195,7 @@ If client authorization is disabled, then the second layer of HS encryption does not offer any additional security, but is still used. + #### Second layer encryption keys The encryption keys and format for the second layer of encryption are @@ -204,6 +212,7 @@ parameters as follows: ``` + #### Second layer plaintext format After decrypting the second layer ciphertext, clients can finally learn the @@ -386,6 +395,7 @@ implementations MUST accept this section even if it is missing its final newline. + ### Deriving hidden service descriptor encryption keys [HS-DESC-ENCRYPTION-KEYS] In this section we present the generic encryption format for hidden service @@ -428,6 +438,7 @@ Here is the key generation logic: ``` + ### Number of introduction points [NUM_INTRO_POINT] This section defines how many introduction points an hidden service @@ -443,4 +454,3 @@ The reason for a maximum value of 20 is to give enough scalability to tools like OnionBalance to be able to load balance up to 120 servers (20 x 6 HSDirs) but also in order for the descriptor size to not overwhelmed hidden service directories with user defined values that could be gigantic. - diff --git a/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md b/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md index df6a790..f981e11 100644 --- a/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md +++ b/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md @@ -1,4 +1,5 @@ + ## Hidden service descriptors: outer wrapper [DESC-OUTER] The format for a hidden service descriptor is as follows, using the @@ -71,4 +72,3 @@ meta-format from dir-spec.txt. HSDirs accept hidden service descriptors of up to 50k bytes (a consensus parameter should also be introduced to control this value). - diff --git a/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md b/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md index 6c56bba..9f6cfc6 100644 --- a/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md +++ b/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md @@ -1,4 +1,5 @@ + # Appendix F: Hidden service directory format [HIDSERVDIR-FORMAT] This appendix section specifies the contents of the HiddenServiceDir directory: @@ -27,4 +28,3 @@ the client. See section [CLIENT-AUTH-MGMT] for more details and the format of the client file. (NOTE: client authorization is implemented as of 0.3.5.1-alpha.) - diff --git a/spec/rend-spec-v3/hidden-services-overview-preliminaries.md b/spec/rend-spec-v3/hidden-services-overview-preliminaries.md index a71790c..382c732 100644 --- a/spec/rend-spec-v3/hidden-services-overview-preliminaries.md +++ b/spec/rend-spec-v3/hidden-services-overview-preliminaries.md @@ -1,5 +1,6 @@ -# Hidden services: overview and preliminaries. + +# Hidden services: overview and preliminaries Hidden services aim to provide responder anonymity for bidirectional stream-based communication on the Tor network. Unlike regular Tor @@ -32,7 +33,8 @@ Operator -- A person running a hidden service ``` -## Improvements over previous versions. + +## Improvements over previous versions Here is a list of improvements of this proposal over the legacy hidden services: @@ -46,6 +48,7 @@ f) Offline keys for onion services g) Advanced client authorization + ## Notation and vocabulary Unless specified otherwise, all multi-octet integers are big-endian. @@ -72,6 +75,7 @@ unsigned integer "val" in N bytes. For example, INT_4(1337) is [00 00 INT_4(42) is 42 % 4294967296 (32 bit). + ## Cryptographic building blocks This specification uses the following cryptographic building blocks: @@ -129,8 +133,8 @@ This specification uses the following cryptographic building blocks: where k_len is htonll(len(k)). ``` - When we need a particular MAC key length below, we choose - MAC_KEY_LEN=32 (256 bits). + When we need a particular MAC key length below, we choose + MAC_KEY_LEN=32 (256 bits). For legacy purposes, we specify compatibility with older versions of the Tor introduction point and rendezvous point protocols. These used @@ -142,6 +146,7 @@ themselves, but over those strings prefixed with a distinguishing value. + ## Protocol building blocks [BUILDING-BLOCKS] In sections below, we need to transmit the locations and identities @@ -177,6 +182,7 @@ material unless they control the secret key corresponding to the server's public key. + ## Assigned relay cell types These relay cell types are reserved for use in the hidden service @@ -237,6 +243,7 @@ protocol. ``` + ## Acknowledgments This design includes ideas from many people, including @@ -306,4 +313,3 @@ editing help from Please forgive me if I've missed you; please forgive me if I've misunderstood your best ideas here too. - diff --git a/spec/rend-spec-v3/introduction-protocol-intro-protocol.md b/spec/rend-spec-v3/introduction-protocol-intro-protocol.md index a9813c2..e1977fc 100644 --- a/spec/rend-spec-v3/introduction-protocol-intro-protocol.md +++ b/spec/rend-spec-v3/introduction-protocol-intro-protocol.md @@ -1,4 +1,5 @@ + # The introduction protocol [INTRO-PROTOCOL] The introduction protocol proceeds in three steps. @@ -28,9 +29,11 @@ the introduction circuit to the hidden service host, and acknowledges the introduction request to the client. + ## Registering an introduction point [REG_INTRO_POINT] + ### Extensible ESTABLISH_INTRO protocol. [EST_INTRO] When a hidden service is establishing a new introduction point, it @@ -111,6 +114,7 @@ Otherwise, the node must associate the key with the circuit, for use later in INTRODUCE1 cells. + #### Denial-of-Service Defense Extension. [EST_INTRO_DOS_EXT] This extension can be used to send Denial-of-Service (DoS) parameters to @@ -210,6 +214,7 @@ Older versions of Tor always use a 1024-bit RSA key for these introduction authentication keys. + ### Acknowledging establishment of introduction point [INTRO_ESTABLISHED] After setting up an introduction circuit, the introduction point reports its @@ -234,6 +239,7 @@ The same rules for multiplicity, ordering, and handling unknown types apply to the extension fields here as described [EST_INTRO] above. + ## Sending an INTRODUCE1 cell to the introduction point. [SEND_INTRO1] In order to participate in the introduction protocol, a client must @@ -260,6 +266,7 @@ or that its request will not succeed. ``` + ### INTRODUCE1 cell format [FMT_INTRO1] When a client is connecting to an introduction point, INTRODUCE1 cells @@ -302,6 +309,7 @@ The same rules for multiplicity, ordering, and handling unknown types apply to the extension fields here as described [EST_INTRO] above. + ### INTRODUCE_ACK cell format. [INTRO_ACK] An INTRODUCE_ACK cell has the following fields: @@ -326,6 +334,7 @@ The same rules for multiplicity, ordering, and handling unknown types apply to the extension fields here as described [EST_INTRO] above. + ## Processing an INTRODUCE2 cell at the hidden service. [PROCESS_INTRO2] Upon receiving an INTRODUCE2 cell, the hidden service host checks whether @@ -422,6 +431,7 @@ The same rules for multiplicity, ordering, and handling unknown types apply to the extension fields here as described [EST_INTRO] above. + ### Introduction handshake encryption requirements [INTRO-HANDSHAKE-REQS] When decoding the encrypted information in an INTRODUCE2 cell, a @@ -569,6 +579,7 @@ computed in tor-spec.txt section 5.1.4, except that instead of using AES-128 and SHA1 for this hop, we use AES-256 and SHA3-256. + ## Authentication during the introduction phase. [INTRO-AUTH] Hidden services may restrict access only to authorized users. @@ -578,7 +589,8 @@ know the credential for a hidden service may connect at all. There is one defined authentication type: `ed25519`. -### Ed25519-based authentication `ed25519`. + +### Ed25519-based authentication `ed25519` (NOTE: This section is not implemented by Tor. It is likely that we would want to change its design substantially before @@ -618,4 +630,3 @@ on the authentication. Users SHOULD NOT use the same public key with multiple hidden services. - diff --git a/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md b/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md index 6802d4b..2051e86 100644 --- a/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md +++ b/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md @@ -1,4 +1,5 @@ + # Appendix G: Managing authorized client data [CLIENT-AUTH-MGMT] Hidden services and clients can configure their authorized client data either @@ -102,4 +103,3 @@ G.1.1. Hidden Service side configuration [XXX what happens when people use both the control port interface and the filesystem interface?] ``` - diff --git a/spec/rend-spec-v3/numeric-values-reserved-this-document.md b/spec/rend-spec-v3/numeric-values-reserved-this-document.md index 82eca4c..1e11c73 100644 --- a/spec/rend-spec-v3/numeric-values-reserved-this-document.md +++ b/spec/rend-spec-v3/numeric-values-reserved-this-document.md @@ -1,5 +1,5 @@ + # Appendix D: Numeric values reserved in this document [TODO: collect all the lists of commands and values mentioned above] - diff --git a/spec/rend-spec-v3/open-questions.md b/spec/rend-spec-v3/open-questions.md index af2b347..140b1f5 100644 --- a/spec/rend-spec-v3/open-questions.md +++ b/spec/rend-spec-v3/open-questions.md @@ -1,5 +1,6 @@ -# Open Questions: + +# Open Questions Scaling hidden services is hard. There are on-going discussions that you might be able to help with. See [SCALING-REFS]. @@ -77,4 +78,3 @@ References: https://lists.torproject.org/pipermail/tor-dev/2017-April/012164.html https://getmonero.org/2017/05/17/disclosure-of-a-major-bug-in-cryptonote-based-currencies.html ``` - diff --git a/spec/rend-spec-v3/protocol-overview.md b/spec/rend-spec-v3/protocol-overview.md index cb47184..1c03e49 100644 --- a/spec/rend-spec-v3/protocol-overview.md +++ b/spec/rend-spec-v3/protocol-overview.md @@ -1,4 +1,5 @@ + # Protocol overview In this section, we outline the hidden service protocol. This section @@ -6,6 +7,7 @@ omits some details in the name of simplicity; those are given more fully below, when we specify the protocol in more detail. + ## View from 10,000 feet A hidden service host prepares to offer a hidden service by choosing @@ -44,6 +46,7 @@ or processes configured by the server; RELAY_DATA cells are used to communicate data on those streams, and so forth. + ## In more detail: naming hidden services [NAMING] A hidden service's name is its long term master identity key. This is @@ -69,6 +72,7 @@ their length. An older name might look like: ``` + ## In more detail: Access control [IMD:AC] Access control for a hidden service is imposed at multiple points through @@ -103,6 +107,7 @@ optional client authorization is enabled, the service may additionally require the client to prove knowledge of a pre-shared private key. + ## In more detail: Distributing hidden service descriptors. [IMD:DIST] Periodically, hidden service descriptors become stored at different @@ -147,6 +152,7 @@ its blinded signing key. The keys for the last period remain valid until the new keys come online. + ## In more detail: Scaling to multiple hosts This design is compatible with our current approaches for scaling hidden @@ -168,6 +174,7 @@ enable the use of older Tor nodes as rendezvous points and introduction points. + ## In more detail: Keeping crypto keys offline In this design, a hidden service's secret identity key may be @@ -194,6 +201,7 @@ only be used to create credentials for the descriptor signing keys. implemented as of 0.3.2.1-alpha.) + ## In more detail: Encryption Keys And Replay Resistance To avoid replays of an introduction request by an introduction point, @@ -204,6 +212,7 @@ time can create a problematic fingerprint. (See proposal 222 for more discussion.) + ## In more detail: A menagerie of keys [In the text below, an "encryption keypair" is roughly "a keypair you @@ -282,6 +291,7 @@ Public/private keypairs defined in this document: ``` + ### In even more detail: Client authorization keys [CLIENT-AUTH] When client authorization is enabled, each authorized client of a hidden @@ -316,4 +326,3 @@ keys should be managed. [TODO: Also specify stealth client authorization.] (NOTE: client authorization is implemented as of 0.3.5.1-alpha.) - diff --git a/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md b/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md index 51f3624..dce072c 100644 --- a/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md +++ b/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md @@ -1,4 +1,5 @@ + ## Publishing shared random values [PUB-SHAREDRANDOM] Our design for limiting the predictability of HSDir upload locations @@ -17,6 +18,7 @@ According to proposal 250, we add two new lines in consensuses: ``` + ### Client behavior in the absence of shared random values If the previous or current shared random value cannot be found in a @@ -33,6 +35,7 @@ rounded down; period_num is calculated as specified in found originally. + ### Hidden services and changing shared random values It's theoretically possible that the consensus shared random values will @@ -44,4 +47,3 @@ should use the new shared random values to find the new responsible HSDirs and upload their descriptors there. XXX How long should they upload descriptors there for? - diff --git a/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md b/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md index 9cbd8b9..3c7142b 100644 --- a/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md +++ b/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md @@ -1,4 +1,5 @@ + # Appendix C: Recommendations for searching for vanity .onions [VANITY] EDITORIAL NOTE: The author thinks that it's silly to brute-force the @@ -42,4 +43,3 @@ independently. See [VANITY-REFS] for a reference implementation of this vanity .onion search scheme. - diff --git a/spec/rend-spec-v3/rendezvous-protocol.md b/spec/rend-spec-v3/rendezvous-protocol.md index 65a70aa..316bcf2 100644 --- a/spec/rend-spec-v3/rendezvous-protocol.md +++ b/spec/rend-spec-v3/rendezvous-protocol.md @@ -1,4 +1,5 @@ + # The rendezvous protocol Before connecting to a hidden service, the client first builds a @@ -19,6 +20,7 @@ but use an anonymous 3-hop circuit if: ``` + ## Establishing a rendezvous point [EST_REND_POINT] The client sends the rendezvous point a RELAY_COMMAND_ESTABLISH_RENDEZVOUS @@ -47,6 +49,7 @@ The client should establish a rendezvous point BEFORE trying to connect to a hidden service. + ## Joining to a rendezvous point [JOIN_REND] To complete a rendezvous, the hidden service host builds a circuit to @@ -87,13 +90,14 @@ Now both parties use the handshake output to derive shared keys for use on the circuit as specified in the section below: + ### Key expansion The hidden service and its client need to derive crypto keys from the NTOR_KEY_SEED part of the handshake output. To do so, they use the KDF construction as follows: -K = KDF(NTOR_KEY_SEED | m_hsexpand, HASH_LEN * 2 + S_KEY_LEN * 2) +K = KDF(NTOR_KEY_SEED | m_hsexpand, HASH_LEN *2 + S_KEY_LEN* 2) The first HASH_LEN bytes of K form the forward digest Df; the next HASH_LEN bytes form the backward digest Db; the next S_KEY_LEN bytes form Kf, and the @@ -113,6 +117,7 @@ contents? It's not necessary, but it could be wise. Similarly, we should make it extensible.] + ## Using legacy hosts as rendezvous points [This section is obsolete and refers to a workaround for now-obsolete Tor @@ -131,4 +136,3 @@ Relays older than 0.2.9.1 should not be used for rendezvous points by next generation onion services because they enforce too-strict length checks to rendezvous cells. Hence the "HSRend" protocol from proposal#264 should be used to select relays for rendezvous points. - diff --git a/spec/rend-spec-v3/reserved-numbers.md b/spec/rend-spec-v3/reserved-numbers.md index 2fe9fba..70c9e36 100644 --- a/spec/rend-spec-v3/reserved-numbers.md +++ b/spec/rend-spec-v3/reserved-numbers.md @@ -1,4 +1,5 @@ + # Appendix E: Reserved numbers We reserve these certificate type values for Ed25519 certificates: @@ -14,4 +15,3 @@ We reserve these certificate type values for Ed25519 certificates: Note: The value "0A" is skipped because it's reserved for the onion key cross-certifying ntor identity key from proposal 228. ``` - diff --git a/spec/rend-spec-v3/selecting-nodes-picknodes.md b/spec/rend-spec-v3/selecting-nodes-picknodes.md index cb4cb13..ef5edb4 100644 --- a/spec/rend-spec-v3/selecting-nodes-picknodes.md +++ b/spec/rend-spec-v3/selecting-nodes-picknodes.md @@ -1,4 +1,5 @@ + # Appendix B: Selecting nodes [PICKNODES] Picking introduction points @@ -7,4 +8,3 @@ Building paths Reusing circuits (TODO: This needs a writeup) - diff --git a/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md b/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md index f4f950d..d0393dd 100644 --- a/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md +++ b/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md @@ -1,7 +1,9 @@ + # Appendix A: Signature scheme with key blinding [KEYBLIND] + ## Key derivation overview As described in [IMD:DIST] and [SUBCRED] above, we require a "key @@ -34,6 +36,7 @@ There is a master keypair (sk, pk). ``` + ## Tor's key derivation scheme We propose the following scheme for key blinding, based on Ed25519. @@ -99,4 +102,3 @@ Verifying the signature: Check whether SB = R+hash(R,A',M)A'. See [KEYBLIND-REFS] for an extensive discussion on this scheme and possible alternatives. Also, see [KEYBLIND-PROOF] for a security proof of this scheme. - diff --git a/spec/rend-spec-v3/text-vectors.md b/spec/rend-spec-v3/text-vectors.md index 71897e8..ad3701b 100644 --- a/spec/rend-spec-v3/text-vectors.md +++ b/spec/rend-spec-v3/text-vectors.md @@ -1,4 +1,5 @@ + # Appendix G: Text vectors G.1. Test vectors for hs-ntor / NTOR-WITH-EXTRA-DATA @@ -98,4 +99,3 @@ AUTH_INPUT_MAC, and computes the same NTOR_KEY_SEED. Now that both parties have the same NTOR_KEY_SEED, they can derive the shared key material they will use for their circuit. - diff --git a/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md b/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md index 45ebc10..9088cf7 100644 --- a/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md +++ b/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md @@ -1,5 +1,6 @@ -# Appendix F: Two methods for managing revision counters. + +# Appendix F: Two methods for managing revision counters Implementations MAY generate revision counters in any way they please, so long as they are monotonically increasing over the lifetime of each @@ -78,4 +79,3 @@ F.1. Increment-on-generation increase forever without resetting it -- doing so links the service across changes in the blinded public key. ``` - diff --git a/spec/socks-extensions.md b/spec/socks-extensions.md index 011c119..6486215 100644 --- a/spec/socks-extensions.md +++ b/spec/socks-extensions.md @@ -13,6 +13,7 @@ Table of Contents ``` + # Overview The SOCKS protocol provides a generic interface for TCP proxies. Client @@ -32,14 +33,17 @@ SOCKS4 to allow addressing by hostname; SOCKS5 supports IPv4, IPv6, and hostnames. + ## Extent of support Tor supports the SOCKS4, SOCKS4A, and SOCKS5 standards, except as follows: BOTH: + - The BIND command is not supported. SOCKS4,4A: + - SOCKS4 usernames are used to implement stream isolation. ```text @@ -67,6 +71,7 @@ SOCKS4,4A: manpage.) + # Name lookup As an extension to SOCKS4A and SOCKS5, Tor implements a new command value, @@ -86,7 +91,8 @@ address" portion of the reply. (This command was not supported before Tor 0.1.2.2-alpha.) -# Other command extensions. + +# Other command extensions Tor 0.1.2.4-alpha added a new command value: "CONNECT_DIR" [F2]. In this case, Tor will open an encrypted direct TCP connection to the @@ -98,6 +104,7 @@ The F2 command value was removed in Tor 0.2.0.10-alpha in favor of a new use_begindir flag in edge_connection_t. + # HTTP-resistance Tor checks the first byte of each SOCKS request to see whether it looks @@ -107,6 +114,7 @@ misconfigured. This is helpful for the many users who mistakenly try to use Tor as an HTTP proxy instead of a SOCKS proxy. + # Optimistic data Tor allows SOCKS clients to send connection data before Tor has sent a @@ -119,6 +127,7 @@ their connection has succeeded or failed _after_ they have sent the data. + # Extended error codes We define a set of additional extension error codes that can be returned @@ -129,7 +138,7 @@ connections. via the ExtendedErrors flag. In Arti, these error codes are enabled whenever onion services are.) -* X'F0' Onion Service Descriptor Can Not be Found +- X'F0' Onion Service Descriptor Can Not be Found ```text The requested onion service descriptor can't be found on the hashring @@ -185,4 +194,3 @@ References: [3] SOCKS5: RFC 1928 https://www.ietf.org/rfc/rfc1928.txt [4] RFC 1929: https://www.ietf.org/rfc/rfc1929.txt ``` - diff --git a/spec/srv-spec-intro.md b/spec/srv-spec-intro.md index d818603..fecb4f8 100644 --- a/spec/srv-spec-intro.md +++ b/spec/srv-spec-intro.md @@ -47,4 +47,3 @@ Table Of Contents: 6.3. Why can't we recover if the 00:00UTC consensus fails? 7. Acknowledgements ``` - diff --git a/spec/srv-spec/acknowledgements.md b/spec/srv-spec/acknowledgements.md index 171cfde..edeb53e 100644 --- a/spec/srv-spec/acknowledgements.md +++ b/spec/srv-spec/acknowledgements.md @@ -1,4 +1,5 @@ + # Acknowledgements Thanks to everyone who has contributed to this design with feedback and @@ -26,4 +27,3 @@ References: [VDFS]: https://eprint.iacr.org/2018/601.pdf ``` - diff --git a/spec/srv-spec/discussion.md b/spec/srv-spec/discussion.md index 9d469fc..ea38318 100644 --- a/spec/srv-spec/discussion.md +++ b/spec/srv-spec/discussion.md @@ -1,7 +1,9 @@ + # Discussion + ## Why the added complexity from proposal 225? The complexity difference between this proposal and prop225 is in part @@ -10,6 +12,7 @@ clients. This proposal spends lots of effort specifying how the two shared random values can always be readily accessible to clients. + ## Why do you do a commit-and-reveal protocol in 24 rounds? The reader might be wondering why we span the protocol over the course of a @@ -28,6 +31,7 @@ big problem. Also, this way we give more chances for a failing dirauth to recover and rejoin the protocol. + ## Why can't we recover if the 00:00UTC consensus fails? If the 00:00UTC consensus fails, there will be no shared random value for @@ -36,4 +40,3 @@ randomness of the day at 01:00UTC instead. However, the engineering issues with adding such recovery logic are too great. For example, it's not easy for an authority who just booted to learn whether a specific consensus failed to be created. - diff --git a/spec/srv-spec/introduction.md b/spec/srv-spec/introduction.md index 5ed203b..37d6473 100644 --- a/spec/srv-spec/introduction.md +++ b/spec/srv-spec/introduction.md @@ -1,7 +1,9 @@ + # Introduction + ## Motivation For the next generation hidden services project, we need the Tor network to @@ -17,6 +19,7 @@ Tor-related protocols (e.g. OnioNS) or even non-Tor-related (e.g. warrant canaries). + ## Previous work Proposal 225 specifies a commit-and-reveal protocol that can be run as an @@ -25,4 +28,3 @@ However, directory authority operators feel unsafe running a third-party script that opens TCP ports and accepts connections from the Internet. Hence, this proposal aims to embed the commit-and-reveal idea in the Tor voting process which should make it smoother to deploy and maintain. - diff --git a/spec/srv-spec/overview.md b/spec/srv-spec/overview.md index 728aad2..6da4599 100644 --- a/spec/srv-spec/overview.md +++ b/spec/srv-spec/overview.md @@ -1,4 +1,5 @@ + # Overview This proposal alters the Tor consensus protocol such that a random number is @@ -10,6 +11,7 @@ The proposal also specifies how the final shared random value is embedded in consensus documents so that clients who need it can get it. + ## Introduction to our commit-and-reveal protocol Every day, before voting for the consensus at 00:00UTC each authority @@ -23,6 +25,7 @@ commitment value you should not be able to derive the underlying reveal value. The construction of these values is specified in section [COMMITREVEAL]. + ## Ten thousand feet view of the protocol Our commit-and-reveal protocol aims to produce a fresh shared random value @@ -58,6 +61,7 @@ Commit phase: ``` + ## How we use the consensus [CONS] The produced shared random values need to be readily available to @@ -81,6 +85,7 @@ For this, a new SR consensus method will be needed to indicate which authorities support this new protocol. + ### Inserting Shared Random Values in the consensus After voting happens, we need to be careful on how we pick which shared @@ -118,6 +123,7 @@ this because changing SRVs in the middle of the day has terrible reachability consequences for hidden service clients. + ## Persistent State of the Protocol [STATE] A directory authority needs to keep a persistent state on disk of the on @@ -133,11 +139,12 @@ previous time period must also be present in the state at all times if they are available. + ## Protocol Illustration An illustration for better understanding the protocol can be found here: -https://people.torproject.org/~asn/hs_notes/shared_rand.jpg + It reads left-to-right. @@ -155,4 +162,3 @@ example, the SRV was computed with 3 authority contributions and its value is "a56fg39h". We advice you to revisit this after you have read the whole document. - diff --git a/spec/srv-spec/protocol.md b/spec/srv-spec/protocol.md index 4567f89..7a17004 100644 --- a/spec/srv-spec/protocol.md +++ b/spec/srv-spec/protocol.md @@ -1,4 +1,5 @@ + # Protocol In this section we give a detailed specification of the protocol. We @@ -8,6 +9,7 @@ encoding of the messages is specified in the next section ([SPEC]). Now we go through the phases of the protocol: + ## Commitment Phase [COMMITMENTPHASE] The commit phase lasts from 00:00UTC to 12:00UTC. @@ -20,6 +22,7 @@ in their permanent state. We call a commit by Alice "authoritative" if it was included in Alice's vote. + ### Voting During Commitment Phase During the commit phase, each authority includes in its votes: @@ -39,6 +42,7 @@ phase, only the first commitment should be taken in account by other authorities. Any subsequent commitments MUST be ignored. + ### Persistent State During Commitment Phase [STATECOMMIT] During the commitment phase, authorities save in their persistent state the @@ -46,6 +50,7 @@ authoritative commits they have received from each authority. Only one commit per authority must be considered trusted and active at a given time. + ## Reveal Phase The reveal phase lasts from 12:00UTC to 00:00UTC. @@ -54,6 +59,7 @@ Now that the commitments have been agreed on, it's time for authorities to reveal their random values. + ### Voting During Reveal Phase During the reveal phase, each authority includes in its votes: @@ -70,6 +76,7 @@ commitment during the reveal phase or introduce a new commitment, the new commitment MUST be ignored. + ### Persistent State During Reveal Phase [STATEREVEAL] During the reveal phase, authorities keep the authoritative commits from the @@ -82,6 +89,7 @@ MUST wait till the next voting round before including that reveal value in its votes. + ## Shared Random Value Calculation At 00:00UTC Finally, at 00:00UTC every day, authorities compute a fresh shared random @@ -98,6 +106,7 @@ Apart from that, authorities at 00:00UTC proceed voting normally as they would in the first round of the commitment phase (section [COMMITMENTPHASE]). + ### Shared Randomness Calculation [SRCALC] An authority that wants to derive the shared random value SRV, should use @@ -123,6 +132,7 @@ To maintain consistent ordering in HASHED_REVEALS, all the ID_a | R_a pairs are ordered based on the R_a value in ascending order. + ## Bootstrapping Procedure As described in [CONS], two shared random values are required for the HSDir @@ -133,6 +143,7 @@ consensus. This should happen after three 00:00UTC consensuses have been produced, which takes 48 hours. + ## Rebooting Directory Authorities [REBOOT] The shared randomness protocol must be able to support directory @@ -152,4 +163,3 @@ Finally, authorities MUST implement their persistent state in such a way that th will never commit two different values in the same protocol run, even if they have to reboot in the middle (assuming that their persistent state file is kept). A suggested way to structure the persistent state is found at [STATEFORMAT]. - diff --git a/spec/srv-spec/security-analysis.md b/spec/srv-spec/security-analysis.md index 698cac5..658c2f8 100644 --- a/spec/srv-spec/security-analysis.md +++ b/spec/srv-spec/security-analysis.md @@ -1,7 +1,9 @@ + # Security Analysis + ## Security of commit-and-reveal and future directions The security of commit-and-reveal protocols is well understood, and has @@ -16,6 +18,7 @@ crypto and more complex protocols so this seems like an acceptable solution for now. Here are some examples of possible future directions: + - Schemes based on threshold signatures (e.g. see [HOPPER]) - Unicorn scheme by Lenstra et al. [UNICORN] - Schemes based on Verifiable Delay Functions [VDFS] @@ -24,6 +27,7 @@ For more alternative approaches on collaborative random number generation also see the discussion at [RNGMESSAGING]. + ## Predicting the shared random value during reveal phase The reveal phase lasts 12 hours, and most authorities will send their @@ -39,6 +43,7 @@ Any other protocols using the shared random value from this system should be aware of this property. + ## Partition attacks This design is not immune to certain partition attacks. We believe they @@ -50,6 +55,7 @@ attacks. Nevertheless, this section describes all possible partition attack and how to detect them. + ### Partition attacks during commit phase A malicious directory authority could send only its commit to one single @@ -67,6 +73,7 @@ coming from an authority should NEVER be different between authorities. If so, this means an attack is ongoing or very bad bug (highly unlikely). + ### Partition attacks during reveal phase Let's consider Alice, a malicious directory authority. Alice could wait @@ -95,4 +102,3 @@ will cause quite some noise. Furthermore, the authority needs to send different votes to different auths which is detectable. Like the commit phase attack, the detection here is to make sure that the commitment values in a vote coming from an authority are always the same for each authority. - diff --git a/spec/srv-spec/specification-spec.md b/spec/srv-spec/specification-spec.md index 1ff7db3..8a35d22 100644 --- a/spec/srv-spec/specification-spec.md +++ b/spec/srv-spec/specification-spec.md @@ -1,7 +1,9 @@ + # Specification [SPEC] + ## Voting This section describes how commitments, reveals and SR values are encoded in @@ -17,6 +19,7 @@ Participating authorities need to include the line: in their votes to announce that they take part in the protocol. + ### Computing commitments and reveals [COMMITREVEAL] A directory authority that wants to participate in this protocol needs to @@ -44,6 +47,7 @@ REVEAL = base64-encode( TIMESTAMP || H(RN) ) ``` + ### Validating commitments and reveals [VALIDATEVALUES] Given a COMMIT message and a REVEAL message it should be possible to verify @@ -59,6 +63,7 @@ Authorities should ignore reveal values during the Reveal Phase that don't correspond to commit values published during the Commitment Phase. + ### Encoding commit/reveal values in votes [COMMITVOTE] An authority puts in its vote the commitments and reveals it has produced and @@ -76,6 +81,7 @@ ALGNAME is the hash algorithm that should be used to compute COMMIT and REVEAL which is "sha3-256" for version 1. + ### Shared Random Value [SRVOTE] Authorities include a shared random value (SRV) in their votes using the @@ -94,12 +100,14 @@ To maintain consistent ordering, the shared random values of the previous period should be listed before the values of the current period. + ## Encoding Shared Random Values in the consensus [SRCONSENSUS] Authorities insert the two active shared random values in the consensus following the same encoding format as in [SRVOTE]. + ## Persistent state format [STATEFORMAT] As a way to keep ground truth state in this protocol, an authority MUST @@ -156,4 +164,3 @@ Finally is the shared random value section. This is the latest shared random value. The fields are the same as in section [SRVOTE]. ``` - diff --git a/spec/tor-spec-intro.md b/spec/tor-spec-intro.md index eb9a32b..185d852 100644 --- a/spec/tor-spec-intro.md +++ b/spec/tor-spec-intro.md @@ -88,4 +88,3 @@ Tor as they become obsolete. This specification is not a design document; most design criteria are not examined. For more information on why Tor acts as it does, see tor-design.pdf. - diff --git a/spec/tor-spec/application-connections-stream-management.md b/spec/tor-spec/application-connections-stream-management.md index 53b9b97..e30ce8b 100644 --- a/spec/tor-spec/application-connections-stream-management.md +++ b/spec/tor-spec/application-connections-stream-management.md @@ -1,3 +1,3 @@ -# Application connections and stream management +# Application connections and stream management diff --git a/spec/tor-spec/cell-packet-format.md b/spec/tor-spec/cell-packet-format.md index b26f012..2550da3 100644 --- a/spec/tor-spec/cell-packet-format.md +++ b/spec/tor-spec/cell-packet-format.md @@ -1,4 +1,5 @@ + # Cell Packet format The basic unit of communication for onion routers and onion @@ -121,4 +122,3 @@ VERSIONS and NETINFO cells are used to set up connections in link protocols v2 and higher; in link protocol v3 and higher, CERTS, AUTH_CHALLENGE, and AUTHENTICATE may also be used. See section 4 below. - diff --git a/spec/tor-spec/circuit-management.md b/spec/tor-spec/circuit-management.md index 3784bab..30e3284 100644 --- a/spec/tor-spec/circuit-management.md +++ b/spec/tor-spec/circuit-management.md @@ -1,3 +1,3 @@ -# Circuit management +# Circuit management diff --git a/spec/tor-spec/closing-streams.md b/spec/tor-spec/closing-streams.md index c85b3cc..9ef1b98 100644 --- a/spec/tor-spec/closing-streams.md +++ b/spec/tor-spec/closing-streams.md @@ -1,4 +1,5 @@ + ## Closing streams When an anonymized TCP connection is closed, or an edge node @@ -97,4 +98,3 @@ to 'CLOSED'. When a stream already in 'DONE_PACKAGING' receives a If an edge node encounters an error on any stream, it sends a 'RELAY_END' cell (if possible) and closes the stream immediately. - diff --git a/spec/tor-spec/connections.md b/spec/tor-spec/connections.md index d540489..a7ba661 100644 --- a/spec/tor-spec/connections.md +++ b/spec/tor-spec/connections.md @@ -1,4 +1,5 @@ + # Connections Connections between two Tor relays, or between a client and a relay, @@ -158,6 +159,7 @@ their IP address changes. Clients MAY send certificates using any of the above handshake variants. + ## Picking TLS ciphersuites Clients SHOULD send a ciphersuite list chosen to emulate some popular @@ -217,6 +219,7 @@ less than HASH_LEN bits. Responders SHOULD NOT select any SSLv3 ciphersuite other than the DHE+3DES suites listed above. + ## TLS security considerations Implementations MUST NOT allow TLS session resumption -- it can @@ -226,4 +229,3 @@ Feb 2013), and it plays havoc with forward secrecy guarantees. Implementations SHOULD NOT allow TLS compression -- although we don't know a way to apply a CRIME-style attack to current Tor directly, it's a waste of resources. - diff --git a/spec/tor-spec/create-created-cells.md b/spec/tor-spec/create-created-cells.md index a6d36a9..56134e4 100644 --- a/spec/tor-spec/create-created-cells.md +++ b/spec/tor-spec/create-created-cells.md @@ -1,4 +1,5 @@ + ## CREATE and CREATED cells Users set up circuits incrementally, one hop at a time. To create a @@ -69,6 +70,7 @@ DESTROY cell to tear down the circuit. [CREATE2 is handled by Tor 0.2.4.7-alpha and later.] + ### Choosing circuit IDs in create cells The CircID for a CREATE/CREATE2 cell is a nonzero integer, selected @@ -102,6 +104,7 @@ attempting to build new circuits on a channel, if a certain number of randomly chosen CircID values are all in use (today's Tor stops after 64). + ### EXTEND and EXTENDED cells To extend an existing circuit, the client sends an EXTEND or EXTEND2 @@ -202,6 +205,7 @@ When encoding a non-TAP handshake in an EXTEND cell, clients SHOULD use the format with 'client handshake type tag'. + ### The "TAP" handshake This handshake uses Diffie-Hellman in Z_p and RSA to compute a set of @@ -255,6 +259,7 @@ Once both parties have g^xy, they derive their shared circuit keys and 'derivative key data' value via the KDF-TOR function in 5.2.1. + ### The "ntor" handshake This handshake uses a set of DH handshakes to compute a set of @@ -333,6 +338,7 @@ into the keys needed for the Tor relay protocol, using the KDF described in 5.2.2 and the tag m_expand. + #### The "ntor-v3" handshake This handshake extends the ntor handshake to include support @@ -488,6 +494,7 @@ Now both parties share the same KEYSTREAM, and can use it to generate their circuit keys. + ### CREATE_FAST/CREATED_FAST cells When initializing the first hop of a circuit, the OP has already @@ -521,6 +528,7 @@ networkstatus parameter as described in dir-spec.txt. [Tor 0.3.1.1-alpha and later disable CREATE_FAST by default.] + ### Additional data in CREATE/CREATED cells Some handshakes (currently ntor-v3 defined above) allow the client or the @@ -571,4 +579,3 @@ format: sendme_inc [1 byte] ``` - diff --git a/spec/tor-spec/creating-circuits.md b/spec/tor-spec/creating-circuits.md index 31bd6a3..c69d2cf 100644 --- a/spec/tor-spec/creating-circuits.md +++ b/spec/tor-spec/creating-circuits.md @@ -1,4 +1,5 @@ + ## Creating circuits When creating a circuit through the network, the circuit creator @@ -79,6 +80,7 @@ until a break in traffic allows time to do so without harming network latency too greatly.) + ### Canonical connections It is possible for an attacker to launch a man-in-the-middle attack @@ -101,4 +103,3 @@ conditions hold: Trusting server IPs makes it easier to covertly impersonate a relay, after stealing its keys. ``` - diff --git a/spec/tor-spec/flow-control.md b/spec/tor-spec/flow-control.md index 4bd8910..960bd21 100644 --- a/spec/tor-spec/flow-control.md +++ b/spec/tor-spec/flow-control.md @@ -1,7 +1,9 @@ + # Flow control + ## Link throttling Each client or relay should do appropriate bandwidth throttling to @@ -25,6 +27,7 @@ to reads and writes for connections that are serving directory information. See proposal 111 for details. + ## Link padding Link padding can be created by sending PADDING or VPADDING cells @@ -60,6 +63,7 @@ zero (to avoid client distinguishability) and ignored by the recipient. For more details on padding behavior, see padding-spec.txt. + ## Circuit-level flow control To control a circuit's bandwidth usage, each OR keeps track of two @@ -111,6 +115,7 @@ RELAY_DATA cell within one increment window. In other word, every 100 cells (increment), random bytes should be introduced in at least one cell. + ### SENDME Cell Format A circuit-level RELAY_SENDME cell always has its StreamID=0. @@ -163,6 +168,7 @@ If the VERSION is unrecognized or below the minimum accepted version (taken from the consensus), the circuit should be torn down. + ## Stream-level flow control Edge nodes use RELAY_SENDME cells to implement end-to-end flow @@ -175,4 +181,3 @@ ten cell payloads remaining to be flushed at that edge. Stream-level RELAY_SENDME cells are distinguished by having nonzero StreamID. They are still empty; the body still SHOULD be ignored. - diff --git a/spec/tor-spec/handling-relay_early-cells.md b/spec/tor-spec/handling-relay_early-cells.md index e88c946..9adbced 100644 --- a/spec/tor-spec/handling-relay_early-cells.md +++ b/spec/tor-spec/handling-relay_early-cells.md @@ -1,4 +1,5 @@ + ## Handling relay_early cells A RELAY_EARLY cell is designed to limit the length any circuit can reach. @@ -17,4 +18,3 @@ RELAY_EARLY cells too, in order to partially conceal the circuit length. [Starting with Tor 0.2.3.11-alpha, relays should reject any EXTEND/EXTEND2 cell not received in a RELAY_EARLY cell.] - diff --git a/spec/tor-spec/handling-resource-exhaustion.md b/spec/tor-spec/handling-resource-exhaustion.md index b8e89eb..7179066 100644 --- a/spec/tor-spec/handling-resource-exhaustion.md +++ b/spec/tor-spec/handling-resource-exhaustion.md @@ -1,8 +1,10 @@ + # Handling resource exhaustion -## Memory exhaustion. + +## Memory exhaustion (See also dos-spec.md.) @@ -29,4 +31,3 @@ more memory is free again. We recommend the following algorithm: cells in both directions on that circuit. Count the amount of memory we recovered towards the total. ``` - diff --git a/spec/tor-spec/negotiating-initializing-connections.md b/spec/tor-spec/negotiating-initializing-connections.md index d281761..dd9be1d 100644 --- a/spec/tor-spec/negotiating-initializing-connections.md +++ b/spec/tor-spec/negotiating-initializing-connections.md @@ -1,4 +1,5 @@ + # Negotiating and initializing connections After Tor instances negotiate handshake with either the "renegotiation" or @@ -35,6 +36,7 @@ and did not permit any command other than VERSIONS as the first cell of the in-protocol handshake.] + ## Negotiating versions with VERSIONS cells There are multiple instances of the Tor link connection protocol. Any @@ -87,6 +89,7 @@ Link protocols differences are: ``` + ## CERTS cells The CERTS cell describes the keys that a Tor instance is claiming @@ -214,6 +217,7 @@ initiator has the ID it claims; to do so, the cells in 4.3 and 4.4 below must be exchanged. + ## AUTH_CHALLENGE cells An AUTH_CHALLENGE cell is a variable-length cell with the following @@ -236,6 +240,7 @@ accept. Only two authentication methods are defined right now: see 4.4.1 and 4.4.2 below. + ## AUTHENTICATE cells If an initiator wants to authenticate, it responds to the @@ -266,6 +271,7 @@ verified the certificates presented in the responder's CERTS cell, and authenticated the responder. + ### Link authentication type 1: RSA-SHA256-TLSSecret If AuthType is 1 (meaning "RSA-SHA256-TLSSecret"), then the @@ -315,7 +321,8 @@ claimed to have an Ed25519 identity. (There is no AuthType 2: It was reserved but never implemented.) -### Link authentication type 3: Ed25519-SHA256-RFC5705. + +### Link authentication type 3: Ed25519-SHA256-RFC5705 If AuthType is 3, meaning "Ed25519-SHA256-RFC5705", the Authentication field of the AuthType cell is as below: @@ -357,6 +364,7 @@ The server MUST ignore any extra bytes in the signed data after the RAND field. + ## NETINFO cells If version 2 or higher is negotiated, each party sends the other a @@ -400,4 +408,3 @@ since the other party can lie about the time or IP addresses it sees. Initiators SHOULD use "this OR's address" to make sure that they have connected to another OR at its canonical address. (See 5.3.1 below.) - diff --git a/spec/tor-spec/opening-streams-transferring-data.md b/spec/tor-spec/opening-streams-transferring-data.md index dc92d51..8300ace 100644 --- a/spec/tor-spec/opening-streams-transferring-data.md +++ b/spec/tor-spec/opening-streams-transferring-data.md @@ -1,4 +1,5 @@ + ## Opening streams and transferring data To open a new anonymized TCP connection, the OP chooses an open @@ -88,6 +89,7 @@ Relay RELAY_DROP cells are long-range dummies; upon receiving such a cell, the OR or OP must drop it. + ### Opening a directory stream If a Tor relay is a directory server, it should respond to a @@ -96,6 +98,7 @@ connection to its directory port. RELAY_BEGIN_DIR cells ignore exit policy, since the stream is local to the Tor process. Directory servers may be: + * authoritative directories (RELAY_BEGIN_DIR, usually non-anonymous), * bridge authoritative directories (RELAY_BEGIN_DIR, anonymous), * directory mirrors (RELAY_BEGIN_DIR, usually non-anonymous), @@ -114,4 +117,3 @@ the payload. [RELAY_BEGIN_DIR was not supported before Tor 0.1.2.2-alpha; clients SHOULD NOT send it to routers running earlier versions of Tor.] - diff --git a/spec/tor-spec/preliminaries.md b/spec/tor-spec/preliminaries.md index 5cb0410..c97e774 100644 --- a/spec/tor-spec/preliminaries.md +++ b/spec/tor-spec/preliminaries.md @@ -1,4 +1,5 @@ + # Preliminaries ```text @@ -9,6 +10,7 @@ ``` + ## Notation and encoding ```text @@ -31,6 +33,7 @@ We use "byte" and "octet" interchangeably. Possibly we shouldn't. Some specs mention "base32". This means RFC4648, without "=" padding. + ### Encoding integers Unless we explicitly say otherwise below, all numeric values in the @@ -39,6 +42,7 @@ integer" means a big-endian 32-bit integer; a "2-byte" integer means a big-endian 16-bit integer, and so forth. + ## Security parameters Tor uses a stream cipher, a public-key cipher, the Diffie-Hellman @@ -67,6 +71,7 @@ KEY_LEN -- the length of the stream cipher's key, in bytes. ``` + ## Ciphers These are the ciphers we use _unless otherwise specified_. Several of @@ -122,7 +127,8 @@ strong pseudorandom number generator seeded from a strong entropy source, unless otherwise noted. -## A bad hybrid encryption algorithm, for legacy purposes. + +## A bad hybrid encryption algorithm, for legacy purposes Some specifications will refer to the "legacy hybrid encryption" of a byte sequence M with a public key KP. It is computed as follows: @@ -143,4 +149,3 @@ allows attackers to modify the bytes not covered by the OAEP -- see Goldberg's PET2006 paper for details. Do not use it as the basis for new protocols! Also note that as used in Tor's protocols, case 1 never occurs. - diff --git a/spec/tor-spec/relay-cells.md b/spec/tor-spec/relay-cells.md index 2f89832..68e0391 100644 --- a/spec/tor-spec/relay-cells.md +++ b/spec/tor-spec/relay-cells.md @@ -1,4 +1,5 @@ + ## Relay cells Within a circuit, the OP and the end node use the contents of @@ -7,6 +8,7 @@ RELAY packets to tunnel end-to-end commands and TCP connections by either edge; streams are initiated by the OP. End nodes that accept streams may be: + * exit relays (RELAY_BEGIN, anonymous), * directory servers (RELAY_BEGIN_DIR, anonymous or non-anonymous), * onion services (RELAY_BEGIN, anonymous via a rendezvous point). @@ -113,6 +115,7 @@ understood, the cell must be dropped and ignored. Its contents still count with respect to the digests and flow control windows, though. + ### Calculating the 'Digest' field The 'Digest' field itself serves the purpose to check if a cell has been @@ -156,4 +159,3 @@ The caveat itself is that only the binary data with the digest bytes set to zero are being taken into account when calculating the running digest. The final plain-text cells (with the digest field set to its actual value) are not taken into the running digest. - diff --git a/spec/tor-spec/remote-hostname-lookup.md b/spec/tor-spec/remote-hostname-lookup.md index 3abb2cf..ba78cf1 100644 --- a/spec/tor-spec/remote-hostname-lookup.md +++ b/spec/tor-spec/remote-hostname-lookup.md @@ -1,4 +1,5 @@ + ## Remote hostname lookup To find the address associated with a hostname, the OP sends a @@ -40,4 +41,3 @@ of the form: corresponding RELAY_RESOLVED cell must use the same streamID. No stream is actually created by the OR when resolving the name. ``` - diff --git a/spec/tor-spec/routing-relay-cells.md b/spec/tor-spec/routing-relay-cells.md index 95888f7..e2c784c 100644 --- a/spec/tor-spec/routing-relay-cells.md +++ b/spec/tor-spec/routing-relay-cells.md @@ -1,7 +1,9 @@ + ## Routing relay cells + ### Circuit ID Checks When a node wants to send a RELAY or RELAY_EARLY cell, it checks the cell's @@ -13,12 +15,14 @@ circID and determines whether it has a corresponding circuit along that connection. If not, the node drops the cell. + ### Forward Direction The forward direction is the direction that CREATE/CREATE2 cells are sent. + #### Routing from the Origin When a relay cell is sent from an OP, the OP encrypts the payload @@ -32,6 +36,7 @@ with the stream cipher as follows: ``` + #### Relaying Forward at Onion Routers When a forward relay cell is received by an OR, it decrypts the payload @@ -53,12 +58,14 @@ sends a DESTROY cell to tear down the circuit. For more information, see section 6 below. + ### Backward Direction The backward direction is the opposite direction from CREATE/CREATE2 cells. + #### Relaying Backward at Onion Routers When a backward relay cell is received by an OR, it encrypts the payload @@ -70,6 +77,7 @@ with the stream cipher, as follows: ``` + ### Routing to the Origin When a relay cell arrives at an OP, the OP decrypts the payload @@ -83,4 +91,3 @@ with the stream cipher as follows: The sending node is I. Stop and process the payload. ``` - diff --git a/spec/tor-spec/setting-circuit-keys.md b/spec/tor-spec/setting-circuit-keys.md index d566ff6..b49f75d 100644 --- a/spec/tor-spec/setting-circuit-keys.md +++ b/spec/tor-spec/setting-circuit-keys.md @@ -1,7 +1,9 @@ + ## Setting circuit keys + ### KDF-TOR This key derivation function is used by the TAP and CREATE_FAST @@ -33,6 +35,7 @@ is used to encrypt the stream of data going from the OP to the OR, and Kb is used to encrypt the stream of data going from the OR to the OP. + ### KDF-RFC5869 For newer KDF needs, Tor uses the key derivation function HKDF from @@ -57,4 +60,3 @@ forward digest Df; the next HASH_LEN form the backward digest Db; the next KEY_LEN form Kf, the next KEY_LEN form Kb, and the final DIGEST_LEN bytes are taken as a nonce to use in the place of KH in the hidden service protocol. Excess bytes from K are discarded. - diff --git a/spec/tor-spec/subprotocol-versioning.md b/spec/tor-spec/subprotocol-versioning.md index 0fd5388..3611c73 100644 --- a/spec/tor-spec/subprotocol-versioning.md +++ b/spec/tor-spec/subprotocol-versioning.md @@ -1,4 +1,5 @@ + # Subprotocol versioning This section specifies the Tor subprotocol versioning. They are broken down @@ -57,6 +58,7 @@ For relays, we will additionally Recommend all protocols which we recommend for clients. + ## "Link" The "link" protocols are those used by clients and relays to initiate and @@ -72,6 +74,7 @@ information. All current Tor versions support "1-3"; versions from support "1-5". Eventually we will drop "1" and "2". + ## "LinkAuth" LinkAuth protocols correspond to varieties of Authenticate cells used for @@ -86,6 +89,7 @@ Current versions are: "3" is the ed25519 link authentication described in 4.4.2 above. + ## "Relay" The "relay" protocols are those used to handle CREATE/CREATE2 @@ -159,6 +163,7 @@ Current versions are: ``` + ## "HSIntro" The "HSIntro" protocol handles introduction points. @@ -175,6 +180,7 @@ The "HSIntro" protocol handles introduction points. ``` + ## "HSRend" The "HSRend" protocol handles rendezvous points. @@ -187,6 +193,7 @@ The "HSRend" protocol handles rendezvous points. ``` + ## "HSDir" The "HSDir" protocols are the set of hidden service document types that can @@ -201,6 +208,7 @@ of URLs available to fetch them. ``` + ## "DirCache" The "DirCache" protocols are the set of documents available for download @@ -212,6 +220,7 @@ fetch them. (This excludes URLs for hidden service objects.) "2" -- adds support for consensus diffs in Tor 0.3.1.1-alpha. + ## "Desc" Describes features present or absent in descriptors. @@ -226,6 +235,7 @@ to understand ed25519 identities. identities. + ## "Microdesc" Describes features present or absent in microdescriptors. @@ -239,6 +249,7 @@ consensus methods. "2" -- consensus method 21 (adds ed25519 keys to microdescs). + ## "Cons" Describes features present or absent in consensus documents. @@ -253,6 +264,7 @@ These correspond more or less with consensus methods. "2" -- consensus method 21 (adds ed25519 keys to microdescs). + ## "Padding" Describes the padding capabilities of the relay. @@ -268,6 +280,7 @@ Describes the padding capabilities of the relay. ``` + ## "FlowCtrl" Describes the flow control protocol at the circuit and stream level. If @@ -284,6 +297,7 @@ control features (version 0). ``` + ## "Datagram" Describes the UDP protocol capabilities of a relay. @@ -293,4 +307,3 @@ Describes the UDP protocol capabilities of a relay. CONNECT_UDP, CONNECTED_UDP and DATAGRAM. See proposal 339 for more details. (Not yet advertised, reserved) ``` - diff --git a/spec/tor-spec/system-overview.md b/spec/tor-spec/system-overview.md index 4897d95..8577928 100644 --- a/spec/tor-spec/system-overview.md +++ b/spec/tor-spec/system-overview.md @@ -1,16 +1,18 @@ + # System overview Tor is a distributed overlay network designed to anonymize low-latency TCP-based applications such as web browsing, secure shell, and instant messaging. Clients choose a path through the network and -build a ``circuit'', in which each node (or ``onion router'' or ``OR'') +build a ``circuit'', in which each node (or``onion router'' or ``OR'') in the path knows its predecessor and successor, but no other nodes in the circuit. Traffic flowing down the circuit is sent in fixed-size ``cells'', which are unwrapped by a symmetric key at each node (like the layers of an onion) and relayed downstream. + ## Keys and names Every Tor relay has multiple public/private keypairs: @@ -70,4 +72,3 @@ The same key or keypair should never be used for separate roles within the Tor protocol suite, unless specifically stated. For example, a relay's identity keys K_relayid should not also be used as the identity keypair for a hidden service K_hs_id (see rend-spec-v3.txt). - diff --git a/spec/tor-spec/tearing-down-circuits.md b/spec/tor-spec/tearing-down-circuits.md index 52f1cc8..3008c05 100644 --- a/spec/tor-spec/tearing-down-circuits.md +++ b/spec/tor-spec/tearing-down-circuits.md @@ -1,4 +1,5 @@ + ## Tearing down circuits Circuits are torn down when an unrecoverable error occurs along @@ -104,4 +105,3 @@ The error codes are: 11 -- DESTROYED (The circuit was destroyed w/o client TRUNCATE) 12 -- NOSUCHSERVICE (Request for unknown hidden service) ``` - diff --git a/spec/version-spec.md b/spec/version-spec.md index 296839b..a4b6373 100644 --- a/spec/version-spec.md +++ b/spec/version-spec.md @@ -9,6 +9,7 @@ Table of Contents ``` + # The Old Way Before 0.1.0, versions were of the format: @@ -29,6 +30,7 @@ say, "0.0.8". Our first pre-release would be "0.0.8pre1", followed by and any eventual bugfix release would be "0.0.8.1". + # The New Way Starting at 0.1.0.1-rc, versions are of the format: @@ -65,7 +67,8 @@ Between these releases, CVS is versioned with a -cvs tag: after suffix instead of the "-cvs" suffix. -# Version status. + +# Version status ```text Sometimes we need to determine whether a Tor version is obsolete, @@ -90,4 +93,3 @@ suffix instead of the "-cvs" suffix. * Finally, if none of the above conditions hold, then the version is "un-recommended." ``` - -- cgit v1.2.3-54-g00ecf