From f7e5a95ee96d8ef52c1732d066c1249a6f84391e Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Thu, 12 Oct 2023 12:27:58 -0400 Subject: Convert text specifications to mdbook. --- spec/.htaccess | 21 + spec/SUMMARY.md | 163 ++ spec/address-spec.md | 108 ++ spec/bandwidth-file-spec-intro.md | 33 + spec/bandwidth-file-spec/definitions.md | 55 + spec/bandwidth-file-spec/format-details.md | 11 + spec/bandwidth-file-spec/header-list-format.md | 414 +++++ spec/bandwidth-file-spec/implementation-details.md | 394 ++++ spec/bandwidth-file-spec/relay-line-format.md | 126 ++ spec/bandwidth-file-spec/sample-data.md | 118 ++ spec/bandwidth-file-spec/scaling-bandwidths.md | 128 ++ spec/bandwidth-file-spec/scope-preliminaries.md | 92 + spec/bridgedb-spec.md | 440 +++++ spec/cert-spec.md | 223 +++ spec/control-spec-intro.md | 93 + spec/control-spec/commands.md | 1876 ++++++++++++++++++++ spec/control-spec/implementation-notes.md | 709 ++++++++ spec/control-spec/message-format.md | 180 ++ spec/control-spec/protocol-outline.md | 70 + spec/control-spec/replies.md | 1764 ++++++++++++++++++ spec/control-spec/scope.md | 22 + spec/dir-list-spec.md | 568 ++++++ spec/dir-spec-intro.md | 70 + ...epting-server-descriptor-extra-info-document.md | 52 + spec/dir-spec/assigning-flags-vote.md | 172 ++ spec/dir-spec/client-operation.md | 292 +++ spec/dir-spec/computing-microdescriptors.md | 169 ++ spec/dir-spec/consensus-negotiation-timeline.md | 16 + ...ting-curve25519-public-key-to-ed25519-public.md | 77 + spec/dir-spec/creating-key-certificates.md | 90 + .../directory-authority-operation-formats.md | 10 + spec/dir-spec/directory-cache-operation.md | 157 ++ spec/dir-spec/exchanging-votes.md | 53 + spec/dir-spec/extra-info-document-format.md | 591 ++++++ spec/dir-spec/general-use-http-urls.md | 132 ++ spec/dir-spec/inferring-missing-proto-lines.md | 15 + spec/dir-spec/limited-ed-diff-format.md | 39 + spec/dir-spec/nonterminals-server-descriptors.md | 32 + spec/dir-spec/outline.md | 249 +++ spec/dir-spec/router-operation-formats.md | 3 + spec/dir-spec/scope-preliminaries.md | 139 ++ spec/dir-spec/server-descriptor-format.md | 508 ++++++ spec/dir-spec/serving-bandwidth-list-files.md | 868 +++++++++ spec/dir-spec/standards-compliance.md | 79 + ...ding-server-descriptors-extra-info-documents.md | 43 + .../vote-consensus-status-document-formats.md | 706 ++++++++ spec/ext-orport-spec.md | 254 +++ spec/gettor-spec.md | 99 ++ spec/glossary.md | 216 +++ spec/guard-spec-intro.md | 40 + spec/guard-spec/algorithm.md | 594 +++++++ spec/guard-spec/appendices.md | 211 +++ ...uit-creation-entry-guard-selection-1000-foot.md | 72 + spec/guard-spec/introduction-motivation.md | 45 + spec/guard-spec/state-instances.md | 49 + .../still-non-addressed-issues-sectiontodo.md | 29 + spec/padding-spec-intro.md | 40 + spec/padding-spec/acknowledgments.md | 28 + spec/padding-spec/circuit-level-padding.md | 281 +++ spec/padding-spec/connection-level-padding.md | 283 +++ spec/padding-spec/overview.md | 28 + spec/param-spec.md | 535 ++++++ spec/path-spec-intro.md | 66 + spec/path-spec/attaching-streams-to-circuits.md | 19 + spec/path-spec/building-circuits.md | 3 + spec/path-spec/cannibalizing-circuits.md | 15 + ...cting-route-manipulation-by-guard-nodes-path.md | 193 ++ spec/path-spec/general-operation.md | 94 + spec/path-spec/guard-nodes.md | 43 + spec/path-spec/handling-failure.md | 17 + spec/path-spec/hidden-service-related-circuits.md | 5 + ...n-to-give-up-timeout-on-circuit-construction.md | 293 +++ spec/path-spec/old-notes.md | 60 + spec/path-spec/path-selection-constraints.md | 130 ++ spec/path-spec/server-descriptor-purposes.md | 19 + spec/path-spec/when-we-build.md | 161 ++ spec/pt-spec-intro.md | 38 + spec/pt-spec/acknowledgments.md | 6 + spec/pt-spec/anonymity-considerations.md | 23 + spec/pt-spec/architecture-overview.md | 37 + .../example-client-pluggable-transport-session.md | 19 + .../example-server-pluggable-transport-session.md | 18 + spec/pt-spec/introduction.md | 26 + ...le-transport-client-per-connection-arguments.md | 38 + ...luggable-transport-configuration-environment.md | 255 +++ spec/pt-spec/pluggable-transport-naming.md | 13 + spec/pt-spec/pluggable-transport-shutdown.md | 32 + ...le-transport-to-parent-process-communication.md | 294 +++ spec/pt-spec/references.md | 22 + spec/pt-spec/specification.md | 52 + spec/rend-spec-v3-intro.md | 98 + ...deriving-blinded-keys-subcredentials-subcred.md | 405 +++++ .../encoding-onion-addresses-onionaddress.md | 27 + .../encrypting-data-between-client-host.md | 12 + ...rating-publishing-hidden-service-descriptors.md | 8 + ...ce-descriptors-encryption-format-hs-desc-enc.md | 446 +++++ ...service-descriptors-outer-wrapper-desc-outer.md | 74 + ...n-service-directory-format-hidservdir-format.md | 30 + .../hidden-services-overview-preliminaries.md | 309 ++++ .../introduction-protocol-intro-protocol.md | 621 +++++++ ...ging-authorized-client-data-client-auth-mgmt.md | 105 ++ .../numeric-values-reserved-this-document.md | 5 + spec/rend-spec-v3/open-questions.md | 80 + spec/rend-spec-v3/protocol-overview.md | 319 ++++ ...ishing-shared-random-values-pub-sharedrandom.md | 47 + ...ommendations-for-searching-for-vanity-onions.md | 45 + spec/rend-spec-v3/rendezvous-protocol.md | 134 ++ spec/rend-spec-v3/reserved-numbers.md | 17 + spec/rend-spec-v3/selecting-nodes-picknodes.md | 10 + .../signature-scheme-with-key-blinding-keyblind.md | 102 ++ spec/rend-spec-v3/text-vectors.md | 101 ++ .../two-methods-for-managing-revision-counters.md | 81 + spec/socks-extensions.md | 188 ++ spec/srv-spec-intro.md | 50 + spec/srv-spec/acknowledgements.md | 29 + spec/srv-spec/discussion.md | 39 + spec/srv-spec/introduction.md | 28 + spec/srv-spec/overview.md | 158 ++ spec/srv-spec/protocol.md | 155 ++ spec/srv-spec/security-analysis.md | 98 + spec/srv-spec/specification-spec.md | 159 ++ spec/tor-spec-intro.md | 91 + .../application-connections-stream-management.md | 3 + spec/tor-spec/cell-packet-format.md | 124 ++ spec/tor-spec/circuit-management.md | 3 + spec/tor-spec/closing-streams.md | 100 ++ spec/tor-spec/connections.md | 229 +++ spec/tor-spec/create-created-cells.md | 574 ++++++ spec/tor-spec/creating-circuits.md | 104 ++ spec/tor-spec/flow-control.md | 178 ++ spec/tor-spec/handling-relay_early-cells.md | 20 + spec/tor-spec/handling-resource-exhaustion.md | 32 + .../negotiating-initializing-connections.md | 403 +++++ spec/tor-spec/opening-streams-transferring-data.md | 117 ++ spec/tor-spec/preliminaries.md | 146 ++ spec/tor-spec/relay-cells.md | 159 ++ spec/tor-spec/remote-hostname-lookup.md | 43 + spec/tor-spec/routing-relay-cells.md | 86 + spec/tor-spec/setting-circuit-keys.md | 60 + spec/tor-spec/subprotocol-versioning.md | 296 +++ spec/tor-spec/system-overview.md | 73 + spec/tor-spec/tearing-down-circuits.md | 107 ++ spec/version-spec.md | 93 + 143 files changed, 23986 insertions(+) create mode 100644 spec/.htaccess create mode 100644 spec/SUMMARY.md create mode 100644 spec/address-spec.md create mode 100644 spec/bandwidth-file-spec-intro.md create mode 100644 spec/bandwidth-file-spec/definitions.md create mode 100644 spec/bandwidth-file-spec/format-details.md create mode 100644 spec/bandwidth-file-spec/header-list-format.md create mode 100644 spec/bandwidth-file-spec/implementation-details.md create mode 100644 spec/bandwidth-file-spec/relay-line-format.md create mode 100644 spec/bandwidth-file-spec/sample-data.md create mode 100644 spec/bandwidth-file-spec/scaling-bandwidths.md create mode 100644 spec/bandwidth-file-spec/scope-preliminaries.md create mode 100644 spec/bridgedb-spec.md create mode 100644 spec/cert-spec.md create mode 100644 spec/control-spec-intro.md create mode 100644 spec/control-spec/commands.md create mode 100644 spec/control-spec/implementation-notes.md create mode 100644 spec/control-spec/message-format.md create mode 100644 spec/control-spec/protocol-outline.md create mode 100644 spec/control-spec/replies.md create mode 100644 spec/control-spec/scope.md create mode 100644 spec/dir-list-spec.md create mode 100644 spec/dir-spec-intro.md create mode 100644 spec/dir-spec/accepting-server-descriptor-extra-info-document.md create mode 100644 spec/dir-spec/assigning-flags-vote.md create mode 100644 spec/dir-spec/client-operation.md create mode 100644 spec/dir-spec/computing-microdescriptors.md create mode 100644 spec/dir-spec/consensus-negotiation-timeline.md create mode 100644 spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md create mode 100644 spec/dir-spec/creating-key-certificates.md create mode 100644 spec/dir-spec/directory-authority-operation-formats.md create mode 100644 spec/dir-spec/directory-cache-operation.md create mode 100644 spec/dir-spec/exchanging-votes.md create mode 100644 spec/dir-spec/extra-info-document-format.md create mode 100644 spec/dir-spec/general-use-http-urls.md create mode 100644 spec/dir-spec/inferring-missing-proto-lines.md create mode 100644 spec/dir-spec/limited-ed-diff-format.md create mode 100644 spec/dir-spec/nonterminals-server-descriptors.md create mode 100644 spec/dir-spec/outline.md create mode 100644 spec/dir-spec/router-operation-formats.md create mode 100644 spec/dir-spec/scope-preliminaries.md create mode 100644 spec/dir-spec/server-descriptor-format.md create mode 100644 spec/dir-spec/serving-bandwidth-list-files.md create mode 100644 spec/dir-spec/standards-compliance.md create mode 100644 spec/dir-spec/uploading-server-descriptors-extra-info-documents.md create mode 100644 spec/dir-spec/vote-consensus-status-document-formats.md create mode 100644 spec/ext-orport-spec.md create mode 100644 spec/gettor-spec.md create mode 100644 spec/glossary.md create mode 100644 spec/guard-spec-intro.md create mode 100644 spec/guard-spec/algorithm.md create mode 100644 spec/guard-spec/appendices.md create mode 100644 spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md create mode 100644 spec/guard-spec/introduction-motivation.md create mode 100644 spec/guard-spec/state-instances.md create mode 100644 spec/guard-spec/still-non-addressed-issues-sectiontodo.md create mode 100644 spec/padding-spec-intro.md create mode 100644 spec/padding-spec/acknowledgments.md create mode 100644 spec/padding-spec/circuit-level-padding.md create mode 100644 spec/padding-spec/connection-level-padding.md create mode 100644 spec/padding-spec/overview.md create mode 100644 spec/param-spec.md create mode 100644 spec/path-spec-intro.md create mode 100644 spec/path-spec/attaching-streams-to-circuits.md create mode 100644 spec/path-spec/building-circuits.md create mode 100644 spec/path-spec/cannibalizing-circuits.md create mode 100644 spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md create mode 100644 spec/path-spec/general-operation.md create mode 100644 spec/path-spec/guard-nodes.md create mode 100644 spec/path-spec/handling-failure.md create mode 100644 spec/path-spec/hidden-service-related-circuits.md create mode 100644 spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md create mode 100644 spec/path-spec/old-notes.md create mode 100644 spec/path-spec/path-selection-constraints.md create mode 100644 spec/path-spec/server-descriptor-purposes.md create mode 100644 spec/path-spec/when-we-build.md create mode 100644 spec/pt-spec-intro.md create mode 100644 spec/pt-spec/acknowledgments.md create mode 100644 spec/pt-spec/anonymity-considerations.md create mode 100644 spec/pt-spec/architecture-overview.md create mode 100644 spec/pt-spec/example-client-pluggable-transport-session.md create mode 100644 spec/pt-spec/example-server-pluggable-transport-session.md create mode 100644 spec/pt-spec/introduction.md create mode 100644 spec/pt-spec/pluggable-transport-client-per-connection-arguments.md create mode 100644 spec/pt-spec/pluggable-transport-configuration-environment.md create mode 100644 spec/pt-spec/pluggable-transport-naming.md create mode 100644 spec/pt-spec/pluggable-transport-shutdown.md create mode 100644 spec/pt-spec/pluggable-transport-to-parent-process-communication.md create mode 100644 spec/pt-spec/references.md create mode 100644 spec/pt-spec/specification.md create mode 100644 spec/rend-spec-v3-intro.md create mode 100644 spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md create mode 100644 spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md create mode 100644 spec/rend-spec-v3/encrypting-data-between-client-host.md create mode 100644 spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md create mode 100644 spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md create mode 100644 spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md create mode 100644 spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md create mode 100644 spec/rend-spec-v3/hidden-services-overview-preliminaries.md create mode 100644 spec/rend-spec-v3/introduction-protocol-intro-protocol.md create mode 100644 spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md create mode 100644 spec/rend-spec-v3/numeric-values-reserved-this-document.md create mode 100644 spec/rend-spec-v3/open-questions.md create mode 100644 spec/rend-spec-v3/protocol-overview.md create mode 100644 spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md create mode 100644 spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md create mode 100644 spec/rend-spec-v3/rendezvous-protocol.md create mode 100644 spec/rend-spec-v3/reserved-numbers.md create mode 100644 spec/rend-spec-v3/selecting-nodes-picknodes.md create mode 100644 spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md create mode 100644 spec/rend-spec-v3/text-vectors.md create mode 100644 spec/rend-spec-v3/two-methods-for-managing-revision-counters.md create mode 100644 spec/socks-extensions.md create mode 100644 spec/srv-spec-intro.md create mode 100644 spec/srv-spec/acknowledgements.md create mode 100644 spec/srv-spec/discussion.md create mode 100644 spec/srv-spec/introduction.md create mode 100644 spec/srv-spec/overview.md create mode 100644 spec/srv-spec/protocol.md create mode 100644 spec/srv-spec/security-analysis.md create mode 100644 spec/srv-spec/specification-spec.md create mode 100644 spec/tor-spec-intro.md create mode 100644 spec/tor-spec/application-connections-stream-management.md create mode 100644 spec/tor-spec/cell-packet-format.md create mode 100644 spec/tor-spec/circuit-management.md create mode 100644 spec/tor-spec/closing-streams.md create mode 100644 spec/tor-spec/connections.md create mode 100644 spec/tor-spec/create-created-cells.md create mode 100644 spec/tor-spec/creating-circuits.md create mode 100644 spec/tor-spec/flow-control.md create mode 100644 spec/tor-spec/handling-relay_early-cells.md create mode 100644 spec/tor-spec/handling-resource-exhaustion.md create mode 100644 spec/tor-spec/negotiating-initializing-connections.md create mode 100644 spec/tor-spec/opening-streams-transferring-data.md create mode 100644 spec/tor-spec/preliminaries.md create mode 100644 spec/tor-spec/relay-cells.md create mode 100644 spec/tor-spec/remote-hostname-lookup.md create mode 100644 spec/tor-spec/routing-relay-cells.md create mode 100644 spec/tor-spec/setting-circuit-keys.md create mode 100644 spec/tor-spec/subprotocol-versioning.md create mode 100644 spec/tor-spec/system-overview.md create mode 100644 spec/tor-spec/tearing-down-circuits.md create mode 100644 spec/version-spec.md diff --git a/spec/.htaccess b/spec/.htaccess new file mode 100644 index 0000000..13ca84d --- /dev/null +++ b/spec/.htaccess @@ -0,0 +1,21 @@ +# Automatically generated redirects +Redirect 302 "/tor-spec" "/tor-spec-intro.html" +Redirect 302 "/cert-spec" "/cert-spec.html" +Redirect 302 "/dir-spec" "/dir-spec-intro.html" +Redirect 302 "/srv-spec" "/srv-spec-intro.html" +Redirect 302 "/path-spec" "/path-spec-intro.html" +Redirect 302 "/guard-spec" "/guard-spec-intro.html" +Redirect 302 "/padding-spec" "/padding-spec-intro.html" +Redirect 302 "/socks-extensions" "/socks-extensions.html" +Redirect 302 "/address-spec" "/address-spec.html" +Redirect 302 "/rend-spec-v3" "/rend-spec-v3-intro.html" +Redirect 302 "/bridgedb-spec" "/bridgedb-spec.html" +Redirect 302 "/ext-orport-spec" "/ext-orport-spec.html" +Redirect 302 "/pt-spec" "/pt-spec-intro.html" +Redirect 302 "/gettor-spec" "/gettor-spec.html" +Redirect 302 "/control-spec" "/control-spec-intro.html" +Redirect 302 "/version-spec" "/version-spec.html" +Redirect 302 "/bandwidth-file-spec" "/bandwidth-file-spec-intro.html" +Redirect 302 "/dir-list-spec" "/dir-list-spec.html" +Redirect 302 "/param-spec" "/param-spec.html" +Redirect 302 "/glossary" "/glossary.html" diff --git a/spec/SUMMARY.md b/spec/SUMMARY.md new file mode 100644 index 0000000..e478e08 --- /dev/null +++ b/spec/SUMMARY.md @@ -0,0 +1,163 @@ +# Summary + +[Introduction](./INTRO.md) +# The core Tor protocol + +- [`Tor Protocol Specification`](./tor-spec-intro.md) + - [Preliminaries](./tor-spec/preliminaries.md) + - [System overview](./tor-spec/system-overview.md) + - [Connections](./tor-spec/connections.md) + - [Cell Packet format](./tor-spec/cell-packet-format.md) + - [Negotiating and initializing connections](./tor-spec/negotiating-initializing-connections.md) + - [Circuit management](./tor-spec/circuit-management.md) + - [CREATE and CREATED cells](./tor-spec/create-created-cells.md) + - [Setting circuit keys](./tor-spec/setting-circuit-keys.md) + - [Creating circuits](./tor-spec/creating-circuits.md) + - [Tearing down circuits](./tor-spec/tearing-down-circuits.md) + - [Routing relay cells](./tor-spec/routing-relay-cells.md) + - [Handling relay_early cells](./tor-spec/handling-relay_early-cells.md) + - [Application connections and stream management](./tor-spec/application-connections-stream-management.md) + - [Relay cells](./tor-spec/relay-cells.md) + - [Opening streams and transferring data](./tor-spec/opening-streams-transferring-data.md) + - [Closing streams](./tor-spec/closing-streams.md) + - [Remote hostname lookup](./tor-spec/remote-hostname-lookup.md) + - [Flow control](./tor-spec/flow-control.md) + - [Handling resource exhaustion](./tor-spec/handling-resource-exhaustion.md) + - [Subprotocol versioning](./tor-spec/subprotocol-versioning.md) + - [`Ed25519 certificates in Tor`](./cert-spec.md) +- [`Tor directory protocol, version 3`](./dir-spec-intro.md) + - [Scope and preliminaries](./dir-spec/scope-preliminaries.md) + - [Outline](./dir-spec/outline.md) + - [Router operation and formats](./dir-spec/router-operation-formats.md) + - [Uploading server descriptors and extra-info documents](./dir-spec/uploading-server-descriptors-extra-info-documents.md) + - [Server descriptor format](./dir-spec/server-descriptor-format.md) + - [Extra-info document format](./dir-spec/extra-info-document-format.md) + - [Nonterminals in server descriptors](./dir-spec/nonterminals-server-descriptors.md) + - [Directory authority operation and formats](./dir-spec/directory-authority-operation-formats.md) + - [Creating key certificates](./dir-spec/creating-key-certificates.md) + - [Accepting server descriptor and extra-info document uploads](./dir-spec/accepting-server-descriptor-extra-info-document.md) + - [Computing microdescriptors](./dir-spec/computing-microdescriptors.md) + - [Exchanging votes](./dir-spec/exchanging-votes.md) + - [Vote and consensus status document formats](./dir-spec/vote-consensus-status-document-formats.md) + - [Assigning flags in a vote](./dir-spec/assigning-flags-vote.md) + - [Serving bandwidth list files](./dir-spec/serving-bandwidth-list-files.md) + - [Directory cache operation](./dir-spec/directory-cache-operation.md) + - [Client operation](./dir-spec/client-operation.md) + - [Standards compliance](./dir-spec/standards-compliance.md) + - [Consensus-negotiation timeline.](./dir-spec/consensus-negotiation-timeline.md) + - [General-use HTTP URLs](./dir-spec/general-use-http-urls.md) + - [Converting a curve25519 public key to an ed25519 public key](./dir-spec/converting-curve25519-public-key-to-ed25519-public.md) + - [Inferring missing proto lines.](./dir-spec/inferring-missing-proto-lines.md) + - [Limited ed diff format](./dir-spec/limited-ed-diff-format.md) + - [`Tor Shared Random Subsystem Specification`](./srv-spec-intro.md) + - [Introduction](./srv-spec/introduction.md) + - [Overview](./srv-spec/overview.md) + - [Protocol](./srv-spec/protocol.md) + - [Specification [SPEC]](./srv-spec/specification-spec.md) + - [Security Analysis](./srv-spec/security-analysis.md) + - [Discussion](./srv-spec/discussion.md) + - [Acknowledgements](./srv-spec/acknowledgements.md) +- [`Tor Path Specification`](./path-spec-intro.md) + - [General operation](./path-spec/general-operation.md) + - [Building circuits](./path-spec/building-circuits.md) + - [When we build](./path-spec/when-we-build.md) + - [Path selection and constraints](./path-spec/path-selection-constraints.md) + - [Cannibalizing circuits](./path-spec/cannibalizing-circuits.md) + - [Learning when to give up ("timeout") on circuit construction](./path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md) + - [Handling failure](./path-spec/handling-failure.md) + - [Attaching streams to circuits](./path-spec/attaching-streams-to-circuits.md) + - [Hidden-service related circuits](./path-spec/hidden-service-related-circuits.md) + - [Guard nodes](./path-spec/guard-nodes.md) + - [Server descriptor purposes](./path-spec/server-descriptor-purposes.md) + - [Detecting route manipulation by Guard nodes (Path Bias)](./path-spec/detecting-route-manipulation-by-guard-nodes-path.md) + - [Old notes](./path-spec/old-notes.md) +- [`Tor Guard Specification`](./guard-spec-intro.md) + - [Introduction and motivation](./guard-spec/introduction-motivation.md) + - [State instances](./guard-spec/state-instances.md) + - [Circuit Creation, Entry Guard Selection (1000 foot view)](./guard-spec/circuit-creation-entry-guard-selection-1000-foot.md) + - [The algorithm.](./guard-spec/algorithm.md) + - [Appendices](./guard-spec/appendices.md) + - [Still non-addressed issues [Section:TODO]](./guard-spec/still-non-addressed-issues-sectiontodo.md) +- [`Tor Padding Specification`](./padding-spec-intro.md) + - [Overview](./padding-spec/overview.md) + - [Connection-level padding](./padding-spec/connection-level-padding.md) + - [Circuit-level padding](./padding-spec/circuit-level-padding.md) + - [Acknowledgments](./padding-spec/acknowledgments.md) +- [Preventing Denial-Of-Service](./dos-spec.md) +# Additional behaviors for clients + +- [`Tor's extensions to the SOCKS protocol`](./socks-extensions.md) +- [`Special Hostnames in Tor`](./address-spec.md) +# Onion services + +- [`Tor Rendezvous Specification - Version 3`](./rend-spec-v3-intro.md) + - [Hidden services: overview and preliminaries.](./rend-spec-v3/hidden-services-overview-preliminaries.md) + - [Protocol overview](./rend-spec-v3/protocol-overview.md) + - [Generating and publishing hidden service descriptors [HSDIR]](./rend-spec-v3/generating-publishing-hidden-service-descriptors.md) + - [Deriving blinded keys and subcredentials [SUBCRED]](./rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md) + - [Publishing shared random values [PUB-SHAREDRANDOM]](./rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md) + - [Hidden service descriptors: outer wrapper [DESC-OUTER]](./rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md) + - [Hidden service descriptors: encryption format [HS-DESC-ENC]](./rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md) + - [The introduction protocol [INTRO-PROTOCOL]](./rend-spec-v3/introduction-protocol-intro-protocol.md) + - [The rendezvous protocol](./rend-spec-v3/rendezvous-protocol.md) + - [Encrypting data between client and host](./rend-spec-v3/encrypting-data-between-client-host.md) + - [Encoding onion addresses [ONIONADDRESS]](./rend-spec-v3/encoding-onion-addresses-onionaddress.md) + - [Open Questions:](./rend-spec-v3/open-questions.md) + - [Appendix A: Signature scheme with key blinding [KEYBLIND]](./rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md) + - [Appendix B: Selecting nodes [PICKNODES]](./rend-spec-v3/selecting-nodes-picknodes.md) + - [Appendix C: Recommendations for searching for vanity .onions [VANITY]](./rend-spec-v3/recommendations-for-searching-for-vanity-onions.md) + - [Appendix D: Numeric values reserved in this document](./rend-spec-v3/numeric-values-reserved-this-document.md) + - [Appendix E: Reserved numbers](./rend-spec-v3/reserved-numbers.md) + - [Appendix F: Hidden service directory format [HIDSERVDIR-FORMAT]](./rend-spec-v3/hidden-service-directory-format-hidservdir-format.md) + - [Appendix G: Managing authorized client data [CLIENT-AUTH-MGMT]](./rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md) + - [Appendix F: Two methods for managing revision counters.](./rend-spec-v3/two-methods-for-managing-revision-counters.md) + - [Appendix G: Text vectors](./rend-spec-v3/text-vectors.md) +# Anticensorship tools and protocols + +- [`BridgeDB specification`](./bridgedb-spec.md) +- [`Extended ORPort for pluggable transports`](./ext-orport-spec.md) +- [`Pluggable Transport Specification (Version 1)`](./pt-spec-intro.md) + - [Introduction](./pt-spec/introduction.md) + - [Architecture Overview](./pt-spec/architecture-overview.md) + - [Specification](./pt-spec/specification.md) + - [Pluggable Transport Naming](./pt-spec/pluggable-transport-naming.md) + - [Pluggable Transport Configuration Environment Variables](./pt-spec/pluggable-transport-configuration-environment.md) + - [Pluggable Transport To Parent Process Communication](./pt-spec/pluggable-transport-to-parent-process-communication.md) + - [Pluggable Transport Shutdown](./pt-spec/pluggable-transport-shutdown.md) + - [Pluggable Transport Client Per-Connection Arguments](./pt-spec/pluggable-transport-client-per-connection-arguments.md) + - [Anonymity Considerations](./pt-spec/anonymity-considerations.md) + - [References](./pt-spec/references.md) + - [Acknowledgments](./pt-spec/acknowledgments.md) + - [Appendix A: Example Client Pluggable Transport Session](./pt-spec/example-client-pluggable-transport-session.md) + - [Appendix B: Example Server Pluggable Transport Session](./pt-spec/example-server-pluggable-transport-session.md) +- [`GetTor specification`](./gettor-spec.md) +# For C Tor only + +- [`The Tor Control Protocol`](./control-spec-intro.md) + - [Scope](./control-spec/scope.md) + - [Protocol outline](./control-spec/protocol-outline.md) + - [Message format](./control-spec/message-format.md) + - [Commands](./control-spec/commands.md) + - [Replies](./control-spec/replies.md) + - [Implementation notes](./control-spec/implementation-notes.md) +- [`How Tor Version Numbers Work`](./version-spec.md) +# Less commonly needed file formats + +- [`Tor Bandwidth File Format`](./bandwidth-file-spec-intro.md) + - [Scope and preliminaries](./bandwidth-file-spec/scope-preliminaries.md) + - [Format details](./bandwidth-file-spec/format-details.md) + - [Definitions](./bandwidth-file-spec/definitions.md) + - [Header List format](./bandwidth-file-spec/header-list-format.md) + - [Relay Line format](./bandwidth-file-spec/relay-line-format.md) + - [Implementation details](./bandwidth-file-spec/implementation-details.md) + - [Sample data](./bandwidth-file-spec/sample-data.md) + - [Scaling bandwidths](./bandwidth-file-spec/scaling-bandwidths.md) +# Implementation details + +- [`Tor Directory List Format`](./dir-list-spec.md) +# Reserved names and numbers + +- [`Tor network parameters`](./param-spec.md) +# Unfinished + +- [`Glossary`](./glossary.md) diff --git a/spec/address-spec.md b/spec/address-spec.md new file mode 100644 index 0000000..272e366 --- /dev/null +++ b/spec/address-spec.md @@ -0,0 +1,108 @@ +```text + Special Hostnames in Tor + Nick Mathewson + +Table of Contents + + 1. Overview + 2. .exit + 3. .onion + 4. .noconnect +``` + + +# Overview + +Most of the time, Tor treats user-specified hostnames as opaque: When +the user connects to www.torproject.org, Tor picks an exit node and uses +that node to connect to "www.torproject.org". Some hostnames, however, +can be used to override Tor's default behavior and circuit-building +rules. + +These hostnames can be passed to Tor as the address part of a SOCKS4a or +SOCKS5 request. If the application is connected to Tor using an IP-only +method (such as SOCKS4, TransPort, or NATDPort), these hostnames can be +substituted for certain IP addresses using the MapAddress configuration +option or the MAPADDRESS control command. + + +# .exit + +```text + SYNTAX: [hostname].[name-or-digest].exit + [name-or-digest].exit +``` + +Hostname is a valid hostname; [name-or-digest] is either the nickname of a +Tor node or the hex-encoded digest of that node's public key. + +When Tor sees an address in this format, it uses the specified hostname as +the exit node. If no "hostname" component is given, Tor defaults to the +published IPv4 address of the exit node. + +It is valid to try to resolve hostnames, and in fact upon success Tor +will cache an internal mapaddress of the form +"www.google.com.foo.exit=64.233.161.99.foo.exit" to speed subsequent +lookups. + +The .exit notation is disabled by default as of Tor 0.2.2.1-alpha, due +to potential application-level attacks. + +```text + EXAMPLES: + www.example.com.exampletornode.exit + + Connect to www.example.com from the node called "exampletornode". + + exampletornode.exit + + Connect to the published IP address of "exampletornode" using + "exampletornode" as the exit. +``` + + +# .onion + +```text + SYNTAX: [digest].onion + [ignored].[digest].onion +``` + +Version 2 addresses (deprecated since 0.4.6.1-alpha), the digest is the first +eighty bits of a SHA1 hash of the identity key for a hidden service, encoded +in base32. + +Version 3 addresses, the digest is defined as: + +```text + onion_address = base32(PUBKEY | CHECKSUM | VERSION) + CHECKSUM = H(".onion checksum" | PUBKEY | VERSION)[:2] + + where: + - PUBKEY is the 32 bytes ed25519 master pubkey of the onion service. + - VERSION is a one byte version field (default value '\x03') + - ".onion checksum" is a constant string + - H is SHA3-256 + - CHECKSUM is truncated to two bytes before inserting it in onion_address +``` + +When Tor sees an address in this format, it tries to look up and connect to +the specified onion service. See rend-spec-v3.txt for full details. + +The "ignored" portion of the address is intended for use in vhosting, and +is supported in Tor 0.2.4.10-alpha and later. + + +# .noconnect + +SYNTAX: [string].noconnect + +When Tor sees an address in this format, it immediately closes the +connection without attaching it to any circuit. This is useful for +controllers that want to test whether a given application is indeed +using the same instance of Tor that they're controlling. + +This feature was added in Tor 0.1.2.4-alpha, and taken out in Tor +0.2.2.1-alpha over fears that it provided another avenue for detecting +Tor users via application-level web tricks. + diff --git a/spec/bandwidth-file-spec-intro.md b/spec/bandwidth-file-spec-intro.md new file mode 100644 index 0000000..1fd5354 --- /dev/null +++ b/spec/bandwidth-file-spec-intro.md @@ -0,0 +1,33 @@ +```text + Tor Bandwidth File Format + juga + teor + +Table of Contents + + 1. Scope and preliminaries + 1.2. Acknowledgements + 1.3. Outline + 1.4. Format Versions + 2. Format details + 2.1. Definitions + 2.2. Header List format + 2.3. Relay Line format + 2.4. Implementation details + 2.4.1. Writing bandwidth files atomically + 2.4.2. Additional KeyValue pair definitions + 2.4.2.1. Simple Bandwidth Scanner + 2.4.2.2. Torflow + A. Sample data + A.1. Generated by Torflow + A.2. Generated by sbws version 0.1.0 + A.3. Generated by sbws version 1.0.3 + A.4. Headers generated by sbws version 1.0.4 + A.5 Generated by sbws version 1.1.0 + B. Scaling bandwidths + B.1. Scaling requirements + B.2. A linear scaling method + B.3. Quota changes + B.4. Torflow aggregation +``` + diff --git a/spec/bandwidth-file-spec/definitions.md b/spec/bandwidth-file-spec/definitions.md new file mode 100644 index 0000000..2749193 --- /dev/null +++ b/spec/bandwidth-file-spec/definitions.md @@ -0,0 +1,55 @@ + +## Definitions + +The following nonterminals are defined in Tor directory protocol +sections 1.2., 2.1.1., 2.1.3.: + +```text + bool + Int + SP (space) + NL (newline) + KeywordChar + ArgumentChar + nickname + hexdigest (a '$', followed by 40 hexadecimal characters + ([A-Fa-f0-9])) + + Nonterminal defined section 2 of version-spec.txt [4]: + + version_number + + We define the following nonterminals: + + Line ::= ArgumentChar* NL + RelayLine ::= KeyValue (SP KeyValue)* NL + HeaderLine ::= KeyValue NL + KeyValue ::= Key "=" Value + Key ::= (KeywordChar | "_")+ + Value ::= ArgumentCharValue+ + ArgumentCharValue ::= any printing ASCII character except NL and SP. + Terminator ::= "=====" or "====" + Generators SHOULD use a 5-character terminator. + Timestamp ::= Int + Bandwidth ::= Int + MasterKey ::= a base64-encoded Ed25519 public key, with + padding characters omitted. + DateTime ::= "YYYY-MM-DDTHH:MM:SS", as in ISO 8601 + CountryCode ::= Two capital ASCII letters ([A-Z]{2}), as defined in + ISO 3166-1 alpha-2 plus "ZZ" to denote unknown country + (eg the destination is in a Content Delivery Network). + CountryCodeList ::= One or more CountryCode(s) separated by a comma + ([A-Z]{2}(,[A-Z]{2})*). +``` + +Note that key_value and value are defined in Tor directory protocol +with different formats to KeyValue and Value here. + +Tor versions earlier than 0.3.5.1-alpha require all lines in the file +to be 510 characters or less. The previous limit was 254 characters in +Tor 0.2.6.2-alpha and earlier. Parsers MAY ignore longer Lines. + +Note that directory authorities are only supported on the two most +recent stable Tor versions, so we expect that line limits will be +removed after Tor 0.4.0 is released in 2019. + diff --git a/spec/bandwidth-file-spec/format-details.md b/spec/bandwidth-file-spec/format-details.md new file mode 100644 index 0000000..247250a --- /dev/null +++ b/spec/bandwidth-file-spec/format-details.md @@ -0,0 +1,11 @@ + +# Format details + +```text + The Bandwidth File MUST contain the following sections: + - Header List (exactly once), which is a partially ordered list of + - Header Lines (one or more times), then + - Relay Lines (zero or more times), in an arbitrary order. + If it does not contain these sections, parsers SHOULD ignore the file. +``` + diff --git a/spec/bandwidth-file-spec/header-list-format.md b/spec/bandwidth-file-spec/header-list-format.md new file mode 100644 index 0000000..b751a93 --- /dev/null +++ b/spec/bandwidth-file-spec/header-list-format.md @@ -0,0 +1,414 @@ + +## Header List format + +It consists of a Timestamp line and zero or more HeaderLines. + +All the header lines MUST conform to the HeaderLine format, except +the first Timestamp line. + +The Timestamp line is not a HeaderLine to keep compatibility with +the legacy Bandwidth File format. + +Some header Lines MUST appear in specific positions, as documented +below. All other Lines can appear in any order. + +If a parser does not recognize any extra material in a header Line, +the Line MUST be ignored. + +If a header Line does not conform to this format, the Line SHOULD be +ignored by parsers. + +It consists of: + +Timestamp NL + +[At start, exactly once.] + +The Unix Epoch time in seconds of the most recent generator bandwidth +result. + +If the generator implementation has multiple threads or +subprocesses which can fail independently, it SHOULD take the most +recent timestamp from each thread and use the oldest value. This +ensures all the threads continue running. + +If there are threads that do not run continuously, they SHOULD be +excluded from the timestamp calculation. + +If there are no recent results, the generator MUST NOT generate a new +file. + +It does not follow the KeyValue format for backwards compatibility +with version 1.0.0. + +"version" version_number NL + +[In second position, zero or one time.] + +The specification document format version. +It uses semantic versioning [5]. + +This Line was added in version 1.1.0 of this specification. + +Version 1.0.0 documents do not contain this Line, and the +version_number is considered to be "1.0.0". + +"software" Value NL + +[Zero or one time.] + +The name of the software that created the document. + +This Line was added in version 1.1.0 of this specification. + +Version 1.0.0 documents do not contain this Line, and the software +is considered to be "torflow". + +"software_version" Value NL + +[Zero or one time.] + +The version of the software that created the document. +The version may be a version_number, a git commit, or some other +version scheme. + +This Line was added in version 1.1.0 of this specification. + +"file_created" DateTime NL + +[Zero or one time.] + +The date and time timestamp in ISO 8601 format and UTC time zone +when the file was created. + +This Line was added in version 1.1.0 of this specification. + +"generator_started" DateTime NL + +[Zero or one time.] + +The date and time timestamp in ISO 8601 format and UTC time zone +when the generator started. + +This Line was added in version 1.1.0 of this specification. + +"earliest_bandwidth" DateTime NL + +[Zero or one time.] + +The date and time timestamp in ISO 8601 format and UTC time zone +when the first relay bandwidth was obtained. + +This Line was added in version 1.1.0 of this specification. + +"latest_bandwidth" DateTime NL + +[Zero or one time.] + +The date and time timestamp in ISO 8601 format and UTC time zone +of the most recent generator bandwidth result. + +This time MUST be identical to the initial Timestamp line. + +This duplicate value is included to make the format easier for people +to read. + +This Line was added in version 1.1.0 of this specification. + +"number_eligible_relays" Int NL + +[Zero or one time.] + +The number of relays that have enough measurements to be +included in the bandwidth file. + +This Line was added in version 1.2.0 of this specification. + +"minimum_percent_eligible_relays" Int NL + +[Zero or one time.] + +The percentage of relays in the consensus that SHOULD be +included in every generated bandwidth file. + +If this threshold is not reached, format versions 1.3.0 and earlier +SHOULD NOT contain any relays. (Bandwidth files always include a +header.) + +Format versions 1.4.0 and later SHOULD include all the relays for +diagnostic purposes, even if this threshold is not reached. But these +relays SHOULD be marked so that Tor does not vote on them. +See section 1.4 for details. + +The minimum percentage is 60% in Torflow, so sbws uses +60% as the default. + +This Line was added in version 1.2.0 of this specification. + +"number_consensus_relays" Int NL + +[Zero or one time.] + +The number of relays in the consensus. + +This Line was added in version 1.2.0 of this specification. + +"percent_eligible_relays" Int NL + +[Zero or one time.] + +The number of eligible relays, as a percentage of the number +of relays in the consensus. + +```text + This line SHOULD be equal to: + (number_eligible_relays * 100.0) / number_consensus_relays + to the number of relays in the consensus to include in this file. + + This Line was added in version 1.2.0 of this specification. + + "minimum_number_eligible_relays" Int NL + + [Zero or one time.] +``` + +The minimum number of relays that SHOULD be included in the bandwidth +file. See minimum_percent_eligible_relays for details. + +```text + This line SHOULD be equal to: + number_consensus_relays * (minimum_percent_eligible_relays / 100.0) + + This Line was added in version 1.2.0 of this specification. + + "scanner_country" CountryCode NL + + [Zero or one time.] + + The country, as in political geolocation, where the generator is run. + + This Line was added in version 1.2.0 of this specification. + + "destinations_countries" CountryCodeList NL + + [Zero or one time.] +``` + +The country, as in political geolocation, or countries where the +destination Web server(s) are located. +The destination Web Servers serve the data that the generator retrieves +to measure the bandwidth. + +This Line was added in version 1.2.0 of this specification. + +"recent_consensus_count" Int NL + +[Zero or one time.]. + +The number of the different consensuses seen in the last data_period +days. (data_period is 5 by default.) + +```text + Assuming that Tor clients fetch a consensus every 1-2 hours, + and that the data_period is 5 days, the Value of this Key SHOULD be + between: + data_period * 24 / 2 = 60 + data_period * 24 = 120 + + This Line was added in version 1.4.0 of this specification. + + "recent_priority_list_count" Int NL + + [Zero or one time.] +``` + +The number of times that a list with a subset of relays prioritized +to be measured has been created in the last data_period days. +(data_period is 5 by default.) + +```text + In 2019, with 7000 relays in the network, the Value of this Key SHOULD be + approximately: + data_period * 24 / 1.5 = 80 + Being 1.5 the approximate number of hours it takes to measure a + priority list of 7000 * 0.05 (350) relays, when the fraction of relays + in a priority list is the 5% (0.05). + + This Line was added in version 1.4.0 of this specification. + + "recent_priority_relay_count" Int NL + + [Zero or one time.] +``` + +The number of relays that has been in in the list of relays prioritized +to be measured in the last data_period days. (data_period is 5 by +default.) + +```text + In 2019, with 7000 relays in the network, the Value of this Key SHOULD be + approximately: + 80 * (7000 * 0.05) = 28000 + Being 0.05 (5%) the fraction of relays in a priority list and 80 + the approximate number of priority lists (see + "recent_priority_list_count"). + + This Line was added in version 1.4.0 of this specification. + + "recent_measurement_attempt_count" Int NL + + [Zero or one time.] +``` + +The number of times that any relay has been queued to be measured +in the last data_period days. (data_period is 5 by default.) + +In 2019, with 7000 relays in the network, the Value of this Key SHOULD be +approximately the same as "recent_priority_relay_count", +assuming that there is one attempt to measure a relay for each relay that +has been prioritized unless there are system, network or implementation +issues. + +This Line was added in version 1.4.0 of this specification and removed +in version 1.5.0. + +"recent_measurement_failure_count" Int NL + +[Zero or one time.] + +The number of times that the scanner attempted to measure a relay in +the last data_period days (5 by default), but the relay has not been +measured because of system, network or implementation issues. + +This Line was added in version 1.4.0 of this specification. + +"recent_measurements_excluded_error_count" Int NL + +[Zero or one time.] + +The number of relays that have no successful measurements in the last +data_period days (5 by default). + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This Line was added in version 1.4.0 of this specification. + +"recent_measurements_excluded_near_count" Int NL + +[Zero or one time.] + +The number of relays that have some successful measurements in the last +data_period days (5 by default), but all those measurements were +performed in a period of time that was too short (by default 1 day). + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This Line was added in version 1.4.0 of this specification. + +"recent_measurements_excluded_old_count" Int NL + +[Zero or one time.] + +The number of relays that have some successful measurements, but all +those measurements are too old (more than 5 days, by default). + +Excludes relays that are already counted in +recent_measurements_excluded_near_count. + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This Line was added in version 1.4.0 of this specification. + +"recent_measurements_excluded_few_count" Int NL + +[Zero or one time.] + +The number of relays that don't have enough recent successful +measurements. (Fewer than 2 measurements in the last 5 days, by +default). + +Excludes relays that are already counted in +recent_measurements_excluded_near_count and +recent_measurements_excluded_old_count. + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This Line was added in version 1.4.0 of this specification. + +"time_to_report_half_network" Int NL + +[Zero or one time.] + +The time in seconds that it would take to report measurements about the +half of the network, given the number of eligible relays and the time +it took in the last days (5 days, by default). + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This Line was added in version 1.4.0 of this specification. + +"tor_version" version_number NL + +[Zero or one time.] + +The Tor version of the Tor process controlled by the generator. + +This Line was added in version 1.4.0 of this specification. + +"mu" Int NL + +[Zero or one time.] + +The network stream bandwidth average calculated as explained in B4.2. + +This Line was added in version 1.7.0 of this specification. + +"muf" Int NL + +[Zero or one time.] + +The network stream bandwidth average filtered calculated as explained in +B4.2. + +This Line was added in version 1.7.0 of this specification. + +KeyValue NL + +[Zero or more times.] + +There MUST NOT be multiple KeyValue header Lines with the same key. +If there are, the parser SHOULD choose an arbitrary Line. + +If a parser does not recognize a Keyword in a KeyValue Line, it +MUST be ignored. + +Future format versions may include additional KeyValue header Lines. +Additional header Lines will be accompanied by a minor version +increment. + +Implementations MAY add additional header Lines as needed. This +specification SHOULD be updated to avoid conflicting meanings for +the same header keys. + +Parsers MUST NOT rely on the order of these additional Lines. + +Additional header Lines MUST NOT use any keywords specified in the +relay measurements format. +If there are, the parser MAY ignore conflicting keywords. + +Terminator NL + +[Zero or one time.] + +The Header List section ends with a Terminator. + +In version 1.0.0, Header List ends when the first relay bandwidth +is found conforming to the next section. + +Implementations of version 1.1.0 and later SHOULD use a 5-character +terminator. + +Tor 0.4.0.1-alpha and later look for a 5-character terminator, +or the first relay bandwidth line. sbws versions 0.1.0 to 1.0.2 +used a 4-character terminator, this bug was fixed in 1.0.3. + diff --git a/spec/bandwidth-file-spec/implementation-details.md b/spec/bandwidth-file-spec/implementation-details.md new file mode 100644 index 0000000..dc9f8c0 --- /dev/null +++ b/spec/bandwidth-file-spec/implementation-details.md @@ -0,0 +1,394 @@ + +## Implementation details + + +### Writing bandwidth files atomically + +To avoid inconsistent reads, implementations SHOULD write bandwidth files +atomically. If the file is transferred from another host, it SHOULD be +written to a temporary path, then renamed to the V3BandwidthsFile path. + +sbws versions 0.7.0 and later write the bandwidth file to an archival +location, create a temporary symlink to that location, then atomically rename +the symlink +to the configured V3BandwidthsFile path. + +Torflow does not write bandwidth files atomically. + + +### Additional KeyValue pair definitions + +KeyValue pairs in RelayLines that current implementations generate. + + +#### Simple Bandwidth Scanner + +sbws RelayLines contain these keys: + +"node_id" hexdigest + +As above. + +"bw" Bandwidth + +As above. + +"nick" nickname + +[Exactly once.] + +The relay nickname. + +Torflow also has a "nick" KeyValue. + +"rtt" Int + +[Zero or one time.] + +The Round Trip Time in milliseconds to obtain 1 byte of data. + +This KeyValue was added in version 1.1.0 of this specification. +It became optional in version 1.3.0 or 1.4.0 of this specification. + +"time" DateTime + +[Exactly once.] + +The date and time timestamp in ISO 8601 format and UTC time zone +when the last bandwidth was obtained. + +This KeyValue was added in version 1.1.0 of this specification. +The Torflow equivalent is "measured_at". + +"success" Int + +[Zero or one time.] + +The number of times that the bandwidth measurements for this relay were +successful. + +This KeyValue was added in version 1.1.0 of this specification. + +"error_circ" Int + +[Zero or one time.] + +The number of times that the bandwidth measurements for this relay +failed because of circuit failures. + +This KeyValue was added in version 1.1.0 of this specification. +The Torflow equivalent is "circ_fail". + +"error_stream" Int + +[Zero or one time.] + +The number of times that the bandwidth measurements for this relay +failed because of stream failures. + +This KeyValue was added in version 1.1.0 of this specification. + +"error_destination" Int + +[Zero or one time.] + +The number of times that the bandwidth measurements for this relay +failed because the destination Web server was not available. + +This KeyValue was added in version 1.4.0 of this specification. + +"error_second_relay" Int + +[Zero or one time.] + +The number of times that the bandwidth measurements for this relay +failed because sbws could not find a second relay for the test circuit. + +This KeyValue was added in version 1.4.0 of this specification. + +"error_misc" Int + +[Zero or one time.] + +The number of times that the bandwidth measurements for this relay +failed because of other reasons. + +This KeyValue was added in version 1.1.0 of this specification. + +"bw_mean" Int + +[Zero or one time.] + +The measured bandwidth mean for this relay in bytes per second. + +This KeyValue was added in version 1.2.0 of this specification. + +"bw_median" Int + +[Zero or one time.] + +The measured bandwidth median for this relay in bytes per second. + +This KeyValue was added in version 1.2.0 of this specification. + +"desc_bw_avg" Int + +[Zero or one time.] + +The descriptor average bandwidth for this relay in bytes per second. + +This KeyValue was added in version 1.2.0 of this specification. + +"desc_bw_obs_last" Int + +[Zero or one time.] + +The last descriptor observed bandwidth for this relay in bytes per +second. + +This KeyValue was added in version 1.2.0 of this specification. + +"desc_bw_obs_mean" Int + +[Zero or one time.] + +The descriptor observed bandwidth mean for this relay in bytes per +second. + +This KeyValue was added in version 1.2.0 of this specification. + +"desc_bw_bur" Int + +[Zero or one time.] + +The descriptor burst bandwidth for this relay in bytes per +second. + +This KeyValue was added in version 1.2.0 of this specification. + +"consensus_bandwidth" Int + +[Zero or one time.] + +The consensus bandwidth for this relay in bytes per second. + +This KeyValue was added in version 1.2.0 of this specification. + +"consensus_bandwidth_is_unmeasured" Bool + +[Zero or one time.] + +If the consensus bandwidth for this relay was not obtained from +three or more bandwidth authorities, this KeyValue is True or +False otherwise. + +This KeyValue was added in version 1.2.0 of this specification. + +"relay_in_recent_consensus_count" Int + +[Zero or one time.] + +The number of times this relay was found in a consensus in the +last data_period days. (Unless otherwise stated, data_period is +5 by default.) + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_priority_list_count" Int + +[Zero or one time.] + +The number of times this relay has been prioritized to be measured +in the last data_period days. + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_measurement_attempt_count" Int + +[Zero or one time.] + +The number of times this relay was tried to be measured in the +last data_period days. + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_measurement_failure_count" Int + +[Zero or one time.] + +The number of times this relay was tried to be measured in the +last data_period days, but it was not possible to obtain a +measurement. + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_measurements_excluded_error_count" Int + +[Zero or one time.] + +The number of recent relay measurement attempts that failed. +Measurements are recent if they are in the last data_period days +(5 by default). + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_measurements_excluded_near_count" Int + +[Zero or one time.] + +When all of a relay's recent successful measurements were performed in +a period of time that was too short (by default 1 day), the relay is +excluded. This KeyValue contains the number of recent successful +measurements for the relay that were ignored for this reason. + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_measurements_excluded_old_count" Int + +[Zero or one time.] + +The number of successful measurements for this relay that are too old +(more than data_period days, 5 by default). + +Excludes measurements that are already counted in +relay_recent_measurements_excluded_near_count. + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This KeyValue was added in version 1.4.0 of this specification. + +"relay_recent_measurements_excluded_few_count" Int + +[Zero or one time.] + +The number of successful measurements for this relay that were ignored +because the relay did not have enough successful measurements (fewer +than 2, by default). + +Excludes measurements that are already counted in +relay_recent_measurements_excluded_near_count or +relay_recent_measurements_excluded_old_count. + +(See the note in section 1.4, version 1.4.0, about excluded relays.) + +This KeyValue was added in version 1.4.0 of this specification. + +"under_min_report" bool + +[Zero or one time.] + +If the value is 1, there are not enough eligible relays in the +bandwidth file, and Tor bandwidth authorities MAY NOT vote on this +relay. (Current Tor versions do not change their behaviour based on +the "under_min_report" key.) + +If the value is 0 or the KeyValue is not present, there are enough +relays in the bandwidth file. + +Because Tor versions released before April 2019 (see section 1.4. for +the full list of versions) ignore "vote=0", generator implementations +MUST NOT change the bandwidths for under_min_report relays. Using the +same bw value makes authorities that do not understand "vote=0" +or "under_min_report=1" produce votes that don't change relay weights +too much. It also avoids flapping when the reporting threshold is +reached. + +This KeyValue was added in version 1.4.0 of this specification. + +"unmeasured" bool + +[Zero or one time.] + +If the value is 1, this relay was not successfully measured and +Tor bandwidth authorities MAY NOT vote on this relay. +(Current Tor versions do not change their behaviour based on +the "unmeasured" key.) + +If the value is 0 or the KeyValue is not present, this relay +was successfully measured. + +Because Tor versions released before April 2019 (see section 1.4. for +the full list of versions) ignore "vote=0", generator implementations +MUST set "bw=1" for unmeasured relays. Using the minimum bw value +makes authorities that do not understand "vote=0" or "unmeasured=1" +produce votes that don't change relay weights too much. + +This KeyValue was added in version 1.4.0 of this specification. + +"vote" bool + +[Zero or one time.] + +If the value is 0, Tor directory authorities SHOULD ignore the relay's +entry in the bandwidth file. They SHOULD vote for the relay the same +way they would vote for a relay that is not present in the file. + +This MAY be the case when this relay was not successfully measured but +it is included in the Bandwidth File, to diagnose why they were not +measured. + +If the value is 1 or the KeyValue is not present, Tor directory +authorities MUST use the relay's bw value in any votes for that relay. + +Implementations MUST also set "bw=1" for unmeasured relays. +But they MUST NOT change the bw for under_min_report relays. +(See the explanations under "unmeasured" and "under_min_report" +for more details.) + +This KeyValue was added in version 1.4.0 of this specification. + +"xoff_recv" Int + +[Zero or one time.] + +The number of times this relay received `XOFF_RECV` stream events while +being measured in the last data_period days. + +This KeyValue was added in version 1.6.0 of this specification. + +"xoff_sent" Int + +[Zero or one time.] + +The number of times this relay received `XOFF_SENT` stream events while +being measured in the last data_period days. + +This KeyValue was added in version 1.6.0 of this specification. + +"r_strm" Float + +[Zero or one time.] + +The stream ratio of this relay calculated as explained in B4.3. + +This KeyValue was added in version 1.7.0 of this specification. + +"r_strm_filt" Float + +[Zero or one time.] + +The filtered stream ratio of this relay calculated as explained in B4.3. + +This KeyValue was added in version 1.7.0 of this specification. + + +#### Torflow + +Torflow RelayLines include node_id and bw, and other KeyValue pairs [2]. + +References: + +```text +1. https://gitweb.torproject.org/torflow.git +2. https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/README.spec.txt#n332 + The Torflow specification is outdated, and does not match the current + implementation. See section A.1. for the format produced by Torflow. +3. https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt +4. https://gitweb.torproject.org/torspec.git/tree/version-spec.txt +5. https://semver.org/ +``` + diff --git a/spec/bandwidth-file-spec/relay-line-format.md b/spec/bandwidth-file-spec/relay-line-format.md new file mode 100644 index 0000000..9e1d1b4 --- /dev/null +++ b/spec/bandwidth-file-spec/relay-line-format.md @@ -0,0 +1,126 @@ + +## Relay Line format + +It consists of zero or more RelayLines containing relay ids and +bandwidths. The relays and their KeyValues are in arbitrary order. + +There MUST NOT be multiple KeyValue pairs with the same key in the same +RelayLine. If there are, the parser SHOULD choose an arbitrary Value. + +There MUST NOT be multiple RelayLines per relay identity (node_id or +master_key_ed25519). If there are, parsers SHOULD issue a warning. +Parers MAY reject the file, choose an arbitrary RelayLine, or ignore +both RelayLines. + +If a parser does not recognize any extra material in a RelayLine, +the extra material MUST be ignored. + +Each RelayLine includes the following KeyValue pairs: + +"node_id" hexdigest + +[Exactly once.] + +The fingerprint for the relay's RSA identity key. + +```text + Note: In bandwidth files read by Tor versions earlier than + 0.3.4.1-alpha, node_id MUST NOT be at the end of the Line. + These authority versions are no longer supported. +``` + +Current Tor versions ignore master_key_ed25519, so node_id MUST be +present in each relay Line. + +Implementations of version 1.1.0 and later SHOULD include both node_id +and master_key_ed25519. Parsers SHOULD accept Lines that contain at +least one of them. + +"master_key_ed25519" MasterKey + +[Zero or one time.] + +The relays's master Ed25519 key, base64 encoded, +without trailing "="s, to avoid ambiguity with KeyValue "=" +character. + +This KeyValue pair SHOULD be present, see the note under node_id. + +This KeyValue was added in version 1.1.0 of this specification. + +"bw" Bandwidth + +[Exactly once.] + +The bandwidth of this relay in kilobytes per second. + +No Zero Bandwidths: +Tor accepts zero bandwidths, but they trigger bugs in older Tor +implementations. Therefore, implementations SHOULD NOT produce zero +bandwidths. Instead, they SHOULD use one as their minimum bandwidth. +If there are zero bandwidths, the parser MAY ignore them. + +Bandwidth Aggregation: +Multiple measurements can be aggregated using an averaging scheme, +such as a mean, median, or decaying average. + +Bandwidth Scaling: +Torflow scales bandwidths to kilobytes per second. Other +implementations SHOULD use kilobytes per second for their initial +bandwidth scaling. + +If different implementations or configurations are used in votes for +the same network, their measurements MAY need further scaling. See +Appendix B for information about scaling, and one possible scaling +method. + +MaxAdvertisedBandwidth: +Bandwidth generators MUST limit the relays' measured bandwidth based +on the MaxAdvertisedBadwidth. +A relay's MaxAdvertisedBandwidth limits the bandwidth-avg in its +descriptor. bandwidth-avg is the minimum of MaxAdvertisedBandwidth, +BandwidthRate, RelayBandwidthRate, BandwidthBurst, and +RelayBandwidthBurst. +Therefore, generators MUST limit a relay's measured bandwidth to its +descriptor's bandwidth-avg. This limit needs to be implemented in the +generator, because generators may scale consensus weights before +sending them to Tor. +Generators SHOULD NOT limit measured bandwidths based on descriptors' +bandwidth-observed, because that penalises new relays. + +sbws limits the relay's measured bandwidth to the bandwidth-avg +advertised. + +Torflow partitions relays based on their bandwidth. For unmeasured +relays, Torflow uses the minimum of all descriptor bandwidths, +including bandwidth-avg (MaxAdvertisedBandwidth) and +bandwidth-observed. Then Torflow measures the relays in each partition +against each other, which implicitly limits a relay's measured +bandwidth to the bandwidths of similar relays. + +Torflow also generates consensus weights based on the ratio between the +measured bandwidth and the minimum of all descriptor bandwidths (at the +time of the measurement). So when an operator reduces the +MaxAdvertisedBandwidth for a relay, Torflow reduces that relay's +measured bandwidth. + +KeyValue + +[Zero or more times.] + +Future format versions may include additional KeyValue pairs on a +RelayLine. +Additional KeyValue pairs will be accompanied by a minor version +increment. + +Implementations MAY add additional relay KeyValue pairs as needed. +This specification SHOULD be updated to avoid conflicting meanings +for the same Keywords. + +Parsers MUST NOT rely on the order of these additional KeyValue +pairs. + +Additional KeyValue pairs MUST NOT use any keywords specified in the +header format. +If there are, the parser MAY ignore conflicting keywords. + diff --git a/spec/bandwidth-file-spec/sample-data.md b/spec/bandwidth-file-spec/sample-data.md new file mode 100644 index 0000000..1c21f89 --- /dev/null +++ b/spec/bandwidth-file-spec/sample-data.md @@ -0,0 +1,118 @@ + +# Sample data + +The following has not been obtained from any real measurement. + + +## Generated by Torflow + +This an example version 1.0.0 document: + +1523911758 +node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test measured_at=1523911725 updated_at=1523911725 pid_error=4.11374090719 pid_error_sum=4.11374090719 pid_bw=57136645 pid_delta=2.12168374577 circ_fail=0.2 scanner=/filepath +node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 measured_at=1523911623 updated_at=1523911623 pid_error=3.96703337994 pid_error_sum=3.96703337994 pid_bw=47422125 pid_delta=2.65469736988 circ_fail=0.0 scanner=/filepath + + +## Generated by sbws version 0.1.0 + +1523911758 +version=1.1.0 +software=sbws +software_version=0.1.0 +latest_bandwidth=2018-04-16T20:49:18 +file_created=2018-04-16T21:49:18 +generator_started=2018-04-16T15:13:25 +earliest_bandwidth=2018-04-16T15:13:26 +==== +bw=380 error_circ=0 error_misc=0 error_stream=1 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ nick=Test node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 rtt=380 success=1 time=2018-05-08T16:13:26 +bw=189 error_circ=0 error_misc=0 error_stream=0 master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I nick=Test2 node_id=$96C15995F30895689291F455587BD94CA427B6FC rtt=378 success=1 time=2018-05-08T16:13:36 + + +## Generated by sbws version 1.0.3 + +1523911758 +version=1.2.0 +latest_bandwidth=2018-04-16T20:49:18 +file_created=2018-04-16T21:49:18 +generator_started=2018-04-16T15:13:25 +earliest_bandwidth=2018-04-16T15:13:26 +minimum_number_eligible_relays=3862 +minimum_percent_eligible_relays=60 +number_consensus_relays=6436 +number_eligible_relays=6000 +percent_eligible_relays=93 +software=sbws +software_version=1.0.3 +===== +bw=38000 bw_mean=1127824 bw_median=1180062 desc_bw_avg=1073741824 desc_bw_obs_last=17230879 desc_bw_obs_mean=14732306 error_circ=0 error_misc=0 error_stream=1 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ nick=Test node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 rtt=380 success=1 time=2018-05-08T16:13:26 +bw=1 bw_mean=199162 bw_median=185675 desc_bw_avg=409600 desc_bw_obs_last=836165 desc_bw_obs_mean=858030 error_circ=0 error_misc=0 error_stream=0 master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I nick=Test2 node_id=$96C15995F30895689291F455587BD94CA427B6FC rtt=378 success=1 time=2018-05-08T16:13:36 + + +### When there are not enough eligible measured relays: + +1540496079 +version=1.2.0 +earliest_bandwidth=2018-10-20T19:35:52 +file_created=2018-10-25T19:35:03 +generator_started=2018-10-25T11:42:56 +latest_bandwidth=2018-10-25T19:34:39 +minimum_number_eligible_relays=3862 +minimum_percent_eligible_relays=60 +number_consensus_relays=6436 +number_eligible_relays=2960 +percent_eligible_relays=46 +software=sbws +software_version=1.0.3 +===== + + +## Headers generated by sbws version 1.0.4 + +1523911758 +version=1.2.0 +latest_bandwidth=2018-04-16T20:49:18 +destinations_countries=TH,ZZ +file_created=2018-04-16T21:49:18 +generator_started=2018-04-16T15:13:25 +earliest_bandwidth=2018-04-16T15:13:26 +minimum_number_eligible_relays=3862 +minimum_percent_eligible_relays=60 +number_consensus_relays=6436 +number_eligible_relays=6000 +percent_eligible_relays=93 +scanner_country=SN +software=sbws +software_version=1.0.4 +===== + + +## Generated by sbws version 1.1.0 + +1523911758 +version=1.4.0 +latest_bandwidth=2018-04-16T20:49:18 +destinations_countries=TH,ZZ +file_created=2018-04-16T21:49:18 +generator_started=2018-04-16T15:13:25 +earliest_bandwidth=2018-04-16T15:13:26 +minimum_number_eligible_relays=3862 +minimum_percent_eligible_relays=60 +number_consensus_relays=6436 +number_eligible_relays=6000 +percent_eligible_relays=93 +recent_measurement_attempt_count=6243 +recent_measurement_failure_count=732 +recent_measurements_excluded_error_count=969 +recent_measurements_excluded_few_count=3946 +recent_measurements_excluded_near_count=90 +recent_measurements_excluded_old_count=0 +recent_priority_list_count=20 +recent_priority_relay_count=6243 +scanner_country=SN +software=sbws +software_version=1.1.0 +time_to_report_half_network=57273 +===== +bw=1 error_circ=1 error_destination=0 error_misc=0 error_second_relay=0 error_stream=0 master_key_ed25519=J3HQ24kOQWac3L1xlFLp7gY91qkb5NuKxjj1BhDi+m8 nick=snap269 node_id=$DC4D609F95A52614D1E69C752168AF1FCAE0B05F relay_recent_measurement_attempt_count=3 relay_recent_measurements_excluded_error_count=1 relay_recent_measurements_excluded_near_count=3 relay_recent_consensus_count=3 relay_recent_priority_list_count=3 success=3 time=2019-03-16T18:20:57 unmeasured=1 vote=0 +bw=1 error_circ=0 error_destination=0 error_misc=0 error_second_relay=0 error_stream=2 master_key_ed25519=h6ZB1E1yBFWIMloUm9IWwjgaPXEpL5cUbuoQDgdSDKg nick=relay node_id=$C4544F9E209A9A9B99591D548B3E2822236C0503 relay_recent_measurement_attempt_count=3 relay_recent_measurements_excluded_error_count=2 relay_recent_measurements_excluded_few_count=1 relay_recent_consensus_count=3 relay_recent_priority_list_count=3 success=1 time=2019-03-17T06:50:58 unmeasured=1 vote=0 + diff --git a/spec/bandwidth-file-spec/scaling-bandwidths.md b/spec/bandwidth-file-spec/scaling-bandwidths.md new file mode 100644 index 0000000..8ae80c9 --- /dev/null +++ b/spec/bandwidth-file-spec/scaling-bandwidths.md @@ -0,0 +1,128 @@ + +# Scaling bandwidths + + +## Scaling requirements + +```text + Tor accepts zero bandwidths, but they trigger bugs in older Tor + implementations. Therefore, scaling methods SHOULD perform the + following checks: + * If the total bandwidth is zero, all relays should be given equal + bandwidths. + * If the scaled bandwidth is zero, it should be rounded up to one. +``` + +Initial experiments indicate that scaling may not be needed for +torflow and sbws, because their measured bandwidths are similar +enough already. + + +## A linear scaling method + +If scaling is required, here is a simple linear bandwidth scaling +method, which ensures that all bandwidth votes contain approximately +the same total bandwidth: + +```text + 1. Calculate the relay quota by dividing the total measured bandwidth + in all votes, by the number of relays with measured bandwidth + votes. In the public tor network, this is approximately 7500 as of + April 2018. The quota should be a consensus parameter, so it can be + adjusted for all generators on the network. + + 2. Calculate a vote quota by multiplying the relay quota by the number + of relays this bandwidth authority has measured + bandwidths for. + + 3. Calculate a scaling factor by dividing the vote quota by the + total unscaled measured bandwidth in this bandwidth + authority's upcoming vote. + + 4. Multiply each unscaled measured bandwidth by the scaling + factor. +``` + +Now, the total scaled bandwidth in the upcoming vote is +approximately equal to the quota. + + +## Quota changes + +If all generators are using scaling, the quota can be gradually +reduced or increased as needed. Smaller quotas decrease the size +of uncompressed consensuses, and may decrease the size of +consensus diffs and compressed consensuses. But if the relay +quota is too small, some relays may be over- or under-weighted. + + +## Torflow aggregation + +Torflow implements two methods to compute the bandwidth values from the +(stream) bandwidth measurements: with and without PID control feedback. +The method described here is without PID control (see Torflow +specification, section 2.2). + +In the following sections, the relays' measured bandwidth refer to the +ones that this bandwidth authority has measured for the relays that +would be included in the next bandwidth authority's upcoming vote. + +```text + 1. Calculate the filtered bandwidth for each relay: + - choose the relay's measurements (`bw_j`) that are equal or greater + than the mean of the measurements for this relay + - calculate the mean of those measurements + + In pseudocode: + + bw_filt_i = mean(max(mean(bw_j), bw_j)) + + 2. Calculate network averages: + - calculate the filtered average by dividing the sum of all the + relays' filtered bandwidth by the number of relays that have been + measured (`n`), ie, calculate the mean average of the relays' + filtered bandwidth. + - calculate the stream average by dividing the sum of all the + relays' measured bandwidth by the number of relays that have been + measured (`n`), ie, calculate the mean average or the relays' + measured bandwidth. + + In pseudocode: + + bw_avg_filt_ = bw_filt_i / n + bw_avg_strm = bw_i / n + + 3. Calculate ratios for each relay: + - calculate the filtered ratio by dividing each relay filtered + bandwidth by the filtered average + - calculate the stream ratio by dividing each relay measured + bandwidth by the stream average + + In pseudocode: +``` + +r_filt_i = bw_filt_i / bw_avg_filt +r_strm_i = bw_i / bw_avg_strm + +```text + 4. Calculate the final ratio for each relay: + The final ratio is the larger between the filtered bandwidth's and the + stream bandwidth's ratio. + + In pseudocode: + + r_i = max(r_filt_i, r_strm_i) + + 5. Calculate the scaled bandwidth for each relay: + The most recent descriptor observed bandwidth (`bw_obs_i`) is + multiplied by the ratio + + In pseudocode: + + bw_new_i = r_i * bw_obs_i +``` + +<> + diff --git a/spec/bandwidth-file-spec/scope-preliminaries.md b/spec/bandwidth-file-spec/scope-preliminaries.md new file mode 100644 index 0000000..969c971 --- /dev/null +++ b/spec/bandwidth-file-spec/scope-preliminaries.md @@ -0,0 +1,92 @@ + +# Scope and preliminaries + +This document describes the format of Tor's Bandwidth File, version +1.0.0 and later. + +It is a new specification for the existing bandwidth file format, +which we call version 1.0.0. It also specifies new format versions +1.1.0 and later, which are backwards compatible with 1.0.0 parsers. + +Since Tor version 0.2.4.12-alpha, the directory authorities use +the Bandwidth File file called "V3BandwidthsFile" generated by +Torflow [1]. The details of this format are described in Torflow's +README.spec.txt. We also summarise the format in this specification. + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL +NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in +RFC 2119. + + +## Acknowledgements + +The original bandwidth generator (Torflow) and format was +created by mike. Teor suggested to write this specification while +contributing on pastly's new bandwidth generator implementation. + +This specification was revised after feedback from: + +Nick Mathewson (nickm) +Iain Learmonth (irl) + + +## Outline + +The Tor directory protocol (dir-spec.txt [3]) sections 3.4.1 +and 3.4.2, use the term bandwidth measurements, to refer to what +here is called Bandwidth File. + +A Bandwidth File contains information on relays' bandwidth +capacities and is produced by bandwidth generators, previously known +as bandwidth scanners. + + +## Format Versions + +1.0.0 - The legacy Bandwidth File format + +```text + 1.1.0 - Adds a header containing information about the bandwidth + file. Document the sbws and Torflow relay line keys. + + 1.2.0 - If there are not enough eligible relays, the bandwidth file + SHOULD contain a header, but no relays. (To match Torflow's + existing behaviour.) + + Adds scanner and destination countries to the header. + Adds new KeyValue Lines to the Header List section with + statistics about the number of relays included in the file. + Adds new KeyValues to Relay Bandwidth Lines, with different + bandwidth values (averages and descriptor bandwidths). + + 1.4.0 - Adds monitoring KeyValues to the header and relay lines. + + RelayLines for excluded relays MAY be present in the bandwidth + file for diagnostic reasons. Similarly, if there are not enough + eligible relays, the bandwidth file MAY contain all known relays. + + Diagnostic relay lines SHOULD be marked with vote=0, and + Tor SHOULD NOT use their bandwidths in its votes. + + Also adds Tor version. + 1.5.0 - Removes "recent_measurement_attempt_count" KeyValue. + 1.6.0 - Adds congestion control stream events KeyValues. + 1.7.0 - Adds ratios KeyValues to the relay lines and network averages + KeyValues to the header. + + All Tor versions can consume format version 1.0.0. +``` + +All Tor versions can consume format version 1.1.0 and later, +but Tor versions earlier than 0.3.5.1-alpha warn if the header +contains any KeyValue lines after the Timestamp. + +```text + Tor versions 0.4.0.3-alpha, 0.3.5.8, 0.3.4.11, and earlier do not + understand "vote=0". Instead, they will vote for the actual bandwidths + that sbws puts in diagnostic relay lines: + * 1 for relays with "unmeasured=1", and + * the relay's measured and scaled bandwidth when "under_min_report=1". +``` + diff --git a/spec/bridgedb-spec.md b/spec/bridgedb-spec.md new file mode 100644 index 0000000..fc95399 --- /dev/null +++ b/spec/bridgedb-spec.md @@ -0,0 +1,440 @@ +BridgeDB specification + +```text + Karsten Loesing + Nick Mathewson + +Table of Contents + + 0. Preliminaries + 1. Importing bridge network statuses and bridge descriptors + 1.1. Parsing bridge network statuses + 1.2. Parsing bridge descriptors + 1.3. Parsing extra-info documents + 2. Assigning bridges to distributors + 3. Giving out bridges upon requests + 4. Selecting bridges to be given out based on IP addresses + 5. Selecting bridges to be given out based on email addresses + 6. Selecting unallocated bridges to be stored in file buckets + 7. Displaying Bridge Information + 8. Writing bridge assignments for statistics +``` + + +# Preliminaries + +This document specifies how BridgeDB processes bridge descriptor files +to learn about new bridges, maintains persistent assignments of bridges +to distributors, and decides which bridges to give out upon user +requests. + +Some of the decisions here may be suboptimal: this document is meant to +specify current behavior as of August 2013, not to specify ideal +behavior. + + +# Importing bridge network statuses and bridge descriptors + +BridgeDB learns about bridges by parsing bridge network statuses, +bridge descriptors, and extra info documents as specified in Tor's +directory protocol. BridgeDB parses one bridge network status file +first and at least one bridge descriptor file and potentially one extra +info file afterwards. + +BridgeDB scans its files on sighup. + +BridgeDB does not validate signatures on descriptors or networkstatus +files: the operator needs to make sure that these documents have come +from a Tor instance that did the validation for us. + + +## Parsing bridge network statuses + +Bridge network status documents contain the information of which bridges +are known to the bridge authority and which flags the bridge authority +assigns to them. +We expect bridge network statuses to contain at least the following two +lines for every bridge in the given order (format fully specified in Tor's +directory protocol): + +```text + "r" SP nickname SP identity SP digest SP publication SP IP SP ORPort + SP DirPort NL + "a" SP address ":" port NL (no more than 8 instances) + "s" SP Flags NL +``` + +BridgeDB parses the identity and the publication timestamp from the "r" +line, the OR address(es) and ORPort(s) from the "a" line(s), and the +assigned flags from the "s" line, specifically checking the assignment +of the "Running" and "Stable" flags. +BridgeDB memorizes all bridges that have the Running flag as the set of +running bridges that can be given out to bridge users. +BridgeDB memorizes assigned flags if it wants to ensure that sets of +bridges given out should contain at least a given number of bridges +with these flags. + + +## Parsing bridge descriptors + +BridgeDB learns about a bridge's most recent IP address and OR port +from parsing bridge descriptors. +In theory, both IP address and OR port of a bridge are also contained +in the "r" line of the bridge network status, so there is no mandatory +reason for parsing bridge descriptors. But the functionality described +in this section is still implemented in case we need data from the +bridge descriptor in the future. + +Bridge descriptor files may contain one or more bridge descriptors. +We expect a bridge descriptor to contain at least the following lines in +the stated order: + +```text + "@purpose" SP purpose NL + "router" SP nickname SP IP SP ORPort SP SOCKSPort SP DirPort NL + "published" SP timestamp + ["opt" SP] "fingerprint" SP fingerprint NL + "router-signature" NL Signature NL +``` + +BridgeDB parses the purpose, IP, ORPort, nickname, and fingerprint +from these lines. +BridgeDB skips bridge descriptors if the fingerprint is not contained +in the bridge network status parsed earlier or if the bridge does not +have the Running flag. +BridgeDB discards bridge descriptors which have a different purpose +than "bridge". BridgeDB can be configured to only accept descriptors +with another purpose or not discard descriptors based on purpose at +all. +BridgeDB memorizes the IP addresses and OR ports of the remaining +bridges. +If there is more than one bridge descriptor with the same fingerprint, +BridgeDB memorizes the IP address and OR port of the most recently +parsed bridge descriptor. +If BridgeDB does not find a bridge descriptor for a bridge contained in +the bridge network status parsed before, it does not add that bridge +to the set of bridges to be given out to bridge users. + + +## Parsing extra-info documents + +BridgeDB learns if a bridge supports a pluggable transport by parsing +extra-info documents. +Extra-info documents contain the name of the bridge (but only if it is +named), the bridge's fingerprint, the type of pluggable transport(s) it +supports, and the IP address and port number on which each transport +listens, respectively. + +Extra-info documents may contain zero or more entries per bridge. We expect +an extra-info entry to contain the following lines in the stated order: + +```text + "extra-info" SP name SP fingerprint NL + "transport" SP transport SP IP ":" PORT ARGS NL +``` + +BridgeDB parses the fingerprint, transport type, IP address, port and any +arguments that are specified on these lines. BridgeDB skips the name. If +the fingerprint is invalid, BridgeDB skips the entry. BridgeDB memorizes +the transport type, IP address, port number, and any arguments that are be +provided and then it assigns them to the corresponding bridge based on the +fingerprint. Arguments are comma-separated and are of the form k=v,k=v. +Bridges that do not have an associated extra-info entry are not invalid. + + +# Assigning bridges to distributors + +A "distributor" is a mechanism by which bridges are given (or not +given) to clients. The current distributors are "email", "https", +and "unallocated". + +BridgeDB assigns bridges to distributors based on an HMAC hash of the +bridge's ID and a secret and makes these assignments persistent. +Persistence is achieved by using a database to map node ID to +distributor. +Each bridge is assigned to exactly one distributor (including +the "unallocated" distributor). +BridgeDB may be configured to support only a non-empty subset of the +distributors specified in this document. +BridgeDB may be configured to use different probabilities for assigning +new bridges to distributors. +BridgeDB does not change existing assignments of bridges to +distributors, even if probabilities for assigning bridges to +distributors change or distributors are disabled entirely. + + +# Giving out bridges upon requests + +Upon receiving a client request, a BridgeDB distributor provides a +subset of the bridges assigned to it. +BridgeDB only gives out bridges that are contained in the most recently +parsed bridge network status and that have the Running flag set (see +Section 1). +BridgeDB may be configured to give out a different number of bridges +(typically 4) depending on the distributor. +BridgeDB may define an arbitrary number of rules. These rules may +specify the criteria by which a bridge is selected. Specifically, +the available rules restrict the IP address version, OR port number, +transport type, bridge relay flag, or country in which the bridge +should not be blocked. + + +# Selecting bridges to be given out based on IP addresses + +```text + BridgeDB may be configured to support one or more distributors which + gives out bridges based on the requestor's IP address. Currently, this + is how the HTTPS distributor works. + The goal is to avoid handing out all the bridges to users in a similar + IP space and time. +# Someone else should look at proposals/ideas/old/xxx-bridge-disbursement +# to see if this section is missing relevant pieces from it. -KL + + BridgeDB fixes the set of bridges to be returned for a defined time + period. + BridgeDB considers all IP addresses coming from the same /24 network + as the same IP address and returns the same set of bridges. From here on, + this non-unique address will be referred to as the IP address's 'area'. + BridgeDB divides the IP address space equally into a small number of +# Note, changed term from "areas" to "disjoint clusters" -MF + disjoint clusters (typically 4) and returns different results for requests + coming from addresses that are placed into different clusters. +# I found that BridgeDB is not strict in returning only bridges for a +# given area. If a ring is empty, it considers the next one. Is this +# expected behavior? -KL +# +# This does not appear to be the case, anymore. If a ring is empty, then +# BridgeDB simply returns an empty set of bridges. -MF +# +# I also found that BridgeDB does not make the assignment to areas +# persistent in the database. So, if we change the number of rings, it +# will assign bridges to other rings. I assume this is okay? -KL + BridgeDB maintains a list of proxy IP addresses and returns the same + set of bridges to requests coming from these IP addresses. + The bridges returned to proxy IP addresses do not come from the same + set as those for the general IP address space. +``` + +BridgeDB can be configured to include bridge fingerprints in replies +along with bridge IP addresses and OR ports. +BridgeDB can be configured to display a CAPTCHA which the user must solve +prior to returning the requested bridges. + +The current algorithm is as follows. An IP-based distributor splits +the bridges uniformly into a set of "rings" based on an HMAC of their +ID. Some of these rings are "area" rings for parts of IP space; some +are "category" rings for categories of IPs (like proxies). When a +client makes a request from an IP, the distributor first sees whether +the IP is in one of the categories it knows. If so, the distributor +returns an IP from the category rings. If not, the distributor +maps the IP into an "area" (that is, a /24), and then uses an HMAC to +map the area to one of the area rings. + +When the IP-based distributor determines from which area ring it is handing +out bridges, it identifies which rules it will use to choose appropriate +bridges. Using this information, it searches its cache of rings for one +that already adheres to the criteria specified in this request. If one +exists, then BridgeDB maps the current "epoch" (N-hour period) and the +IP's area (/24) to a point on the ring based on HMAC, and hands out +bridges at that point. If a ring does not already exist which satisfies this +request, then a new ring is created and filled with bridges that fulfill +the requirements. This ring is then used to select bridges as described. + +"Mapping X to Y based on an HMAC" above means one of the following: + +```text + - We keep all of the elements of Y in some order, with a mapping + from all 160-bit strings to positions in Y. + - We take an HMAC of X using some fixed string as a key to get a + 160-bit value. We then map that value to the next position of Y. +``` + +When giving out bridges based on a position in a ring, BridgeDB first +looks at flag requirements and port requirements. For example, +BridgeDB may be configured to "Give out at least L bridges with port +443, and at least M bridges with Stable, and at most N bridges +total." To do this, BridgeDB combines to the results: + +```text + - The first L bridges in the ring after the position that have the + port 443, and + - The first M bridges in the ring after the position that have the + flag stable and that it has not already decided to give out, and + - The first N-L-M bridges in the ring after the position that it + has not already decided to give out. + + After BridgeDB selects appropriate bridges to return to the requestor, it + then prioritises the ordering of them in a list so that as many criteria + are fulfilled as possible within the first few bridges. This list is then + truncated to N bridges, if possible. N is currently defined as a + piecewise function of the number of bridges in the ring such that: + + / + | 1, if len(ring) < 20 + | + N = | 2, if 20 <= len(ring) <= 100 + | + | 3, if 100 <= len(ring) + \ + + The bridges in this sublist, containing no more than N bridges, are the + bridges returned to the requestor. +``` + + +# Selecting bridges to be given out based on email addresses + +```text + BridgeDB can be configured to support one or more distributors that are + giving out bridges based on the requestor's email address. Currently, + this is how the email distributor works. + The goal is to bootstrap based on one or more popular email service's + sybil prevention algorithms. +# Someone else should look at proposals/ideas/old/xxx-bridge-disbursement +# to see if this section is missing relevant pieces from it. -KL + + BridgeDB rejects email addresses containing other characters than the + ones that RFC2822 allows. + BridgeDB may be configured to reject email addresses containing other + characters it might not process correctly. +# I don't think we do this, is it worthwhile? -MF + BridgeDB rejects email addresses coming from other domains than a + configured set of permitted domains. + BridgeDB normalizes email addresses by removing "." characters and by + removing parts after the first "+" character. + BridgeDB can be configured to discard requests that do not have the + value "pass" in their X-DKIM-Authentication-Result header or does not + have this header. The X-DKIM-Authentication-Result header is set by + the incoming mail stack that needs to check DKIM authentication. + + BridgeDB does not return a new set of bridges to the same email address + until a given time period (typically a few hours) has passed. +# Why don't we fix the bridges we give out for a global 3-hour time period +# like we do for IP addresses? This way we could avoid storing email +# addresses. -KL +# The 3-hour value is probably much too short anyway. If we take longer +# time values, then people get new bridges when bridges show up, as +# opposed to then we decide to reset the bridges we give them. (Yes, this +# problem exists for the IP distributor). -NM +# I'm afraid I don't fully understand what you mean here. Can you +# elaborate? -KL +# +# Assuming an average churn rate, if we use short time periods, then a +# requestor will receive new bridges based on rate-limiting and will (likely) +# eventually work their way around the ring; eventually exhausting all bridges +# available to them from this distributor. If we use a longer time period, +# then each time the period expires there will be more bridges in the ring +# thus reducing the likelihood of all bridges being blocked and increasing +# the time and effort required to enumerate all bridges. (This is my +# understanding, not from Nick) -MF +# Also, we presently need the cache to prevent replays and because if a user +# sent multiple requests with different criteria in each then we would leak +# additional bridges otherwise. -MF + BridgeDB can be configured to include bridge fingerprints in replies + along with bridge IP addresses and OR ports. + BridgeDB can be configured to sign all replies using a PGP signing key. + BridgeDB periodically discards old email-address-to-bridge mappings. + BridgeDB rejects too frequent email requests coming from the same + normalized address. +``` + +To map previously unseen email addresses to a set of bridges, BridgeDB +proceeds as follows: + +```text + - It normalizes the email address as above, by stripping out dots, + removing all of the localpart after the +, and putting it all + in lowercase. (Example: "John.Doe+bridges@example.COM" becomes + "johndoe@example.com".) + - It maps an HMAC of the normalized address to a position on its ring + of bridges. + - It hands out bridges starting at that position, based on the + port/flag requirements, as specified at the end of section 4. + + See section 4 for the details of how bridges are selected from the ring + and returned to the requestor. +``` + + +# Selecting unallocated bridges to be stored in file buckets + +# Kaner should have a look at this section. -NM + +```text + BridgeDB can be configured to reserve a subset of bridges and not give + them out via one of the distributors. + BridgeDB assigns reserved bridges to one or more file buckets of fixed + sizes and write these file buckets to disk for manual distribution. + BridgeDB ensures that a file bucket always contains the requested + number of running bridges. + If the requested number of bridges in a file bucket is reduced or the + file bucket is not required anymore, the unassigned bridges are + returned to the reserved set of bridges. + If a bridge stops running, BridgeDB replaces it with another bridge + from the reserved set of bridges. +# I'm not sure if there's a design bug in file buckets. What happens if +# we add a bridge X to file bucket A, and X goes offline? We would add +# another bridge Y to file bucket A. OK, but what if A comes back? We +# cannot put it back in file bucket A, because it's full. Are we going to +# add it to a different file bucket? Doesn't that mean that most bridges +# will be contained in most file buckets over time? -KL +# +# This should be handled the same as if the file bucket is reduced in size. +# If X returns, then it should be added to the appropriate distributor. -MF +``` + + +# Displaying Bridge Information + +After bridges are selected using one of the methods described in +Sections 4 - 6, they are output in one of two formats. Bridges are +formatted as: + + NL + +Pluggable transports are formatted as: + + SP [SP arglist] NL + +where arglist is an optional space-separated list of key-value pairs in +the form of k=v. + +Previously, each line was prepended with the "bridge" keyword, such as + +"bridge" SP NL + +"bridge" SP SP [SP arglist] NL + +# We don't do this anymore because Vidalia and TorLauncher don't expect it. +# See the commit message for b70347a9c5fd769c6d5d0c0eb5171ace2999a736. + + +# Writing bridge assignments for statistics + +BridgeDB can be configured to write bridge assignments to disk for +statistical analysis. +The start of a bridge assignment is marked by the following line: + +"bridge-pool-assignment" SP YYYY-MM-DD HH:MM:SS NL + +YYYY-MM-DD HH:MM:SS is the time, in UTC, when BridgeDB has completed +loading new bridges and assigning them to distributors. + +For every running bridge there is a line with the following format: + +fingerprint SP distributor (SP key "=" value)* NL + +The distributor is one out of "email", "https", or "unallocated". + +Both "email" and "https" distributors support adding keys for "port", +"flag" and "transport". Respectively, the port number, flag name, and +transport types are the values. These are used to indicate that +a bridge matches certain port, flag, transport criteria of requests. + +The "https" distributor also allows the key "ring" with a number as +value to indicate to which IP address area the bridge is returned. + +The "unallocated" distributor allows the key "bucket" with the file +bucket name as value to indicate which file bucket a bridge is assigned +to. + diff --git a/spec/cert-spec.md b/spec/cert-spec.md new file mode 100644 index 0000000..36c9c0b --- /dev/null +++ b/spec/cert-spec.md @@ -0,0 +1,223 @@ +Ed25519 certificates in Tor + +Table of Contents + +```text + 1. Scope and Preliminaries + 1.1. Signing + 1.2. Integer encoding + 2. Document formats + 2.1. Ed25519 Certificates + 2.2. Basic extensions + 2.2.1. Signed-with-ed25519-key extension [type 04] + 2.3. RSA->Ed25519 cross-certificate + A.1. List of certificate types (CERT_TYPE field) + A.2. List of extension types + A.3. List of signature prefixes + A.4. List of certified key types (CERT_KEY_TYPE field) +``` + + +# Scope and Preliminaries + +This document describes a certificate format that Tor uses for +its Ed25519 internal certificates. It is not the only +certificate format that Tor uses. For the certificates that +authorities use for their signing keys, see dir-spec.txt. +Additionally, Tor uses TLS, which depends on X.509 certificates; +see tor-spec.txt for details. + +The certificates in this document were first introduced in +proposal 220, and were first supported by Tor in Tor version +0.2.7.2-alpha. + + +## Signing + +All signatures here, unless otherwise specified, are computed +using an Ed25519 key. + +In order to future-proof the format, before signing anything, the +signed document is prefixed with a personalization string, which +will be different in each case. + + +## Integer encoding + +Network byte order (big-endian) is used to encode all integer values +in Ed25519 certificates unless explicitly specified otherwise. + + +# Document formats + + +## Ed25519 Certificates + +When generating a signing key, we also generate a certificate for it. +Unlike the certificates for authorities' signing keys, these +certificates need to be sent around frequently, in significant +numbers. So we'll choose a compact representation. + +```text + VERSION [1 Byte] + CERT_TYPE [1 Byte] + EXPIRATION_DATE [4 Bytes] + CERT_KEY_TYPE [1 byte] + CERTIFIED_KEY [32 Bytes] + N_EXTENSIONS [1 byte] + EXTENSIONS [N_EXTENSIONS times] + SIGNATURE [64 Bytes] +``` + +The "VERSION" field holds the value [01]. The "CERT_TYPE" field +holds a value depending on the type of certificate. (See appendix +A.1.) The CERTIFIED_KEY field is an Ed25519 public key if +CERT_KEY_TYPE is [01], or a digest of some other key type +depending on the value of CERT_KEY_TYPE. (See appendix A.4.) +The EXPIRATION_DATE is a date, given in HOURS since the epoch, +after which this certificate isn't valid. (A four-byte field here +will work fine until 10136 A.D.) + +The EXTENSIONS field contains zero or more extensions, each of +the format: + +```text + ExtLength [2 bytes] + ExtType [1 byte] + ExtFlags [1 byte] + ExtData [ExtLength bytes] + + The meaning of the ExtData field in an extension is type-dependent. + + The ExtFlags field holds flags; this flag is currently defined: + + 1 -- AFFECTS_VALIDATION. If this flag is present, then the + extension affects whether the certificate is valid; clients + must not accept the certificate as valid unless they + understand the extension. +``` + +It is an error for an extension to be truncated; such a +certificate is invalid. + +Before processing any certificate, parties SHOULD know which +identity key it is supposed to be signed by, and then check the +signature. The signature is created by signing all the fields in +the certificate up until "SIGNATURE" (that is, signing +sizeof(ed25519_cert) - 64 bytes). + + +## Basic extensions + + +### Signed-with-ed25519-key extension [type 04] + +In several places, it's desirable to bundle the key signing a +certificate along with the certificate. We do so with this +extension. + +```text + ExtLength = 32 + ExtData = + An ed25519 key [32 bytes] +``` + +When this extension is present, it MUST match the key used to +sign the certificate. + + +## RSA->Ed25519 cross-certificate + +Certificate type [07] (Cross-certification of Ed25519 identity +with RSA key) contains the following data: + +```text + ED25519_KEY [32 bytes] + EXPIRATION_DATE [4 bytes] + SIGLEN [1 byte] + SIGNATURE [SIGLEN bytes] +``` + +Here, the Ed25519 identity key is signed with router's RSA +identity key, to indicate that authenticating with a key +certified by the Ed25519 key counts as certifying with RSA +identity key. (The signature is computed on the SHA256 hash of +the non-signature parts of the certificate, prefixed with the +string "Tor TLS RSA/Ed25519 cross-certificate".) + +Just like with the Ed25519 certificates above, the EXPIRATION_DATE +operates in HOURS after the epoch. + +This certificate type is used to mean, "This Ed25519 identity key +acts with the authority of the RSA key that signed this +certificate." + + +## List of certificate types (CERT_TYPE field) + +The values marked with asterisks are not types corresponding to +the certificate format of section 2.1. Instead, they are +reserved for RSA-signed certificates to avoid conflicts between +the certificate type enumeration of the CERTS cell and the +certificate type enumeration of in our Ed25519 certificates. + +```text + **[00],[01],[02],[03] - Reserved to avoid conflict with types used + in CERTS cells. + + [04] - Ed25519 signing key with an identity key + (see prop220 section 4.2) + + [05] - TLS link certificate signed with ed25519 signing key + (see prop220 section 4.2) + + [06] - Ed25519 authentication key signed with ed25519 signing key + (see prop220 section 4.2) + + **[07] - Reserved for RSA identity cross-certification; + (see section 2.3 above, and tor-spec.txt section 4.2) + + [08] - Onion service: short-term descriptor signing key, signed + with blinded public key. + (See rend-spec-v3.txt, section [DESC_OUTER]) + + [09] - Onion service: intro point authentication key, cross-certifying the + descriptor signing key. + (See rend-spec-v3.txt, description of "auth-key") + + [0A] - ntor onion key cross-certifying ed25519 identity key + (see dir-spec.txt, description of "ntor-onion-key-crosscert") + + [0B] - Onion service: ntor-extra encryption key, cross-certifying + descriptor signing key. + (see rend-spec-v3.txt, description of "enc-key-cert") +``` + + +## List of extension types + +[04] - signed-with-ed25519-key (section 2.2.1) + + +## List of signature prefixes + +We describe various documents as being signed with a prefix. Here +are those prefixes: + +"Tor router descriptor signature v1" (see dir-spec.txt) + + +## List of certified key types (CERT_KEY_TYPE field) + +```text + [01] ed25519 key + [02] SHA256 hash of an RSA key. (Not currently used.) + [03] SHA256 hash of an X.509 certificate. (Used with certificate + type 5.) +``` + +(NOTE: Up till 0.4.5.1-alpha, all versions of Tor have incorrectly used +"01" for all types of certified key. Implementations SHOULD +allow "01" in this position, and infer the actual key type from +the CERT_TYPE field.) + diff --git a/spec/control-spec-intro.md b/spec/control-spec-intro.md new file mode 100644 index 0000000..209d40c --- /dev/null +++ b/spec/control-spec-intro.md @@ -0,0 +1,93 @@ +TC: A Tor control protocol (Version 1) + +Table of Contents + +```text + 0. Scope + 1. Protocol outline + 1.1. Forward-compatibility + 2. Message format + 2.1. Description format + 2.1.1. Notes on an escaping bug + 2.2. Commands from controller to Tor + 2.3. Replies from Tor to the controller + 2.4. General-use tokens + 3. Commands + 3.1. SETCONF + 3.2. RESETCONF + 3.3. GETCONF + 3.4. SETEVENTS + 3.5. AUTHENTICATE + 3.6. SAVECONF + 3.7. SIGNAL + 3.8. MAPADDRESS + 3.9. GETINFO + 3.10. EXTENDCIRCUIT + 3.11. SETCIRCUITPURPOSE + 3.12. SETROUTERPURPOSE + 3.13. ATTACHSTREAM + 3.14. POSTDESCRIPTOR + 3.15. REDIRECTSTREAM + 3.16. CLOSESTREAM + 3.17. CLOSECIRCUIT + 3.18. QUIT + 3.19. USEFEATURE + 3.20. RESOLVE + 3.21. PROTOCOLINFO + 3.22. LOADCONF + 3.23. TAKEOWNERSHIP + 3.24. AUTHCHALLENGE + 3.25. DROPGUARDS + 3.26. HSFETCH + 3.27. ADD_ONION + 3.28. DEL_ONION + 3.29. HSPOST + 3.30. ONION_CLIENT_AUTH_ADD + 3.31. ONION_CLIENT_AUTH_REMOVE + 3.32. ONION_CLIENT_AUTH_VIEW + 3.33. DROPOWNERSHIP + 3.34. DROPTIMEOUTS + 4. Replies + 4.1. Asynchronous events + 4.1.1. Circuit status changed + 4.1.2. Stream status changed + 4.1.3. OR Connection status changed + 4.1.4. Bandwidth used in the last second + 4.1.5. Log messages + 4.1.6. New descriptors available + 4.1.7. New Address mapping + 4.1.8. Descriptors uploaded to us in our role as authoritative dirserver + 4.1.9. Our descriptor changed + 4.1.10. Status events + 4.1.11. Our set of guard nodes has changed + 4.1.12. Network status has changed + 4.1.13. Bandwidth used on an application stream + 4.1.14. Per-country client stats + 4.1.15. New consensus networkstatus has arrived + 4.1.16. New circuit buildtime has been set + 4.1.17. Signal received + 4.1.18. Configuration changed + 4.1.19. Circuit status changed slightly + 4.1.20. Pluggable transport launched + 4.1.21. Bandwidth used on an OR or DIR or EXIT connection + 4.1.22. Bandwidth used by all streams attached to a circuit + 4.1.23. Per-circuit cell stats + 4.1.24. Token buckets refilled + 4.1.25. HiddenService descriptors + 4.1.26. HiddenService descriptors content + 4.1.27. Network liveness has changed + 4.1.28. Pluggable Transport Logs + 4.1.29. Pluggable Transport Status + 5. Implementation notes + 5.1. Authentication + 5.2. Don't let the buffer get too big + 5.3. Backward compatibility with v0 control protocol + 5.4. Tor config options for use by controllers + 5.5. Phases from the Bootstrap status event + 5.5.1. Overview of Bootstrap reporting. + 5.5.2. Phases in Bootstrap Stage 1 + 5.5.3. Phases in Bootstrap Stage 2 + 5.5.4. Phases in Bootstrap Stage 3 + 5.6 Bootstrap phases reported by older versions of Tor +``` + diff --git a/spec/control-spec/commands.md b/spec/control-spec/commands.md new file mode 100644 index 0000000..254633e --- /dev/null +++ b/spec/control-spec/commands.md @@ -0,0 +1,1876 @@ + +# Commands + +All commands are case-insensitive, but most keywords are case-sensitive. + + +## SETCONF + +Change the value of one or more configuration variables. The syntax is: + +```text + "SETCONF" 1*(SP keyword ["=" value]) CRLF + value = String / QuotedString +``` + +Tor behaves as though it had just read each of the key-value pairs +from its configuration file. Keywords with no corresponding values have +their configuration values reset to 0 or NULL (use RESETCONF if you want +to set it back to its default). SETCONF is all-or-nothing: if there +is an error in any of the configuration settings, Tor sets none of them. + +Tor responds with a "250 OK" reply on success. +If some of the listed keywords can't be found, Tor replies with a +"552 Unrecognized option" message. Otherwise, Tor responds with a +"513 syntax error in configuration values" reply on syntax error, or a +"553 impossible configuration setting" reply on a semantic error. + +Some configuration options (e.g. "Bridge") take multiple values. Also, +some configuration keys (e.g. for hidden services and for entry +guard lists) form a context-sensitive group where order matters (see +GETCONF below). In these cases, setting _any_ of the options in a +SETCONF command is taken to reset all of the others. For example, +if two ORListenAddress values are configured, and a SETCONF command +arrives containing a single ORListenAddress value, the new command's +value replaces the two old values. + +Sometimes it is not possible to change configuration options solely by +issuing a series of SETCONF commands, because the value of one of the +configuration options depends on the value of another which has not yet +been set. Such situations can be overcome by setting multiple configuration +options with a single SETCONF command (e.g. SETCONF ORPort=443 +ORListenAddress=9001). + + +## RESETCONF + +Remove all settings for a given configuration option entirely, assign +its default value (if any), and then assign the String provided. +Typically the String is left empty, to simply set an option back to +its default. The syntax is: + +"RESETCONF" 1*(SP keyword ["=" String]) CRLF + +Otherwise it behaves like SETCONF above. + + +## GETCONF + +Request the value of zero or more configuration variable(s). +The syntax is: + +"GETCONF" *(SP keyword) CRLF + +If all of the listed keywords exist in the Tor configuration, Tor replies +with a series of reply lines of the form: + +250 keyword=value + +If any option is set to a 'default' value semantically different from an +empty string, Tor may reply with a reply line of the form: + +250 keyword + +Value may be a raw value or a quoted string. Tor will try to use unquoted +values except when the value could be misinterpreted through not being +quoted. (Right now, Tor supports no such misinterpretable values for +configuration options.) + +If some of the listed keywords can't be found, Tor replies with a +"552 unknown configuration keyword" message. + +If an option appears multiple times in the configuration, all of its +key-value pairs are returned in order. + +If no keywords were provided, Tor responds with "250 OK" message. + +Some options are context-sensitive, and depend on other options with +different keywords. These cannot be fetched directly. Currently there +is only one such option: clients should use the "HiddenServiceOptions" +virtual keyword to get all HiddenServiceDir, HiddenServicePort, +HiddenServiceVersion, and HiddenserviceAuthorizeClient option settings. + + +## SETEVENTS + +Request the server to inform the client about interesting events. The +syntax is: + +"SETEVENTS" [SP "EXTENDED"] *(SP EventCode) CRLF + +EventCode = 1*(ALPHA / "_") (see section 4.1.x for event types) + +Any events *not* listed in the SETEVENTS line are turned off; thus, sending +SETEVENTS with an empty body turns off all event reporting. + +The server responds with a "250 OK" reply on success, and a "552 +Unrecognized event" reply if one of the event codes isn't recognized. (On +error, the list of active event codes isn't changed.) + +If the flag string "EXTENDED" is provided, Tor may provide extra +information with events for this connection; see 4.1 for more information. +NOTE: All events on a given connection will be provided in extended format, +or none. +NOTE: "EXTENDED" was first supported in Tor 0.1.1.9-alpha; it is +always-on in Tor 0.2.2.1-alpha and later. + +Each event is described in more detail in Section 4.1. + + +## AUTHENTICATE + +Sent from the client to the server. The syntax is: + +"AUTHENTICATE" [ SP 1*HEXDIG / QuotedString ] CRLF + +This command is used to authenticate to the server. The provided string is +one of the following: + +```text + * (For the HASHEDPASSWORD authentication method; see 3.21) + The original password represented as a QuotedString. + + * (For the COOKIE is authentication method; see 3.21) + The contents of the cookie file, formatted in hexadecimal + + * (For the SAFECOOKIE authentication method; see 3.21) + The HMAC based on the AUTHCHALLENGE message, in hexadecimal. +``` + +The server responds with "250 OK" on success or "515 Bad authentication" if +the authentication cookie is incorrect. Tor closes the connection on an +authentication failure. + +The authentication token can be specified as either a quoted ASCII string, +or as an unquoted hexadecimal encoding of that same string (to avoid escaping +issues). + +For information on how the implementation securely stores authentication +information on disk, see section 5.1. + +Before the client has authenticated, no command other than +PROTOCOLINFO, AUTHCHALLENGE, AUTHENTICATE, or QUIT is valid. If the +controller sends any other command, or sends a malformed command, or +sends an unsuccessful AUTHENTICATE command, or sends PROTOCOLINFO or +AUTHCHALLENGE more than once, Tor sends an error reply and closes +the connection. + +To prevent some cross-protocol attacks, the AUTHENTICATE command is still +required even if all authentication methods in Tor are disabled. In this +case, the controller should just send "AUTHENTICATE" CRLF. + +(Versions of Tor before 0.1.2.16 and 0.2.0.4-alpha did not close the +connection after an authentication failure.) + + +## SAVECONF + +Sent from the client to the server. The syntax is: + +"SAVECONF" [SP "FORCE"] CRLF + +Instructs the server to write out its config options into its torrc. Server +returns "250 OK" if successful, or "551 Unable to write configuration +to disk" if it can't write the file or some other error occurs. + +If the %include option is used on torrc, SAVECONF will not write the +configuration to disk. If the flag string "FORCE" is provided, the +configuration will be overwritten even if %include is used. Using %include +on defaults-torrc does not affect SAVECONF. (Introduced in 0.3.1.1-alpha.) + +See also the "getinfo config-text" command, if the controller wants +to write the torrc file itself. + +See also the "getinfo config-can-saveconf" command, to tell if the FORCE +flag will be required. (Also introduced in 0.3.1.1-alpha.) + + +## SIGNAL + +Sent from the client to the server. The syntax is: + +"SIGNAL" SP Signal CRLF + +```text + Signal = "RELOAD" / "SHUTDOWN" / "DUMP" / "DEBUG" / "HALT" / + "HUP" / "INT" / "USR1" / "USR2" / "TERM" / "NEWNYM" / + "CLEARDNSCACHE" / "HEARTBEAT" / "ACTIVE" / "DORMANT" + + The meaning of the signals are: + + RELOAD -- Reload: reload config items. + SHUTDOWN -- Controlled shutdown: if server is an OP, exit immediately. + If it's an OR, close listeners and exit after + ShutdownWaitLength seconds. + DUMP -- Dump stats: log information about open connections and + circuits. + DEBUG -- Debug: switch all open logs to loglevel debug. + HALT -- Immediate shutdown: clean up and exit now. + CLEARDNSCACHE -- Forget the client-side cached IPs for all hostnames. + NEWNYM -- Switch to clean circuits, so new application requests + don't share any circuits with old ones. Also clears + the client-side DNS cache. (Tor MAY rate-limit its + response to this signal.) + HEARTBEAT -- Make Tor dump an unscheduled Heartbeat message to log. + DORMANT -- Tell Tor to become "dormant". A dormant Tor will + try to avoid CPU and network usage until it receives + user-initiated network request. (Don't use this + on relays or hidden services yet!) + ACTIVE -- Tell Tor to stop being "dormant", as if it had received + a user-initiated network request. +``` + +The server responds with "250 OK" if the signal is recognized (or simply +closes the socket if it was asked to close immediately), or "552 +Unrecognized signal" if the signal is unrecognized. + +Note that not all of these signals have POSIX signal equivalents. The +ones that do are as below. You may also use these POSIX names for the +signal that have them. + +```text + RELOAD: HUP + SHUTDOWN: INT + HALT: TERM + DUMP: USR1 + DEBUG: USR2 + + [SIGNAL DORMANT and SIGNAL ACTIVE were added in 0.4.0.1-alpha.] +``` + + +## MAPADDRESS + +Sent from the client to the server. The syntax is: + +"MAPADDRESS" 1*(Address "=" Address SP) CRLF + +The first address in each pair is an "original" address; the second is a +"replacement" address. The client sends this message to the server in +order to tell it that future SOCKS requests for connections to the original +address should be replaced with connections to the specified replacement +address. If the addresses are well-formed, and the server is able to +fulfill the request, the server replies with a 250 message: + +```text + 250-OldAddress1=NewAddress1 + 250 OldAddress2=NewAddress2 +``` + +containing the source and destination addresses. If request is +malformed, the server replies with "512 syntax error in command +argument". If the server can't fulfill the request, it replies with +"451 resource exhausted". + +The client may decline to provide a body for the original address, and +instead send a special null address ("0.0.0.0" for IPv4, "::0" for IPv6, or +"." for hostname), signifying that the server should choose the original +address itself, and return that address in the reply. The server +should ensure that it returns an element of address space that is unlikely +to be in actual use. If there is already an address mapped to the +destination address, the server may reuse that mapping. + +If the original address is already mapped to a different address, the old +mapping is removed. If the original address and the destination address +are the same, the server removes any mapping in place for the original +address. + +Example: + +```text + C: MAPADDRESS 1.2.3.4=torproject.org + S: 250 1.2.3.4=torproject.org + + C: GETINFO address-mappings/control + S: 250-address-mappings/control=1.2.3.4 torproject.org NEVER + S: 250 OK + + C: MAPADDRESS 1.2.3.4=1.2.3.4 + S: 250 1.2.3.4=1.2.3.4 + + C: GETINFO address-mappings/control + S: 250-address-mappings/control= + S: 250 OK +``` + +{Note: This feature is designed to be used to help Tor-ify applications +that need to use SOCKS4 or hostname-less SOCKS5. There are three +approaches to doing this: + +```text + 1. Somehow make them use SOCKS4a or SOCKS5-with-hostnames instead. + 2. Use tor-resolve (or another interface to Tor's resolve-over-SOCKS + feature) to resolve the hostname remotely. This doesn't work + with special addresses like x.onion or x.y.exit. + 3. Use MAPADDRESS to map an IP address to the desired hostname, and then + arrange to fool the application into thinking that the hostname + has resolved to that IP. + + This functionality is designed to help implement the 3rd approach.} +``` + +Mappings set by the controller last until the Tor process exits: +they never expire. If the controller wants the mapping to last only +a certain time, then it must explicitly un-map the address when that +time has elapsed. + +MapAddress replies MAY contain mixed status codes. + +Example: + +```text + C: MAPADDRESS xxx=@@@ 0.0.0.0=bogus1.google.com + S: 512-syntax error: invalid address '@@@' + S: 250 127.199.80.246=bogus1.google.com +``` + + +## GETINFO + +Sent from the client to the server. The syntax is as for GETCONF: + +"GETINFO" 1*(SP keyword) CRLF + +Unlike GETCONF, this message is used for data that are not stored in the Tor +configuration file, and that may be longer than a single line. On success, +one ReplyLine is sent for each requested value, followed by a final 250 OK +ReplyLine. If a value fits on a single line, the format is: + +```text + 250-keyword=value + If a value must be split over multiple lines, the format is: + + 250+keyword= + value + . + The server sends a 551 or 552 error on failure. + + Recognized keys and their values include: + + "version" -- The version of the server's software, which MAY include the + name of the software, such as "Tor 0.0.9.4". The name of the software, + if absent, is assumed to be "Tor". + + "config-file" -- The location of Tor's configuration file ("torrc"). + + "config-defaults-file" -- The location of Tor's configuration + defaults file ("torrc.defaults"). This file gets parsed before + torrc, and is typically used to replace Tor's default + configuration values. [First implemented in 0.2.3.9-alpha.] + + "config-text" -- The contents that Tor would write if you send it + a SAVECONF command, so the controller can write the file to + disk itself. [First implemented in 0.2.2.7-alpha.] + + "exit-policy/default" -- The default exit policy lines that Tor will + *append* to the ExitPolicy config option. + + "exit-policy/reject-private/default" -- The default exit policy lines + that Tor will *prepend* to the ExitPolicy config option when + ExitPolicyRejectPrivate is 1. + + "exit-policy/reject-private/relay" -- The relay-specific exit policy + lines that Tor will *prepend* to the ExitPolicy config option based + on the current values of ExitPolicyRejectPrivate and + ExitPolicyRejectLocalInterfaces. These lines are based on the public + addresses configured in the torrc and present on the relay's + interfaces. Will send 552 error if the server is not running as + onion router. Will send 551 on internal error which may be transient. + + "exit-policy/ipv4" + "exit-policy/ipv6" + "exit-policy/full" -- This OR's exit policy, in IPv4-only, IPv6-only, or + all-entries flavors. Handles errors in the same way as "exit-policy/ + reject-private/relay" does. + + "desc/id/" or "desc/name/" -- the latest + server descriptor for a given OR. (Note that modern Tor clients + do not download server descriptors by default, but download + microdescriptors instead. If microdescriptors are enabled, you'll + need to use "md" instead.) + + "md/all" -- all known microdescriptors for the entire Tor network. + Each microdescriptor is terminated by a newline. + [First implemented in 0.3.5.1-alpha] + + "md/id/" or "md/name/" -- the latest + microdescriptor for a given OR. Empty if we have no microdescriptor for + that OR (because we haven't downloaded one, or it isn't in the + consensus). [First implemented in 0.2.3.8-alpha.] + + "desc/download-enabled" -- "1" if we try to download router descriptors; + "0" otherwise. [First implemented in 0.3.2.1-alpha] + + "md/download-enabled" -- "1" if we try to download microdescriptors; + "0" otherwise. [First implemented in 0.3.2.1-alpha] + + "dormant" -- A nonnegative integer: zero if Tor is currently active and + building circuits, and nonzero if Tor has gone idle due to lack of use + or some similar reason. [First implemented in 0.2.3.16-alpha] + + "desc-annotations/id/" -- outputs the annotations string + (source, timestamp of arrival, purpose, etc) for the corresponding + descriptor. [First implemented in 0.2.0.13-alpha.] + + "extra-info/digest/" -- the extrainfo document whose digest (in + hex) is . Only available if we're downloading extra-info + documents. + + "ns/id/" or "ns/name/" -- the latest router + status info (v3 directory style) for a given OR. Router status + info is as given in dir-spec.txt, and reflects the latest + consensus opinion about the + router in question. Like directory clients, controllers MUST + tolerate unrecognized flags and lines. The published date and + descriptor digest are those believed to be best by this Tor, + not necessarily those for a descriptor that Tor currently has. + [First implemented in 0.1.2.3-alpha.] + [In 0.2.0.9-alpha this switched from v2 directory style to v3] + + "ns/all" -- Router status info (v3 directory style) for all ORs we + that the consensus has an opinion about, joined by newlines. + [First implemented in 0.1.2.3-alpha.] + [In 0.2.0.9-alpha this switched from v2 directory style to v3] + + "ns/purpose/" -- Router status info (v3 directory style) + for all ORs of this purpose. Mostly designed for /ns/purpose/bridge + queries. + [First implemented in 0.2.0.13-alpha.] + [In 0.2.0.9-alpha this switched from v2 directory style to v3] + [In versions before 0.4.1.1-alpha we set the Running flag on + bridges when /ns/purpose/bridge is accessed] + [In 0.4.1.1-alpha we set the Running flag on bridges when the + bridge networkstatus file is written to disk] + + "desc/all-recent" -- the latest server descriptor for every router that + Tor knows about. (See md note about "desc/id" and "desc/name" above.) + + "network-status" -- [Deprecated in 0.3.1.1-alpha, removed + in 0.4.5.1-alpha.] + + "address-mappings/all" + "address-mappings/config" + "address-mappings/cache" + "address-mappings/control" -- a \r\n-separated list of address + mappings, each in the form of "from-address to-address expiry". + The 'config' key returns those address mappings set in the + configuration; the 'cache' key returns the mappings in the + client-side DNS cache; the 'control' key returns the mappings set + via the control interface; the 'all' target returns the mappings + set through any mechanism. + Expiry is formatted as with ADDRMAP events, except that "expiry" is + always a time in UTC or the string "NEVER"; see section 4.1.7. + First introduced in 0.2.0.3-alpha. + + "addr-mappings/*" -- as for address-mappings/*, but without the + expiry portion of the value. Use of this value is deprecated + since 0.2.0.3-alpha; use address-mappings instead. + + "address" -- the best guess at our external IP address. If we + have no guess, return a 551 error. (Added in 0.1.2.2-alpha) + + "address/v4" + "address/v6" + the best guess at our respective external IPv4 or IPv6 address. + If we have no guess, return a 551 error. (Added in 0.4.5.1-alpha) + + "fingerprint" -- the contents of the fingerprint file that Tor + writes as a relay, or a 551 if we're not a relay currently. + (Added in 0.1.2.3-alpha) + + "circuit-status" + A series of lines as for a circuit status event. Each line is of + the form described in section 4.1.1, omitting the initial + "650 CIRC ". Note that clients must be ready to accept additional + arguments as described in section 4.1. + + "stream-status" + A series of lines as for a stream status event. Each is of the form: + StreamID SP StreamStatus SP CircuitID SP Target CRLF + + "orconn-status" + A series of lines as for an OR connection status event. In Tor + 0.1.2.2-alpha with feature VERBOSE_NAMES enabled and in Tor + 0.2.2.1-alpha and later by default, each line is of the form: + LongName SP ORStatus CRLF + + In Tor versions 0.1.2.2-alpha through 0.2.2.1-alpha with feature + VERBOSE_NAMES turned off and before version 0.1.2.2-alpha, each line + is of the form: + ServerID SP ORStatus CRLF + + "entry-guards" + A series of lines listing the currently chosen entry guards, if any. + In Tor 0.1.2.2-alpha with feature VERBOSE_NAMES enabled and in Tor + 0.2.2.1-alpha and later by default, each line is of the form: + LongName SP Status [SP ISOTime] CRLF + + In Tor versions 0.1.2.2-alpha through 0.2.2.1-alpha with feature + VERBOSE_NAMES turned off and before version 0.1.2.2-alpha, each line + is of the form: + ServerID2 SP Status [SP ISOTime] CRLF + ServerID2 = Nickname / 40*HEXDIG + + The definition of Status is the same for both: + Status = "up" / "never-connected" / "down" / + "unusable" / "unlisted" + + [From 0.1.1.4-alpha to 0.1.1.10-alpha, entry-guards was called + "helper-nodes". Tor still supports calling "helper-nodes", but it + is deprecated and should not be used.] + + [Older versions of Tor (before 0.1.2.x-final) generated 'down' instead + of unlisted/unusable. Between 0.1.2.x-final and 0.2.6.3-alpha, + 'down' was never generated.] + + [XXXX ServerID2 differs from ServerID in not prefixing fingerprints + with a $. This is an implementation error. It would be nice to add + the $ back in if we can do so without breaking compatibility.] + + "traffic/read" -- Total bytes read (downloaded). + + "traffic/written" -- Total bytes written (uploaded). + + "uptime" -- Uptime of the Tor daemon (in seconds). Added in + 0.3.5.1-alpha. + + "accounting/enabled" + "accounting/hibernating" + "accounting/bytes" + "accounting/bytes-left" + "accounting/interval-start" + "accounting/interval-wake" + "accounting/interval-end" + Information about accounting status. If accounting is enabled, + "enabled" is 1; otherwise it is 0. The "hibernating" field is "hard" + if we are accepting no data; "soft" if we're accepting no new + connections, and "awake" if we're not hibernating at all. The "bytes" + and "bytes-left" fields contain (read-bytes SP write-bytes), for the + start and the rest of the interval respectively. The 'interval-start' + and 'interval-end' fields are the borders of the current interval; the + 'interval-wake' field is the time within the current interval (if any) + where we plan[ned] to start being active. The times are UTC. + + "config/names" + A series of lines listing the available configuration options. Each is + of the form: + OptionName SP OptionType [ SP Documentation ] CRLF + OptionName = Keyword + OptionType = "Integer" / "TimeInterval" / "TimeMsecInterval" / + "DataSize" / "Float" / "Boolean" / "Time" / "CommaList" / + "Dependent" / "Virtual" / "String" / "LineList" + Documentation = Text + Note: The incorrect spelling "Dependant" was used from the time this key + was introduced in Tor 0.1.1.4-alpha until it was corrected in Tor + 0.3.0.2-alpha. It is recommended that clients accept both spellings. + + "config/defaults" + A series of lines listing default values for each configuration + option. Options which don't have a valid default don't show up + in the list. Introduced in Tor 0.2.4.1-alpha. + OptionName SP OptionValue CRLF + OptionName = Keyword + OptionValue = Text + + "info/names" + A series of lines listing the available GETINFO options. Each is of + one of these forms: + OptionName SP Documentation CRLF + OptionPrefix SP Documentation CRLF + OptionPrefix = OptionName "/*" + The OptionPrefix form indicates a number of options beginning with the + prefix. So if "config/*" is listed, other options beginning with + "config/" will work, but "config/*" itself is not an option. + + "events/names" + A space-separated list of all the events supported by this version of + Tor's SETEVENTS. + + "features/names" + A space-separated list of all the features supported by this version + of Tor's USEFEATURE. + + "signal/names" + A space-separated list of all the values supported by the SIGNAL + command. + + "ip-to-country/ipv4-available" + "ip-to-country/ipv6-available" + "1" if the relevant geoip or geoip6 database is present; "0" otherwise. + This field was added in Tor 0.3.2.1-alpha. + + "ip-to-country/*" + Maps IP addresses to 2-letter country codes. For example, + "GETINFO ip-to-country/18.0.0.1" should give "US". + + "process/pid" -- Process id belonging to the main tor process. + "process/uid" -- User id running the tor process, -1 if unknown (this is + unimplemented on Windows, returning -1). + "process/user" -- Username under which the tor process is running, + providing an empty string if none exists (this is unimplemented on + Windows, returning an empty string). + "process/descriptor-limit" -- Upper bound on the file descriptor limit, -1 + if unknown + + "dir/status-vote/current/consensus" [added in Tor 0.2.1.6-alpha] + "dir/status-vote/current/consensus-microdesc" [added in Tor 0.4.3.1-alpha] + "dir/status/authority" + "dir/status/fp/" + "dir/status/fp/++" + "dir/status/all" + "dir/server/fp/" + "dir/server/fp/++" + "dir/server/d/" + "dir/server/d/++" + "dir/server/authority" + "dir/server/all" + A series of lines listing directory contents, provided according to the + specification for the URLs listed in Section 4.4 of dir-spec.txt. Note + that Tor MUST NOT provide private information, such as descriptors for + routers not marked as general-purpose. When asked for 'authority' + information for which this Tor is not authoritative, Tor replies with + an empty string. + + Note that, as of Tor 0.2.3.3-alpha, Tor clients don't download server + descriptors anymore, but microdescriptors. So, a "551 Servers + unavailable" reply to all "GETINFO dir/server/*" requests is actually + correct. If you have an old program which absolutely requires server + descriptors to work, try setting UseMicrodescriptors 0 or + FetchUselessDescriptors 1 in your client's torrc. + + "status/circuit-established" + "status/enough-dir-info" + "status/good-server-descriptor" + "status/accepted-server-descriptor" + "status/..." + These provide the current internal Tor values for various Tor + states. See Section 4.1.10 for explanations. (Only a few of the + status events are available as getinfo's currently. Let us know if + you want more exposed.) + "status/reachability-succeeded/or" + 0 or 1, depending on whether we've found our ORPort reachable. + "status/reachability-succeeded/dir" + 0 or 1, depending on whether we've found our DirPort reachable. + 1 if there is no DirPort, and therefore no need for a reachability + check. + "status/reachability-succeeded" + "OR=" ("0"/"1") SP "DIR=" ("0"/"1") + Combines status/reachability-succeeded/*; controllers MUST ignore + unrecognized elements in this entry. + "status/bootstrap-phase" + Returns the most recent bootstrap phase status event + sent. Specifically, it returns a string starting with either + "NOTICE BOOTSTRAP ..." or "WARN BOOTSTRAP ...". Controllers should + use this getinfo when they connect or attach to Tor to learn its + current bootstrap state. + "status/version/recommended" + List of currently recommended versions. + "status/version/current" + Status of the current version. One of: new, old, unrecommended, + recommended, new in series, obsolete, unknown. + "status/clients-seen" + A summary of which countries we've seen clients from recently, + formatted the same as the CLIENTS_SEEN status event described in + Section 4.1.14. This GETINFO option is currently available only + for bridge relays. + "status/fresh-relay-descs" + Provides fresh server and extra-info descriptors for our relay. Note + this is *not* the latest descriptors we've published, but rather what we + would generate if we needed to make a new descriptor right now. + + "net/listeners/*" + + A quoted, space-separated list of the locations where Tor is listening + for connections of the specified type. These can contain IPv4 + network address... + + "127.0.0.1:9050" "127.0.0.1:9051" + + ... or local unix sockets... + + "unix:/home/my_user/.tor/socket" + + ... or IPv6 network addresses: + + "[2001:0db8:7000:0000:0000:dead:beef:1234]:9050" + + [New in Tor 0.2.2.26-beta.] + + "net/listeners/or" + + Listeners for OR connections. Talks Tor protocol as described in + tor-spec.txt. + + "net/listeners/dir" + + Listeners for Tor directory protocol, as described in dir-spec.txt. + + "net/listeners/socks" + + Listeners for onion proxy connections that talk SOCKS4/4a/5 protocol. + + "net/listeners/trans" + + Listeners for transparent connections redirected by firewall, such as + pf or netfilter. + + "net/listeners/natd" + + Listeners for transparent connections redirected by natd. + + "net/listeners/dns" + + Listeners for a subset of DNS protocol that Tor network supports. + + "net/listeners/control" + + Listeners for Tor control protocol, described herein. + + "net/listeners/extor" + + Listeners corresponding to Extended ORPorts for integration with + pluggable transports. See proposals 180 and 196. + + "net/listeners/httptunnel" + + Listeners for onion proxy connections that leverage HTTP CONNECT + tunnelling. + + [The extor and httptunnel lists were added in 0.3.2.12, 0.3.3.10, and + 0.3.4.6-rc.] + + "dir-usage" + A newline-separated list of how many bytes we've served to answer + each type of directory request. The format of each line is: + Keyword 1*SP Integer 1*SP Integer + where the first integer is the number of bytes written, and the second + is the number of requests answered. + + [This feature was added in Tor 0.2.2.1-alpha, and removed in + Tor 0.2.9.1-alpha. Even when it existed, it only provided + useful output when the Tor client was built with either the + INSTRUMENT_DOWNLOADS or RUNNING_DOXYGEN compile-time options.] + + "bw-event-cache" + A space-separated summary of recent BW events in chronological order + from oldest to newest. Each event is represented by a comma-separated + tuple of "R,W", R is the number of bytes read, and W is the number of + bytes written. These entries each represent about one second's worth + of traffic. + [New in Tor 0.2.6.3-alpha] + + "consensus/valid-after" + "consensus/fresh-until" + "consensus/valid-until" + Each of these produces an ISOTime describing part of the lifetime of + the current (valid, accepted) consensus that Tor has. + [New in Tor 0.2.6.3-alpha] + + "hs/client/desc/id/" + Prints the content of the hidden service descriptor corresponding to + the given which is an onion address without the ".onion" part. + The client's cache is queried to find the descriptor. The format of + the descriptor is described in section 1.3 of the rend-spec.txt + document. + + If is unrecognized or if not found in the cache, a 551 error is + returned. + + [New in Tor 0.2.7.1-alpha] + [HS v3 support added 0.3.3.1-alpha] + + "hs/service/desc/id/" + Prints the content of the hidden service descriptor corresponding to + the given which is an onion address without the ".onion" part. + The service's local descriptor cache is queried to find the descriptor. + The format of the descriptor is described in section 1.3 of the + rend-spec.txt document. + + If is unrecognized or if not found in the cache, a 551 error is + returned. + + [New in Tor 0.2.7.2-alpha] + [HS v3 support added 0.3.3.1-alpha] + + "onions/current" + "onions/detached" + A newline-separated list of the Onion ("Hidden") Services created + via the "ADD_ONION" command. The 'current' key returns Onion Services + belonging to the current control connection. The 'detached' key + returns Onion Services detached from the parent control connection + (as in, belonging to no control connection). + The format of each line is: + HSAddress + [New in Tor 0.2.7.1-alpha.] + [HS v3 support added 0.3.3.1-alpha] + + "network-liveness" + The string "up" or "down", indicating whether we currently believe the + network is reachable. + + "downloads/" + The keys under downloads/ are used to query download statuses; they all + return either a sequence of newline-terminated hex encoded digests, or + a "serialized download status" as follows: + + SerializedDownloadStatus = + -- when do we plan to next attempt to download this object? + "next-attempt-at" SP ISOTime CRLF + -- how many times have we failed since the last success? + "n-download-failures" SP UInt CRLF + -- how many times have we tried to download this? + "n-download-attempts" SP UInt CRLF + -- according to which schedule rule will we download this? + "schedule" SP DownloadSchedule CRLF + -- do we want to fetch this from an authority, or will any cache do? + "want-authority" SP DownloadWantAuthority CRLF + -- do we increase our download delay whenever we fail to fetch this, + -- or whenever we attempt fetching this? + "increment-on" SP DownloadIncrementOn CRLF + -- do we increase the download schedule deterministically, or at + -- random? + "backoff" SP DownloadBackoff CRLF + [ + -- with an exponential backoff, where are we in the schedule? + "last-backoff-position" Uint CRLF + -- with an exponential backoff, what was our last delay? + "last-delay-used UInt CRLF + ] + + where + + DownloadSchedule = + "DL_SCHED_GENERIC" / "DL_SCHED_CONSENSUS" / "DL_SCHED_BRIDGE" + DownloadWantAuthority = + "DL_WANT_ANY_DIRSERVER" / "DL_WANT_AUTHORITY" + DownloadIncrementOn = + "DL_SCHED_INCREMENT_FAILURE" / "DL_SCHED_INCREMENT_ATTEMPT" + DownloadBackoff = + "DL_SCHED_DETERMINISTIC" / "DL_SCHED_RANDOM_EXPONENTIAL" + + The optional last two lines must be present if DownloadBackoff is + "DL_SCHED_RANDOM_EXPONENTIAL" and must be absent if DownloadBackoff + is "DL_SCHED_DETERMINISTIC". + + In detail, the keys supported are: + + "downloads/networkstatus/ns" + The SerializedDownloadStatus for the NS-flavored consensus for + whichever bootstrap state Tor is currently in. + + "downloads/networkstatus/ns/bootstrap" + The SerializedDownloadStatus for the NS-flavored consensus at + bootstrap time, regardless of whether we are currently bootstrapping. + + "downloads/networkstatus/ns/running" + + The SerializedDownloadStatus for the NS-flavored consensus when + running, regardless of whether we are currently bootstrapping. + + "downloads/networkstatus/microdesc" + The SerializedDownloadStatus for the microdesc-flavored consensus for + whichever bootstrap state Tor is currently in. + + "downloads/networkstatus/microdesc/bootstrap" + The SerializedDownloadStatus for the microdesc-flavored consensus at + bootstrap time, regardless of whether we are currently bootstrapping. + + "downloads/networkstatus/microdesc/running" + The SerializedDownloadStatus for the microdesc-flavored consensus when + running, regardless of whether we are currently bootstrapping. + + "downloads/cert/fps" + + A newline-separated list of hex-encoded digests for authority + certificates for which we have download status available. + + "downloads/cert/fp/" + A SerializedDownloadStatus for the default certificate for the + identity digest returned by the downloads/cert/fps key. + + "downloads/cert/fp//sks" + A newline-separated list of hex-encoded signing key digests for the + authority identity digest returned by the + downloads/cert/fps key. + + "downloads/cert/fp//" + A SerializedDownloadStatus for the certificate for the identity + digest returned by the downloads/cert/fps key and signing + key digest returned by the downloads/cert/fp// + sks key. + + "downloads/desc/descs" + A newline-separated list of hex-encoded router descriptor digests + [note, not identity digests - the Tor process may not have seen them + yet while downloading router descriptors]. If the Tor process is not + using a NS-flavored consensus, a 551 error is returned. + + "downloads/desc/" + A SerializedDownloadStatus for the router descriptor with digest + as returned by the downloads/desc/descs key. If the Tor + process is not using a NS-flavored consensus, a 551 error is returned. + + "downloads/bridge/bridges" + A newline-separated list of hex-encoded bridge identity digests. If + the Tor process is not using bridges, a 551 error is returned. + + "downloads/bridge/" + A SerializedDownloadStatus for the bridge descriptor with identity + digest as returned by the downloads/bridge/bridges key. If + the Tor process is not using bridges, a 551 error is returned. + + "sr/current" + "sr/previous" + The current or previous shared random value, as received in the + consensus, base-64 encoded. An empty value means that either + the consensus has no shared random value, or Tor has no consensus. + + "current-time/local" + "current-time/utc" + The current system or UTC time, as returned by the system, in ISOTime2 + format. (Introduced in 0.3.4.1-alpha.) + + "stats/ntor/requested" + "stats/ntor/assigned" + The NTor circuit onion handshake rephist values which are requested or + assigned. (Introduced in 0.4.5.1-alpha) + + "stats/tap/requested" + "stats/tap/assigned" + The TAP circuit onion handshake rephist values which are requested or + assigned. (Introduced in 0.4.5.1-alpha) + + "config-can-saveconf" + 0 or 1, depending on whether it is possible to use SAVECONF without the + FORCE flag. (Introduced in 0.3.1.1-alpha.) + + "limits/max-mem-in-queues" + The amount of memory that Tor's out-of-memory checker will allow + Tor to allocate (in places it can see) before it starts freeing memory + and killing circuits. See the MaxMemInQueues option for more + details. Unlike the option, this value reflects Tor's actual limit, and + may be adjusted depending on the available system memory rather than on + the MaxMemInQueues option. (Introduced in 0.2.5.4-alpha) + + Examples: + + C: GETINFO version desc/name/moria1 + S: 250+desc/name/moria= + S: [Descriptor for moria] + S: . + S: 250-version=Tor 0.1.1.0-alpha-cvs + S: 250 OK +``` + + +## EXTENDCIRCUIT + +Sent from the client to the server. The format is: + +```text + "EXTENDCIRCUIT" SP CircuitID + [SP ServerSpec *("," ServerSpec)] + [SP "purpose=" Purpose] CRLF +``` + +This request takes one of two forms: either the CircuitID is zero, in +which case it is a request for the server to build a new circuit, +or the CircuitID is nonzero, in which case it is a request for the +server to extend an existing circuit with that ID according to the +specified path. + +If the CircuitID is 0, the controller has the option of providing +a path for Tor to use to build the circuit. If it does not provide +a path, Tor will select one automatically from high capacity nodes +according to path-spec.txt. + +If CircuitID is 0 and "purpose=" is specified, then the circuit's +purpose is set. Two choices are recognized: "general" and +"controller". If not specified, circuits are created as "general". + +If the request is successful, the server sends a reply containing a +message body consisting of the CircuitID of the (maybe newly created) +circuit. The syntax is "250" SP "EXTENDED" SP CircuitID CRLF. + + +## SETCIRCUITPURPOSE + +Sent from the client to the server. The format is: + +"SETCIRCUITPURPOSE" SP CircuitID SP "purpose=" Purpose CRLF + +This changes the circuit's purpose. See EXTENDCIRCUIT above for details. + + +## SETROUTERPURPOSE + +Sent from the client to the server. The format is: + +"SETROUTERPURPOSE" SP NicknameOrKey SP Purpose CRLF + +This changes the descriptor's purpose. See +POSTDESCRIPTOR below +for details. + +NOTE: This command was disabled and made obsolete as of Tor +0.2.0.8-alpha. It doesn't exist anymore, and is listed here only for +historical interest. + + +## ATTACHSTREAM + +Sent from the client to the server. The syntax is: + +"ATTACHSTREAM" SP StreamID SP CircuitID [SP "HOP=" HopNum] CRLF + +This message informs the server that the specified stream should be +associated with the specified circuit. Each stream may be associated with +at most one circuit, and multiple streams may share the same circuit. +Streams can only be attached to completed circuits (that is, circuits that +have sent a circuit status 'BUILT' event or are listed as built in a +GETINFO circuit-status request). + +If the circuit ID is 0, responsibility for attaching the given stream is +returned to Tor. + +If HOP=HopNum is specified, Tor will choose the HopNumth hop in the +circuit as the exit node, rather than the last node in the circuit. +Hops are 1-indexed; generally, it is not permitted to attach to hop 1. + +Tor responds with "250 OK" if it can attach the stream, 552 if the +circuit or stream didn't exist, 555 if the stream isn't in an +appropriate state to be attached (e.g. it's already open), or 551 if +the stream couldn't be attached for another reason. + +{Implementation note: Tor will close unattached streams by itself, +roughly two minutes after they are born. Let the developers know if +that turns out to be a problem.} + +{Implementation note: By default, Tor automatically attaches streams to +circuits itself, unless the configuration variable +"__LeaveStreamsUnattached" is set to "1". Attempting to attach streams +via TC when "__LeaveStreamsUnattached" is false may cause a race between +Tor and the controller, as both attempt to attach streams to circuits.} + +{Implementation note: You can try to attachstream to a stream that +has already sent a connect or resolve request but hasn't succeeded +yet, in which case Tor will detach the stream from its current circuit +before proceeding with the new attach request.} + + +## POSTDESCRIPTOR + +Sent from the client to the server. The syntax is: + +```text + "+POSTDESCRIPTOR" [SP "purpose=" Purpose] [SP "cache=" Cache] + CRLF Descriptor CRLF "." CRLF +``` + +This message informs the server about a new descriptor. If Purpose is +specified, it must be either "general", "controller", or "bridge", +else we return a 552 error. The default is "general". + +If Cache is specified, it must be either "no" or "yes", else we +return a 552 error. If Cache is not specified, Tor will decide for +itself whether it wants to cache the descriptor, and controllers +must not rely on its choice. + +The descriptor, when parsed, must contain a number of well-specified +fields, including fields for its nickname and identity. + +If there is an error in parsing the descriptor, the server must send a +"554 Invalid descriptor" reply. If the descriptor is well-formed but +the server chooses not to add it, it must reply with a 251 message +whose body explains why the server was not added. If the descriptor +is added, Tor replies with "250 OK". + + +## REDIRECTSTREAM + +Sent from the client to the server. The syntax is: + +"REDIRECTSTREAM" SP StreamID SP Address [SP Port] CRLF + +Tells the server to change the exit address on the specified stream. If +Port is specified, changes the destination port as well. No remapping +is performed on the new provided address. + +To be sure that the modified address will be used, this event must be sent +after a new stream event is received, and before attaching this stream to +a circuit. + +Tor replies with "250 OK" on success. + + +## CLOSESTREAM + +Sent from the client to the server. The syntax is: + +"CLOSESTREAM" SP StreamID SP Reason *(SP Flag) CRLF + +Tells the server to close the specified stream. The reason should be one +of the Tor RELAY_END reasons given in tor-spec.txt, as a decimal. Flags is +not used currently; Tor servers SHOULD ignore unrecognized flags. Tor may +hold the stream open for a while to flush any data that is pending. + +Tor replies with "250 OK" on success, or a 512 if there aren't enough +arguments, or a 552 if it doesn't recognize the StreamID or reason. + + +## CLOSECIRCUIT + +The syntax is: + +```text + "CLOSECIRCUIT" SP CircuitID *(SP Flag) CRLF + Flag = "IfUnused" +``` + +Tells the server to close the specified circuit. If "IfUnused" is +provided, do not close the circuit unless it is unused. + +Other flags may be defined in the future; Tor SHOULD ignore unrecognized +flags. + +Tor replies with "250 OK" on success, or a 512 if there aren't enough +arguments, or a 552 if it doesn't recognize the CircuitID. + + +## QUIT + +Tells the server to hang up on this controller connection. This command +can be used before authenticating. + + +## USEFEATURE + +Adding additional features to the control protocol sometimes will break +backwards compatibility. Initially such features are added into Tor and +disabled by default. USEFEATURE can enable these additional features. + +The syntax is: + +```text + "USEFEATURE" *(SP FeatureName) CRLF + FeatureName = 1*(ALPHA / DIGIT / "_" / "-") + + Feature names are case-insensitive. +``` + +Once enabled, a feature stays enabled for the duration of the connection +to the controller. A new connection to the controller must be opened to +disable an enabled feature. + +Features are a forward-compatibility mechanism; each feature will eventually +become a standard part of the control protocol. Once a feature becomes part +of the protocol, it is always-on. Each feature documents the version it was +introduced as a feature and the version in which it became part of the +protocol. + +Tor will ignore a request to use any feature that is always-on. Tor will give +a 552 error in response to an unrecognized feature. + +EXTENDED_EVENTS + +```text + Same as passing 'EXTENDED' to SETEVENTS; this is the preferred way to + request the extended event syntax. + + This feature was first introduced in 0.1.2.3-alpha. It is always-on + and part of the protocol in Tor 0.2.2.1-alpha and later. + + VERBOSE_NAMES + + Replaces ServerID with LongName in events and GETINFO results. LongName + provides a Fingerprint for all routers, an indication of Named status, + and a Nickname if one is known. LongName is strictly more informative + than ServerID, which only provides either a Fingerprint or a Nickname. + + This feature was first introduced in 0.1.2.2-alpha. It is always-on and + part of the protocol in Tor 0.2.2.1-alpha and later. +``` + + +## RESOLVE + +The syntax is + +```text + "RESOLVE" *Option *Address CRLF + Option = "mode=reverse" + Address = a hostname or IPv4 address +``` + +This command launches a remote hostname lookup request for every specified +request (or reverse lookup if "mode=reverse" is specified). Note that the +request is done in the background: to see the answers, your controller will +need to listen for ADDRMAP events; see 4.1.7 below. + +[Added in Tor 0.2.0.3-alpha] + + +## PROTOCOLINFO + +The syntax is: + +"PROTOCOLINFO" *(SP PIVERSION) CRLF + +The server reply format is: + +"250-PROTOCOLINFO" SP PIVERSION CRLF *InfoLine "250 OK" CRLF + +InfoLine = AuthLine / VersionLine / OtherLine + +```text + AuthLine = "250-AUTH" SP "METHODS=" AuthMethod *("," AuthMethod) + *(SP "COOKIEFILE=" AuthCookieFile) CRLF + VersionLine = "250-VERSION" SP "Tor=" TorVersion OptArguments CRLF + + AuthMethod = + "NULL" / ; No authentication is required + "HASHEDPASSWORD" / ; A controller must supply the original password + "COOKIE" / ; ... or supply the contents of a cookie file + "SAFECOOKIE" ; ... or prove knowledge of a cookie file's contents + + AuthCookieFile = QuotedString + TorVersion = QuotedString + + OtherLine = "250-" Keyword OptArguments CRLF + + PIVERSION: 1*DIGIT +``` + +This command tells the controller what kinds of authentication are +supported. + +Tor MAY give its InfoLines in any order; controllers MUST ignore InfoLines +with keywords they do not recognize. Controllers MUST ignore extraneous +data on any InfoLine. + +PIVERSION is there in case we drastically change the syntax one day. For +now it should always be "1". Controllers MAY provide a list of the +protocolinfo versions they support; Tor MAY select a version that the +controller does not support. + +AuthMethod is used to specify one or more control authentication +methods that Tor currently accepts. + +AuthCookieFile specifies the absolute path and filename of the +authentication cookie that Tor is expecting and is provided iff the +METHODS field contains the method "COOKIE" and/or "SAFECOOKIE". +Controllers MUST handle escape sequences inside this string. + +All authentication cookies are 32 bytes long. Controllers MUST NOT +use the contents of a non-32-byte-long file as an authentication +cookie. + +If the METHODS field contains the method "SAFECOOKIE", every +AuthCookieFile must contain the same authentication cookie. + +The COOKIE authentication method exposes the user running a +controller to an unintended information disclosure attack whenever +the controller has greater filesystem read access than the process +that it has connected to. (Note that a controller may connect to a +process other than Tor.) It is almost never safe to use, even if +the controller's user has explicitly specified which filename to +read an authentication cookie from. For this reason, the COOKIE +authentication method has been deprecated and will be removed from +a future version of Tor. + +The VERSION line contains the Tor version. + +[Unlike other commands besides AUTHENTICATE, PROTOCOLINFO may be used (but +only once!) before AUTHENTICATE.] + +[PROTOCOLINFO was not supported before Tor 0.2.0.5-alpha.] + + +## LOADCONF + +The syntax is: + +"+LOADCONF" CRLF ConfigText CRLF "." CRLF + +This command allows a controller to upload the text of a config file +to Tor over the control port. This config file is then loaded as if +it had been read from disk. + +[LOADCONF was added in Tor 0.2.1.1-alpha.] + + +## TAKEOWNERSHIP + +The syntax is: + +"TAKEOWNERSHIP" CRLF + +This command instructs Tor to shut down when this control +connection is closed. This command affects each control connection +that sends it independently; if multiple control connections send +the TAKEOWNERSHIP command to a Tor instance, Tor will shut down when +any of those connections closes. + +(As of Tor 0.2.5.2-alpha, Tor does not wait a while for circuits to +close when shutting down because of an exiting controller. If you +want to ensure a clean shutdown--and you should!--then send "SIGNAL +SHUTDOWN" and wait for the Tor process to close.) + +This command is intended to be used with the +__OwningControllerProcess configuration option. A controller that +starts a Tor process which the user cannot easily control or stop +should 'own' that Tor process: + +```text + * When starting Tor, the controller should specify its PID in an + __OwningControllerProcess on Tor's command line. This will + cause Tor to poll for the existence of a process with that PID, + and exit if it does not find such a process. (This is not a + completely reliable way to detect whether the 'owning + controller' is still running, but it should work well enough in + most cases.) + + * Once the controller has connected to Tor's control port, it + should send the TAKEOWNERSHIP command along its control + connection. At this point, *both* the TAKEOWNERSHIP command and + the __OwningControllerProcess option are in effect: Tor will + exit when the control connection ends *and* Tor will exit if it + detects that there is no process with the PID specified in the + __OwningControllerProcess option. + + * After the controller has sent the TAKEOWNERSHIP command, it + should send "RESETCONF __OwningControllerProcess" along its + control connection. This will cause Tor to stop polling for the + existence of a process with its owning controller's PID; Tor + will still exit when the control connection ends. + + [TAKEOWNERSHIP was added in Tor 0.2.2.28-beta.] +``` + + +## AUTHCHALLENGE + +The syntax is: + +```text + "AUTHCHALLENGE" SP "SAFECOOKIE" + SP ClientNonce + CRLF + + ClientNonce = 2*HEXDIG / QuotedString +``` + +This command is used to begin the authentication routine for the +SAFECOOKIE method of authentication. + +If the server accepts the command, the server reply format is: + +```text + "250 AUTHCHALLENGE" + SP "SERVERHASH=" ServerHash + SP "SERVERNONCE=" ServerNonce + CRLF + + ServerHash = 64*64HEXDIG + ServerNonce = 64*64HEXDIG +``` + +The ClientNonce, ServerHash, and ServerNonce values are +encoded/decoded in the same way as the argument passed to the +AUTHENTICATE command. ServerNonce MUST be 32 bytes long. + +ServerHash is computed as: + +```text + HMAC-SHA256("Tor safe cookie authentication server-to-controller hash", + CookieString | ClientNonce | ServerNonce) + + (with the HMAC key as its first argument) +``` + +After a controller sends a successful AUTHCHALLENGE command, the +next command sent on the connection must be an AUTHENTICATE command, +and the only authentication string which that AUTHENTICATE command +will accept is: + +```text + HMAC-SHA256("Tor safe cookie authentication controller-to-server hash", + CookieString | ClientNonce | ServerNonce) +``` + +[Unlike other commands besides AUTHENTICATE, AUTHCHALLENGE may be +used (but only once!) before AUTHENTICATE.] + +[AUTHCHALLENGE was added in Tor 0.2.3.13-alpha.] + + +## DROPGUARDS + +The syntax is: + +"DROPGUARDS" CRLF + +Tells the server to drop all guard nodes. Do not invoke this command +lightly; it can increase vulnerability to tracking attacks over time. + +Tor replies with "250 OK" on success. + +[DROPGUARDS was added in Tor 0.2.5.2-alpha.] + + +## HSFETCH + +The syntax is: + +```text + "HSFETCH" SP (HSAddress / "v" Version "-" DescId) + *[SP "SERVER=" Server] CRLF + + HSAddress = 16*Base32Character / 56*Base32Character + Version = "2" / "3" + DescId = 32*Base32Character + Server = LongName +``` + +This command launches hidden service descriptor fetch(es) for the given +HSAddress or DescId. + +HSAddress can be version 2 or version 3 addresses. DescIDs can only be +version 2 IDs. Version 2 addresses consist of 16*Base32Character and +version 3 addresses consist of 56*Base32Character. + +If a DescId is specified, at least one Server MUST also be provided, +otherwise a 512 error is returned. If no DescId and Server(s) are specified, +it behaves like a normal Tor client descriptor fetch. If one or more +Server are given, they are used instead triggering a fetch on each of them +in parallel. + +The caching behavior when fetching a descriptor using this command is +identical to normal Tor client behavior. + +Details on how to compute a descriptor id (DescId) can be found in +rend-spec.txt section 1.3. + +If any values are unrecognized, a 513 error is returned and the command is +stopped. On success, Tor replies "250 OK" then Tor MUST eventually follow +this with both a HS_DESC and HS_DESC_CONTENT events with the results. If +SERVER is specified then events are emitted for each location. + +Examples are: + +```text + C: HSFETCH v2-gezdgnbvgy3tqolbmjrwizlgm5ugs2tl + SERVER=9695DFC35FFEB861329B9F1AB04C46397020CE31 + S: 250 OK + + C: HSFETCH ajkhdsfuygaesfaa + S: 250 OK + + C: HSFETCH vww6ybal4bd7szmgncyruucpgfkqahzddi37ktceo3ah7ngmcopnpyyd + S: 250 OK +``` + +[HSFETCH was added in Tor 0.2.7.1-alpha] +[HS v3 support added 0.4.1.1-alpha] + + +## ADD_ONION + +The syntax is: + +```text + "ADD_ONION" SP KeyType ":" KeyBlob + [SP "Flags=" Flag *("," Flag)] + [SP "MaxStreams=" NumStreams] + 1*(SP "Port=" VirtPort ["," Target]) + *(SP "ClientAuth=" ClientName [":" ClientBlob]) CRLF + *(SP "ClientAuthV3=" V3Key) CRLF + + KeyType = + "NEW" / ; The server should generate a key of algorithm KeyBlob + "RSA1024" / ; The server should use the 1024 bit RSA key provided + in as KeyBlob (v2). + "ED25519-V3"; The server should use the ed25519 v3 key provided in as + KeyBlob (v3). + + KeyBlob = + "BEST" / ; The server should generate a key using the "best" + supported algorithm (KeyType == "NEW"). + [As of 0.4.2.3-alpha, ED25519-V3 is used] + "RSA1024" / ; The server should generate a 1024 bit RSA key + (KeyType == "NEW") (v2). + "ED25519-V3"; The server should generate an ed25519 private key + (KeyType == "NEW") (v3). + String ; A serialized private key (without whitespace) + + Flag = + "DiscardPK" / ; The server should not include the newly generated + private key as part of the response. + "Detach" / ; Do not associate the newly created Onion Service + to the current control connection. + "BasicAuth" / ; Client authorization is required using the "basic" + method (v2 only). + "V3Auth" / ; Version 3 client authorization is required (v3 only). + + "NonAnonymous" /; Add a non-anonymous Single Onion Service. Tor + checks this flag matches its configured hidden + service anonymity mode. + "MaxStreamsCloseCircuit"; Close the circuit is the maximum streams + allowed is reached. + + NumStreams = A value between 0 and 65535 which is used as the maximum + streams that can be attached on a rendezvous circuit. Setting + it to 0 means unlimited which is also the default behavior. + + VirtPort = The virtual TCP Port for the Onion Service (As in the + HiddenServicePort "VIRTPORT" argument). + + Target = The (optional) target for the given VirtPort (As in the + optional HiddenServicePort "TARGET" argument). + + ClientName = An identifier 1 to 16 characters long, using only + characters in A-Za-z0-9+-_ (no spaces) (v2 only). + + ClientBlob = Authorization data for the client, in an opaque format + specific to the authorization method (v2 only). + + V3Key = The client's base32-encoded x25519 public key, using only the key + part of rend-spec-v3.txt section G.1.2 (v3 only). + + The server reply format is: + + "250-ServiceID=" ServiceID CRLF + ["250-PrivateKey=" KeyType ":" KeyBlob CRLF] + *("250-ClientAuth=" ClientName ":" ClientBlob CRLF) + "250 OK" CRLF + + ServiceID = The Onion Service address without the trailing ".onion" + suffix +``` + +Tells the server to create a new Onion ("Hidden") Service, with the +specified private key and algorithm. If a KeyType of "NEW" is selected, +the server will generate a new keypair using the selected algorithm. +The "Port" argument's VirtPort and Target values have identical +semantics to the corresponding HiddenServicePort configuration values. + +The server response will only include a private key if the server was +requested to generate a new keypair, and also the "DiscardPK" flag was +not specified. (Note that if "DiscardPK" flag is specified, there is no +way to recreate the generated keypair and the corresponding Onion +Service at a later date). + +If client authorization is enabled using the "BasicAuth" flag (which is v2 +only), the service will not be accessible to clients without valid +authorization data (configured with the "HidServAuth" option). The list of +authorized clients is specified with one or more "ClientAuth" parameters. +If "ClientBlob" is not specified for a client, a new credential will be +randomly generated and returned. + +Tor instances can either be in anonymous hidden service mode, or +non-anonymous single onion service mode. All hidden services on the same +tor instance have the same anonymity. To guard against unexpected loss +of anonymity, Tor checks that the ADD_ONION "NonAnonymous" flag matches +the current hidden service anonymity mode. The hidden service anonymity +mode is configured using the Tor options HiddenServiceSingleHopMode and +HiddenServiceNonAnonymousMode. If both these options are 1, the +"NonAnonymous" flag must be provided to ADD_ONION. If both these options +are 0 (the Tor default), the flag must NOT be provided. + +Once created the new Onion Service will remain active until either the +Onion Service is removed via "DEL_ONION", the server terminates, or the +control connection that originated the "ADD_ONION" command is closed. +It is possible to override disabling the Onion Service on control +connection close by specifying the "Detach" flag. + +It is the Onion Service server application's responsibility to close +existing client connections if desired after the Onion Service is +removed. + +(The KeyBlob format is left intentionally opaque, however for "RSA1024" +keys it is currently the Base64 encoded DER representation of a PKCS#1 +RSAPrivateKey, with all newlines removed. For a "ED25519-V3" key is +the Base64 encoding of the concatenation of the 32-byte ed25519 secret +scalar in little-endian and the 32-byte ed25519 PRF secret.) + +[Note: The ED25519-V3 format is not the same as, e.g., SUPERCOP +ed25519/ref, which stores the concatenation of the 32-byte ed25519 +hash seed concatenated with the 32-byte public key, and which derives +the secret scalar and PRF secret by expanding the hash seed with +SHA-512. Our key blinding scheme is incompatible with storing +private keys as seeds, so we store the secret scalar alongside the +PRF secret, and just pay the cost of recomputing the public key when +importing an ED25519-V3 key.] + +Examples: + +```text + C: ADD_ONION NEW:BEST Flags=DiscardPK Port=80 + S: 250-ServiceID=exampleoniont2pqglbny66wpovyvao3ylc23eileodtevc4b75ikpad + S: 250 OK + + C: ADD_ONION RSA1024:[Blob Redacted] Port=80,192.168.1.1:8080 + S: 250-ServiceID=sampleonion12456 + S: 250 OK + + C: ADD_ONION NEW:BEST Port=22 Port=80,8080 + S: 250-ServiceID=sampleonion4t2pqglbny66wpovyvao3ylc23eileodtevc4b75ikpad + S: 250-PrivateKey=ED25519-V3:[Blob Redacted] + S: 250 OK + + C: ADD_ONION NEW:RSA1024 Flags=DiscardPK,BasicAuth Port=22 + ClientAuth=alice:[Blob Redacted] ClientAuth=bob + S: 250-ServiceID=testonion1234567 + S: 250-ClientAuth=bob:[Blob Redacted] + S: 250 OK + + C: ADD_ONION NEW:ED25519-V3 ClientAuthV3=[Blob Redacted] Port=22 + S: 250-ServiceID=n35etu3yjxrqjpntmfziom5sjwspoydchmelc4xleoy4jk2u4lziz2yd + S: 250-ClientAuthV3=[Blob Redacted] + S: 250 OK + + Examples with Tor in anonymous onion service mode: + + C: ADD_ONION NEW:BEST Flags=DiscardPK Port=22 + S: 250-ServiceID=exampleoniont2pqglbny66wpovyvao3ylc23eileodtevc4b75ikpad + S: 250 OK + + C: ADD_ONION NEW:BEST Flags=DiscardPK,NonAnonymous Port=22 + S: 512 Tor is in anonymous hidden service mode + + Examples with Tor in non-anonymous onion service mode: + + C: ADD_ONION NEW:BEST Flags=DiscardPK Port=22 + S: 512 Tor is in non-anonymous hidden service mode + + C: ADD_ONION NEW:BEST Flags=DiscardPK,NonAnonymous Port=22 + S: 250-ServiceID=exampleoniont2pqglbny66wpovyvao3ylc23eileodtevc4b75ikpad + S: 250 OK +``` + +[ADD_ONION was added in Tor 0.2.7.1-alpha.] +[MaxStreams and MaxStreamsCloseCircuit were added in Tor 0.2.7.2-alpha] +[ClientAuth was added in Tor 0.2.9.1-alpha. It is v2 only.] +[NonAnonymous was added in Tor 0.2.9.3-alpha.] +[HS v3 support added 0.3.3.1-alpha] +[ClientV3Auth support added 0.4.6.1-alpha] + + +## DEL_ONION + +The syntax is: + +"DEL_ONION" SP ServiceID CRLF + +```text + ServiceID = The Onion Service address without the trailing ".onion" + suffix +``` + +Tells the server to remove an Onion ("Hidden") Service, that was +previously created via an "ADD_ONION" command. It is only possible to +remove Onion Services that were created on the same control connection +as the "DEL_ONION" command, and those that belong to no control +connection in particular (The "Detach" flag was specified at creation). + +If the ServiceID is invalid, or is neither owned by the current control +connection nor a detached Onion Service, the server will return a 552. + +It is the Onion Service server application's responsibility to close +existing client connections if desired after the Onion Service has been +removed via "DEL_ONION". + +Tor replies with "250 OK" on success, or a 512 if there are an invalid +number of arguments, or a 552 if it doesn't recognize the ServiceID. + +[DEL_ONION was added in Tor 0.2.7.1-alpha.] +[HS v3 support added 0.3.3.1-alpha] + + +## HSPOST + +The syntax is: + +```text + "+HSPOST" *[SP "SERVER=" Server] [SP "HSADDRESS=" HSAddress] + CRLF Descriptor CRLF "." CRLF + + Server = LongName + HSAddress = 56*Base32Character + Descriptor = The text of the descriptor formatted as specified + in rend-spec.txt section 1.3. +``` + +The "HSAddress" key is optional and only applies for v3 descriptors. A 513 +error is returned if used with v2. + +This command launches a hidden service descriptor upload to the specified +HSDirs. If one or more Server arguments are provided, an upload is triggered +on each of them in parallel. If no Server options are provided, it behaves +like a normal HS descriptor upload and will upload to the set of responsible +HS directories. + +If any value is unrecognized, a 552 error is returned and the command is +stopped. If there is an error in parsing the descriptor, the server +must send a "554 Invalid descriptor" reply. + +On success, Tor replies "250 OK" then Tor MUST eventually follow +this with a HS_DESC event with the result for each upload location. + +Examples are: + +```text + C: +HSPOST SERVER=9695DFC35FFEB861329B9F1AB04C46397020CE31 + [DESCRIPTOR] + . + S: 250 OK + + [HSPOST was added in Tor 0.2.7.1-alpha] +``` + + +## ONION_CLIENT_AUTH_ADD + +The syntax is: + +```text + "ONION_CLIENT_AUTH_ADD" SP HSAddress + SP KeyType ":" PrivateKeyBlob + [SP "ClientName=" Nickname] + [SP "Flags=" TYPE] CRLF + + HSAddress = 56*Base32Character + KeyType = "x25519" is the only one supported right now + PrivateKeyBlob = base64 encoding of x25519 key +``` + +Tells the connected Tor to add client-side v3 client auth credentials for the +onion service with "HSAddress". The "PrivateKeyBlob" is the x25519 private +key that should be used for this client, and "Nickname" is an optional +nickname for the client. + +FLAGS is a comma-separated tuple of flags for this new client. For now, the +currently supported flags are: + +```text + "Permanent" - This client's credentials should be stored in the filesystem. + If this is not set, the client's credentials are ephemeral + and stored in memory. +``` + +If client auth credentials already existed for this service, replace them +with the new ones. + +If Tor has cached onion service descriptors that it has been unable to +decrypt in the past (due to lack of client auth credentials), attempt to +decrypt those descriptors as soon as this command succeeds. + +On success, "250 OK" is returned. Otherwise, the following error codes exist: + +```text + 251 - Client auth credentials for this onion service already existed and replaced. + 252 - Added client auth credentials and successfully decrypted a cached descriptor. + 451 - We reached authorized client capacity + 512 - Syntax error in "HSAddress", or "PrivateKeyBlob" or "Nickname" + 551 - Client with with this "Nickname" already exists + 552 - Unrecognized KeyType + + [ONION_CLIENT_AUTH_ADD was added in Tor 0.4.3.1-alpha] +``` + + +## ONION_CLIENT_AUTH_REMOVE + +The syntax is: + +"ONION_CLIENT_AUTH_REMOVE" SP HSAddress + +KeyType = "x25519" is the only one supported right now + +Tells the connected Tor to remove the client-side v3 client auth credentials +for the onion service with "HSAddress". + +On success "250 OK" is returned. Otherwise, the following error codes exist: + +```text + 512 - Syntax error in "HSAddress". + 251 - Client credentials for "HSAddress" did not exist. + + [ONION_CLIENT_AUTH_REMOVE was added in Tor 0.4.3.1-alpha] +``` + + +## ONION_CLIENT_AUTH_VIEW + +The syntax is: + +"ONION_CLIENT_AUTH_VIEW" [SP HSAddress] CRLF + +Tells the connected Tor to list all the stored client-side v3 client auth +credentials for "HSAddress". If no "HSAddress" is provided, list all the +stored client-side v3 client auth credentials. + +The server reply format is: + +```text + "250-ONION_CLIENT_AUTH_VIEW" [SP HSAddress] CRLF + *("250-CLIENT" SP HSAddress SP KeyType ":" PrivateKeyBlob + [SP "ClientName=" Nickname] + [SP "Flags=" FLAGS] CRLF) + "250 OK" CRLF + + HSAddress = The onion address under which this credential is stored + KeyType = "x25519" is the only one supported right now + PrivateKeyBlob = base64 encoding of x25519 key +``` + +"Nickname" is an optional nickname for this client, which can be set either +through the ONION_CLIENT_AUTH_ADD command, or it's the filename of this +client if the credentials are stored in the filesystem. + +FLAGS is a comma-separated field of flags for this client, the currently +supported flags are: + +"Permanent" - This client's credentials are stored in the filesystem. + +On success "250 OK" is returned. Otherwise, the following error codes exist: + +512 - Syntax error in "HSAddress". + +[ONION_CLIENT_AUTH_VIEW was added in Tor 0.4.3.1-alpha] + + +## DROPOWNERSHIP + +The syntax is: + +"DROPOWNERSHIP" CRLF + +This command instructs Tor to relinquish ownership of its control +connection. As such tor will not shut down when this control +connection is closed. + +This method is idempotent. If the control connection does not +already have ownership this method returns successfully, and +does nothing. + +The controller can call TAKEOWNERSHIP again to re-establish +ownership. + +[DROPOWNERSHIP was added in Tor 0.4.0.0-alpha] + + +## DROPTIMEOUTS + +```text + The syntax is: + "DROPTIMEOUTS" CRLF +``` + +Tells the server to drop all circuit build times. Do not invoke this command +lightly; it can increase vulnerability to tracking attacks over time. + +Tor replies with "250 OK" on success. Tor also emits the BUILDTIMEOUT_SET +RESET event right after this "250 OK". + +[DROPTIMEOUTS was added in Tor 0.4.5.0-alpha.] + diff --git a/spec/control-spec/implementation-notes.md b/spec/control-spec/implementation-notes.md new file mode 100644 index 0000000..1a610bb --- /dev/null +++ b/spec/control-spec/implementation-notes.md @@ -0,0 +1,709 @@ + +# Implementation notes + + +## Authentication + +If the control port is open and no authentication operation is enabled, Tor +trusts any local user that connects to the control port. This is generally +a poor idea. + +If the 'CookieAuthentication' option is true, Tor writes a "magic +cookie" file named "control_auth_cookie" into its data directory (or +to another file specified in the 'CookieAuthFile' option). To +authenticate, the controller must demonstrate that it can read the +contents of the cookie file: + +* Current versions of Tor support cookie authentication + +```text + using the "COOKIE" authentication method: the controller sends the + contents of the cookie file, encoded in hexadecimal. This + authentication method exposes the user running a controller to an + unintended information disclosure attack whenever the controller + has greater filesystem read access than the process that it has + connected to. (Note that a controller may connect to a process + other than Tor.) It is almost never safe to use, even if the + controller's user has explicitly specified which filename to read + an authentication cookie from. For this reason, the COOKIE + authentication method has been deprecated and will be removed from + Tor before some future version of Tor. + + * 0.2.2.x versions of Tor starting with 0.2.2.36, and all versions of + + Tor after 0.2.3.12-alpha, support cookie authentication using the + "SAFECOOKIE" authentication method, which discloses much less + information about the contents of the cookie file. +``` + +If the 'HashedControlPassword' option is set, it must contain the salted +hash of a secret password. The salted hash is computed according to the +S2K algorithm in RFC 2440 (OpenPGP), and prefixed with the s2k specifier. +This is then encoded in hexadecimal, prefixed by the indicator sequence +"16:". Thus, for example, the password 'foo' could encode to: + +```text + 16:660537E3E1CD49996044A3BF558097A981F539FEA2F9DA662B4626C1C2 + ++++++++++++++++**^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + salt hashed value + indicator + + You can generate the salt of a password by calling + + 'tor --hash-password ' +``` + +or by using the example code in the Python and Java controller libraries. +To authenticate under this scheme, the controller sends Tor the original +secret that was used to generate the password, either as a quoted string +or encoded in hexadecimal. + + +## Don't let the buffer get too big. + +With old versions of Tor (before 0.2.0.16-alpha), if you ask for +lots of events, and 16MB of them queue up on the buffer, the Tor +process will close the socket. + +Newer Tor versions do not have this 16 MB buffer limit. However, +if you leave huge numbers of events unread, Tor may still run out +of memory, so you should still be careful about buffer size. + + +## Backward compatibility with v0 control protocol. + +The 'version 0' control protocol was replaced in Tor 0.1.1.x. Support +was removed in Tor 0.2.0.x. Every non-obsolete version of Tor now +supports the version 1 control protocol. + +For backward compatibility with the "version 0" control protocol, +Tor used to check whether the third octet of the first command is zero. +(If it was, Tor assumed that version 0 is in use.) + +This compatibility was removed in Tor 0.1.2.16 and 0.2.0.4-alpha. + + +## Tor config options for use by controllers + +Tor provides a few special configuration options for use by controllers. +These options are not saved to disk by SAVECONF. Most can be set and +examined by the SETCONF and GETCONF commands, but some (noted below) can +only be given in a torrc file or on the command line. + +Generally, these options make Tor unusable by disabling a portion of Tor's +normal operations. Unless a controller provides replacement functionality +to fill this gap, Tor will not correctly handle user requests. + +__AllDirActionsPrivate + +```text + If true, Tor will try to launch all directory operations through + anonymous connections. (Ordinarily, Tor only tries to anonymize + requests related to hidden services.) This option will slow down + directory access, and may stop Tor from working entirely if it does not + yet have enough directory information to build circuits. + + (Boolean. Default: "0".) + + __DisablePredictedCircuits + + If true, Tor will not launch preemptive "general-purpose" circuits for + streams to attach to. (It will still launch circuits for testing and + for hidden services.) + + (Boolean. Default: "0".) + + __LeaveStreamsUnattached + + If true, Tor will not automatically attach new streams to circuits; + instead, the controller must attach them with ATTACHSTREAM. If the + controller does not attach the streams, their data will never be routed. + + (Boolean. Default: "0".) + + __HashedControlSessionPassword + + As HashedControlPassword, but is not saved to the torrc file by + SAVECONF. Added in Tor 0.2.0.20-rc. + + __ReloadTorrcOnSIGHUP + + If this option is true (the default), we reload the torrc from disk + every time we get a SIGHUP (from the controller or via a signal). + Otherwise, we don't. This option exists so that controllers can keep + their options from getting overwritten when a user sends Tor a HUP for + some other reason (for example, to rotate the logs). + + (Boolean. Default: "1") + + __OwningControllerProcess + + If this option is set to a process ID, Tor will periodically check + whether a process with the specified PID exists, and exit if one + does not. Added in Tor 0.2.2.28-beta. This option's intended use + is documented in section 3.23 with the related TAKEOWNERSHIP + command. + + Note that this option can only specify a single process ID, unlike + the TAKEOWNERSHIP command which can be sent along multiple control + connections. + + (String. Default: unset.) + + __OwningControllerFD + + If this option is a valid socket, Tor will start with an open control + connection on this socket. Added in Tor 0.3.3.1-alpha. + + This socket will be an owning controller, as if it had already called + TAKEOWNERSHIP. It will be automatically authenticated. This option + should only be used by other programs that are starting Tor. + + This option cannot be changed via SETCONF; it must be set in a torrc or + via the command line. + + (Integer. Default: -1.) + + __DisableSignalHandlers + + If this option is set to true during startup, then Tor will not install + any signal handlers to watch for POSIX signals. The SIGNAL controller + command will still work. + + This option is meant for embedding Tor inside another process, when + the controlling process would rather handle signals on its own. + + This option cannot be changed via SETCONF; it must be set in a torrc or + via the command line. + + (Boolean. Default: 0.) +``` + + +## Phases from the Bootstrap status event. + +```text + [For the bootstrap phases reported by Tor prior to 0.4.0.x, see + Section 5.6.] +``` + +This section describes the various bootstrap phases currently reported +by Tor. Controllers should not assume that the percentages and tags +listed here will continue to match up, or even that the tags will stay +in the same order. Some phases might also be skipped (not reported) +if the associated bootstrap step is already complete, or if the phase +no longer is necessary. Only "starting" and "done" are guaranteed to +exist in all future versions. + +Current Tor versions enter these phases in order, monotonically. +Future Tors MAY revisit earlier phases, for example, if the network +fails. + + +### Overview of Bootstrap reporting. + +Bootstrap phases can be viewed as belonging to one of three stages: + +1. Initial connection to a Tor relay or bridge +2. Obtaining directory information +3. Building an application circuit + +Tor doesn't specifically enter Stage 1; that is a side effect of +other actions that Tor is taking. Tor could be making a connection +to a fallback directory server, or it could be making a connection +to a guard candidate. Either one counts as Stage 1 for the purposes +of bootstrap reporting. + +Stage 2 might involve Tor contacting directory servers, or it might +involve reading cached directory information from a previous +session. Large parts of Stage 2 might be skipped if there is already +enough cached directory information to build circuits. Tor will +defer reporting progress in Stage 2 until Stage 1 is complete. + +Tor defers this reporting because Tor can already have enough +directory information to build circuits, yet not be able to connect +to a relay. Without that deferral, a user might misleadingly see Tor +stuck at a large amount of progress when something as fundamental as +making a TCP connection to any relay is failing. + +Tor also doesn't specifically enter Stage 3; that is a side effect +of Tor building circuits for some purpose or other. In a typical +client, Tor builds predicted circuits to provide lower latency for +application connection requests. In Stage 3, Tor might make new +connections to relays or bridges that it did not connect to in Stage +1. + + +### Phases in Bootstrap Stage 1. + +Phase 0: +tag=starting summary="Starting" + +Tor starts out in this phase. + +```text + Phase 1: + tag=conn_pt summary="Connecting to pluggable transport" + [This phase is new in 0.4.0.x] +``` + +Tor is making a TCP connection to the transport plugin for a +pluggable transport. Tor will use this pluggable transport to make +its first connection to a bridge. + +```text + Phase 2: + tag=conn_done_pt summary="Connected to pluggable transport" + [New in 0.4.0.x] +``` + +Tor has completed its TCP connection to the transport plugin for the +pluggable transport. + +```text + Phase 3: + tag=conn_proxy summary="Connecting to proxy" + [New in 0.4.0.x] +``` + +Tor is making a TCP connection to a proxy to make its first +connection to a relay or bridge. + +```text + Phase 4: + tag=conn_done_proxy summary="Connected to proxy" + [New in 0.4.0.x] +``` + +Tor has completed its TCP connection to a proxy to make its first +connection to a relay or bridge. + +```text + Phase 5: + tag=conn summary="Connecting to a relay" + [New in 0.4.0.x; prior versions of Tor had a "conn_dir" phase that + sometimes but not always corresponded to connecting to a directory server] +``` + +Tor is making its first connection to a relay. This might be through +a pluggable transport or proxy connection that Tor has already +established. + +```text + Phase 10: + tag=conn_done summary="Connected to a relay" + [New in 0.4.0.x] + + Tor has completed its first connection to a relay. + + Phase 14: + tag=handshake summary="Handshaking with a relay" + [New in 0.4.0.x; prior versions of Tor had a "handshake_dir" phase] + + Tor is in the process of doing a TLS handshake with a relay. + + Phase 15: + tag=handshake_done summary="Handshake with a relay done" + [New in 0.4.0.x] + + Tor has completed its TLS handshake with a relay. +``` + + +### Phases in Bootstrap Stage 2. + +```text + Phase 20: + tag=onehop_create summary="Establishing an encrypted directory connection" + [prior to 0.4.0.x, this was numbered 15] +``` + +Once TLS is finished with a relay, Tor will send a CREATE_FAST cell +to establish a one-hop circuit for retrieving directory information. +It will remain in this phase until it receives the CREATED_FAST cell +back, indicating that the circuit is ready. + +```text + Phase 25: + tag=requesting_status summary="Asking for networkstatus consensus" + [prior to 0.4.0.x, this was numbered 20] +``` + +Once we've finished our one-hop circuit, we will start a new stream +for fetching the networkstatus consensus. We'll stay in this phase +until we get the 'connected' relay cell back, indicating that we've +established a directory connection. + +```text + Phase 30: + tag=loading_status summary="Loading networkstatus consensus" + [prior to 0.4.0.x, this was numbered 25] +``` + +Once we've established a directory connection, we will start fetching +the networkstatus consensus document. This could take a while; this +phase is a good opportunity for using the "progress" keyword to indicate +partial progress. + +This phase could stall if the directory server we picked doesn't +have a copy of the networkstatus consensus so we have to ask another, +or it does give us a copy but we don't find it valid. + +Phase 40: +tag=loading_keys summary="Loading authority key certs" + +Sometimes when we've finished loading the networkstatus consensus, +we find that we don't have all the authority key certificates for the +keys that signed the consensus. At that point we put the consensus we +fetched on hold and fetch the keys so we can verify the signatures. + +Phase 45 +tag=requesting_descriptors summary="Asking for relay descriptors" + +Once we have a valid networkstatus consensus and we've checked all +its signatures, we start asking for relay descriptors. We stay in this +phase until we have received a 'connected' relay cell in response to +a request for descriptors. + +```text + [Some versions of Tor (starting with 0.2.6.2-alpha but before + 0.4.0.x): Tor could report having internal paths only; see Section + 5.6] +``` + +Phase 50: +tag=loading_descriptors summary="Loading relay descriptors" + +We will ask for relay descriptors from several different locations, +so this step will probably make up the bulk of the bootstrapping, +especially for users with slow connections. We stay in this phase until +we have descriptors for a significant fraction of the usable relays +listed in the networkstatus consensus (this can be between 25% and 95% +depending on Tor's configuration and network consensus parameters). +This phase is also a good opportunity to use the "progress" keyword to +indicate partial steps. + +```text + [Some versions of Tor (starting with 0.2.6.2-alpha but before + 0.4.0.x): Tor could report having internal paths only; see Section + 5.6] + + Phase 75: + tag=enough_dirinfo summary="Loaded enough directory info to build + circuits" + [New in 0.4.0.x; previously, Tor would misleadingly report the + "conn_or" tag once it had enough directory info.] +``` + + +### Phases in Bootstrap Stage 3. + +```text + Phase 76: + tag=ap_conn_pt summary="Connecting to pluggable transport to build + circuits" + [New in 0.4.0.x] +``` + +This is similar to conn_pt, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 77: + tag=ap_conn_done_pt summary="Connected to pluggable transport to build circuits" + [New in 0.4.0.x] +``` + +This is similar to conn_done_pt, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 78: + tag=ap_conn_proxy summary="Connecting to proxy to build circuits" + [New in 0.4.0.x] +``` + +This is similar to conn_proxy, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 79: + tag=ap_conn_done_proxy summary="Connected to proxy to build circuits" + [New in 0.4.0.x] +``` + +This is similar to conn_done_proxy, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 80: + tag=ap_conn summary="Connecting to a relay to build circuits" + [New in 0.4.0.x] +``` + +This is similar to conn, except for making connections to additional +relays or bridges that Tor needs to use to build application +circuits. + +```text + Phase 85: + tag=ap_conn_done summary="Connected to a relay to build circuits" + [New in 0.4.0.x] +``` + +This is similar to conn_done, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 89: + tag=ap_handshake summary="Finishing handshake with a relay to build circuits" + [New in 0.4.0.x] +``` + +This is similar to handshake, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 90: + tag=ap_handshake_done summary="Handshake finished with a relay to build circuits" + [New in 0.4.0.x] +``` + +This is similar to handshake_done, except for making connections to +additional relays or bridges that Tor needs to use to build +application circuits. + +```text + Phase 95: + tag=circuit_create summary="Establishing a[n internal] Tor circuit" + [prior to 0.4.0.x, this was numbered 90] +``` + +Once we've finished our TLS handshake with the first hop of a circuit, +we will set about trying to make some 3-hop circuits in case we need them +soon. + +```text + [Some versions of Tor (starting with 0.2.6.2-alpha but before + 0.4.0.x): Tor could report having internal paths only; see Section + 5.6] +``` + +Phase 100: +tag=done summary="Done" + +A full 3-hop circuit has been established. Tor is ready to handle +application connections now. + +```text + [Some versions of Tor (starting with 0.2.6.2-alpha but before + 0.4.0.x): Tor could report having internal paths only; see Section + 5.6] +``` + + +## Bootstrap phases reported by older versions of Tor + +These phases were reported by Tor older than 0.4.0.x. For newer +versions of Tor, see Section 5.5. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. When bootstrap completes, Tor will be ready + to handle an application requesting an exit circuit to services like the + World Wide Web. +``` + +If the consensus does not contain Exits, Tor will only build internal +circuits. In this case, earlier statuses will have included "internal" +as indicated above. When bootstrap completes, Tor will be ready to handle +an application requesting an internal circuit to hidden services at +".onion" addresses. + +If a future consensus contains Exits, exit circuits may become available.] + +Phase 0: +tag=starting summary="Starting" + +Tor starts out in this phase. + +Phase 5: +tag=conn_dir summary="Connecting to directory server" + +Tor sends this event as soon as Tor has chosen a directory server -- +e.g. one of the authorities if bootstrapping for the first time or +after a long downtime, or one of the relays listed in its cached +directory information otherwise. + +Tor will stay at this phase until it has successfully established +a TCP connection with some directory server. Problems in this phase +generally happen because Tor doesn't have a network connection, or +because the local firewall is dropping SYN packets. + +Phase 10: +tag=handshake_dir summary="Finishing handshake with directory server" + +This event occurs when Tor establishes a TCP connection with a relay or +authority used as a directory server (or its https proxy if it's using +one). Tor remains in this phase until the TLS handshake with the relay +or authority is finished. + +Problems in this phase generally happen because Tor's firewall is +doing more sophisticated MITM attacks on it, or doing packet-level +keyword recognition of Tor's handshake. + +Phase 15: +tag=onehop_create summary="Establishing an encrypted directory connection" + +Once TLS is finished with a relay, Tor will send a CREATE_FAST cell +to establish a one-hop circuit for retrieving directory information. +It will remain in this phase until it receives the CREATED_FAST cell +back, indicating that the circuit is ready. + +Phase 20: +tag=requesting_status summary="Asking for networkstatus consensus" + +Once we've finished our one-hop circuit, we will start a new stream +for fetching the networkstatus consensus. We'll stay in this phase +until we get the 'connected' relay cell back, indicating that we've +established a directory connection. + +Phase 25: +tag=loading_status summary="Loading networkstatus consensus" + +Once we've established a directory connection, we will start fetching +the networkstatus consensus document. This could take a while; this +phase is a good opportunity for using the "progress" keyword to indicate +partial progress. + +This phase could stall if the directory server we picked doesn't +have a copy of the networkstatus consensus so we have to ask another, +or it does give us a copy but we don't find it valid. + +Phase 40: +tag=loading_keys summary="Loading authority key certs" + +Sometimes when we've finished loading the networkstatus consensus, +we find that we don't have all the authority key certificates for the +keys that signed the consensus. At that point we put the consensus we +fetched on hold and fetch the keys so we can verify the signatures. + +```text + Phase 45 + tag=requesting_descriptors summary="Asking for relay descriptors + [ for internal paths]" +``` + +Once we have a valid networkstatus consensus and we've checked all +its signatures, we start asking for relay descriptors. We stay in this +phase until we have received a 'connected' relay cell in response to +a request for descriptors. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will ask for + descriptors for both exit and internal paths. If not, Tor will only ask + for descriptors for internal paths. In this case, this status will + include "internal" as indicated above.] + + Phase 50: + tag=loading_descriptors summary="Loading relay descriptors[ for internal + paths]" +``` + +We will ask for relay descriptors from several different locations, +so this step will probably make up the bulk of the bootstrapping, +especially for users with slow connections. We stay in this phase until +we have descriptors for a significant fraction of the usable relays +listed in the networkstatus consensus (this can be between 25% and 95% +depending on Tor's configuration and network consensus parameters). +This phase is also a good opportunity to use the "progress" keyword to +indicate partial steps. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will download + descriptors for both exit and internal paths. If not, Tor will only + download descriptors for internal paths. In this case, this status will + include "internal" as indicated above.] +``` + +Phase 80: +tag=conn_or summary="Connecting to the Tor network[ internally]" + +Once we have a valid consensus and enough relay descriptors, we choose +entry guard(s) and start trying to build some circuits. This step +is similar to the "conn_dir" phase above; the only difference is +the context. + +If a Tor starts with enough recent cached directory information, +its first bootstrap status event will be for the conn_or phase. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. If not, Tor will only build internal circuits. + In this case, this status will include "internal(ly)" as indicated above.] + + Phase 85: + tag=handshake_or summary="Finishing handshake with first hop[ of internal + circuit]" +``` + +This phase is similar to the "handshake_dir" phase, but it gets reached +if we finish a TCP connection to a Tor relay and we have already reached +the "conn_or" phase. We'll stay in this phase until we complete a TLS +handshake with a Tor relay. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor may be finishing + a handshake with the first hop if either an exit or internal circuit. In + this case, it won't specify which type. If the consensus contains no Exits, + Tor will only build internal circuits. In this case, this status will + include "internal" as indicated above.] +``` + +Phase 90: +tag=circuit_create summary="Establishing a[n internal] Tor circuit" + +Once we've finished our TLS handshake with the first hop of a circuit, +we will set about trying to make some 3-hop circuits in case we need them +soon. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. If not, Tor will only build internal circuits. + In this case, this status will include "internal" as indicated above.] +``` + +Phase 100: +tag=done summary="Done" + +A full 3-hop circuit has been established. Tor is ready to handle +application connections now. + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. At this stage, Tor will be ready to handle + an application requesting an exit circuit to services like the World + Wide Web. +``` + +If the consensus does not contain Exits, Tor will only build internal +circuits. In this case, earlier statuses will have included "internal" +as indicated above. At this stage, Tor will be ready to handle an +application requesting an internal circuit to hidden services at ".onion" +addresses. + +If a future consensus contains Exits, exit circuits may become available.] + diff --git a/spec/control-spec/message-format.md b/spec/control-spec/message-format.md new file mode 100644 index 0000000..182f8cc --- /dev/null +++ b/spec/control-spec/message-format.md @@ -0,0 +1,180 @@ + +# Message format + + +## Description format + +The message formats listed below use ABNF as described in RFC 2234. +The protocol itself is loosely based on SMTP (see RFC 2821). + +We use the following nonterminals from RFC 2822: atom, qcontent + +We define the following general-use nonterminals: + +QuotedString = DQUOTE *qcontent DQUOTE + +There are explicitly no limits on line length. All 8-bit characters +are permitted unless explicitly disallowed. In QuotedStrings, +backslashes and quotes must be escaped; other characters need not be +escaped. + +Wherever CRLF is specified to be accepted from the controller, Tor MAY also +accept LF. Tor, however, MUST NOT generate LF instead of CRLF. +Controllers SHOULD always send CRLF. + + +### Notes on an escaping bug + +CString = DQUOTE *qcontent DQUOTE + +Note that although these nonterminals have the same grammar, they +are interpreted differently. In a QuotedString, a backslash +followed by any character represents that character. But +in a CString, the escapes "\n", "\t", "\r", and the octal escapes +"\0" ... "\377" represent newline, tab, carriage return, and the +256 possible octet values respectively. + +The use of CString in this document reflects a bug in Tor; +they should have been QuotedString instead. In the future, they +may migrate to use QuotedString instead. If they do, the +QuotedString implementation will never place a backslash before a +"n", "t", "r", or digit, to ensure that old controllers don't get +confused. + +For future-proofing, controller implementors MAY use the following +rules to be compatible with buggy Tor implementations and with +future ones that implement the spec as intended: + +```text + Read \n \t \r and \0 ... \377 as C escapes. + Treat a backslash followed by any other character as that character. +``` + +Currently, many of the QuotedString instances below that Tor +outputs are in fact CStrings. We intend to fix this in future +versions of Tor, and document which ones were broken. (See +bugtracker ticket #14555 for a bit more information.) + +Note that this bug exists only in strings generated by Tor for the +Tor controller; Tor should parse input QuotedStrings from the +controller correctly. + + +## Commands from controller to Tor + +```text + Command = Keyword OptArguments CRLF / "+" Keyword OptArguments CRLF CmdData + Keyword = 1*ALPHA + OptArguments = [ SP *(SP / VCHAR) ] +``` + +A command is either a single line containing a Keyword and arguments, or a +multiline command whose initial keyword begins with +, and whose data +section ends with a single "." on a line of its own. (We use a special +character to distinguish multiline commands so that Tor can correctly parse +multi-line commands that it does not recognize.) Specific commands and +their arguments are described below in section 3. + + +## Replies from Tor to the controller + +```text + Reply = SyncReply / AsyncReply + SyncReply = *(MidReplyLine / DataReplyLine) EndReplyLine + AsyncReply = *(MidReplyLine / DataReplyLine) EndReplyLine + + MidReplyLine = StatusCode "-" ReplyLine + DataReplyLine = StatusCode "+" ReplyLine CmdData + EndReplyLine = StatusCode SP ReplyLine + ReplyLine = [ReplyText] CRLF + ReplyText = XXXX + StatusCode = 3DIGIT +``` + +Unless specified otherwise, multiple lines in a single reply from +Tor to the controller are guaranteed to share the same status +code. Specific replies are mentioned below in section 3, and +described more fully in section 4. + +[Compatibility note: versions of Tor before 0.2.0.3-alpha sometimes +generate AsyncReplies of the form "*(MidReplyLine / DataReplyLine)". +This is incorrect, but controllers that need to work with these +versions of Tor should be prepared to get multi-line AsyncReplies with +the final line (usually "650 OK") omitted.] + + +## General-use tokens + +; CRLF means, "the ASCII Carriage Return character (decimal value 13) +; followed by the ASCII Linefeed character (decimal value 10)." +CRLF = CR LF + +; How a controller tells Tor about a particular OR. There are four +; possible formats: +; $Fingerprint -- The router whose identity key hashes to the fingerprint. +; This is the preferred way to refer to an OR. +; $Fingerprint~Nickname -- The router whose identity key hashes to the +; given fingerprint, but only if the router has the given nickname. +; $Fingerprint=Nickname -- The router whose identity key hashes to the +; given fingerprint, but only if the router is Named and has the given +; nickname. +; Nickname -- The Named router with the given nickname, or, if no such +; router exists, any router whose nickname matches the one given. +; This is not a safe way to refer to routers, since Named status +; could under some circumstances change over time. +; +; The tokens that implement the above follow: + +ServerSpec = LongName / Nickname +LongName = Fingerprint [ "~" Nickname ] + +; For tors older than 0.3.1.3-alpha, LongName may have included an equal +; sign ("=") in lieu of a tilde ("~"). The presence of an equal sign +; denoted that the OR possessed the "Named" flag: + +LongName = Fingerprint [ ( "=" / "~" ) Nickname ] + +Fingerprint = "$" 40*HEXDIG +NicknameChar = "a"-"z" / "A"-"Z" / "0" - "9" +Nickname = 1*19 NicknameChar + +; What follows is an outdated way to refer to ORs. +; Feature VERBOSE_NAMES replaces ServerID with LongName in events and +; GETINFO results. VERBOSE_NAMES can be enabled starting in Tor version +; 0.1.2.2-alpha and it is always-on in 0.2.2.1-alpha and later. +ServerID = Nickname / Fingerprint + +; Unique identifiers for streams or circuits. Currently, Tor only +; uses digits, but this may change +StreamID = 1*16 IDChar +CircuitID = 1*16 IDChar +ConnID = 1*16 IDChar +QueueID = 1*16 IDChar +IDChar = ALPHA / DIGIT + +Address = ip4-address / ip6-address / hostname (XXXX Define these) + +; A "CmdData" section is a sequence of octets concluded by the terminating +; sequence CRLF "." CRLF. The terminating sequence may not appear in the +; body of the data. Leading periods on lines in the data are escaped with +; an additional leading period as in RFC 2821 section 4.5.2. +CmdData = *DataLine "." CRLF +DataLine = CRLF / "." 1*LineItem CRLF / NonDotItem *LineItem CRLF +LineItem = NonCR / 1*CR NonCRLF +NonDotItem = NonDotCR / 1*CR NonCRLF + +; ISOTime, ISOTime2, and ISOTime2Frac are time formats as specified in +; ISO8601. +; example ISOTime: "2012-01-11 12:15:33" +; example ISOTime2: "2012-01-11T12:15:33" +; example ISOTime2Frac: "2012-01-11T12:15:33.51" +IsoDatePart = 4*DIGIT "-" 2*DIGIT "-" 2*DIGIT +IsoTimePart = 2*DIGIT ":" 2*DIGIT ":" 2*DIGIT +ISOTime = IsoDatePart " " IsoTimePart +ISOTime2 = IsoDatePart "T" IsoTimePart +ISOTime2Frac = IsoTime2 [ "." 1*DIGIT ] + +; Numbers +LeadingDigit = "1" - "9" +UInt = LeadingDigit *Digit + diff --git a/spec/control-spec/protocol-outline.md b/spec/control-spec/protocol-outline.md new file mode 100644 index 0000000..b0dd9a9 --- /dev/null +++ b/spec/control-spec/protocol-outline.md @@ -0,0 +1,70 @@ + +# Protocol outline + +TC is a bidirectional message-based protocol. It assumes an underlying +stream for communication between a controlling process (the "client" +or "controller") and a Tor process (or "server"). The stream may be +implemented via TCP, TLS-over-TCP, a Unix-domain socket, or so on, +but it must provide reliable in-order delivery. For security, the +stream should not be accessible by untrusted parties. + +In TC, the client and server send typed messages to each other over the +underlying stream. The client sends "commands" and the server sends +"replies". + +By default, all messages from the server are in response to messages from +the client. Some client requests, however, will cause the server to send +messages to the client indefinitely far into the future. Such +"asynchronous" replies are marked as such. + +Servers respond to messages in the order messages are received. + + +## Forward-compatibility + +This is an evolving protocol; new client and server behavior will be +allowed in future versions. To allow new backward-compatible behavior +on behalf of the client, we may add new commands and allow existing +commands to take new arguments in future versions. To allow new +backward-compatible server behavior, we note various places below +where servers speaking a future version of this protocol may insert +new data, and note that clients should/must "tolerate" unexpected +elements in these places. There are two ways that we do this: + +* Adding a new field to a message: + +```text + For example, we might say "This message has three space-separated + fields; clients MUST tolerate more fields." This means that a + client MUST NOT crash or otherwise fail to parse the message or + other subsequent messages when there are more than three fields, and + that it SHOULD function at least as well when more fields are + provided as it does when it only gets the fields it accepts. The + most obvious way to do this is by ignoring additional fields; the + next-most-obvious way is to report additional fields verbatim to the + user, perhaps as part of an expert UI. + + * Adding a new possible value to a list of alternatives: + + For example, we might say "This field will be OPEN, CLOSED, or + CONNECTED. Clients MUST tolerate unexpected values." This means + that a client MUST NOT crash or otherwise fail to parse the message + or other subsequent messages when there are unexpected values, and + that it SHOULD try to handle the rest of the message as well as it + can. The most obvious way to do this is by pretending that each + list of alternatives has an additional "unrecognized value" element, + and mapping any unrecognized values to that element; the + next-most-obvious way is to create a separate "unrecognized value" + element for each unrecognized value. + + Clients SHOULD NOT "tolerate" unrecognized alternatives by + pretending that the message containing them is absent. For example, + a stream closed for an unrecognized reason is nevertheless closed, + and should be reported as such. + + (If some list of alternatives is given, and there isn't an explicit + statement that clients must tolerate unexpected values, clients still + must tolerate unexpected values. The only exception would be if there + were an explicit statement that no future values will ever be added.) +``` + diff --git a/spec/control-spec/replies.md b/spec/control-spec/replies.md new file mode 100644 index 0000000..5fb71e7 --- /dev/null +++ b/spec/control-spec/replies.md @@ -0,0 +1,1764 @@ + +# Replies + +Reply codes follow the same 3-character format as used by SMTP, with the +first character defining a status, the second character defining a +subsystem, and the third designating fine-grained information. + +The TC protocol currently uses the following first characters: + +```text + 2yz Positive Completion Reply + The command was successful; a new request can be started. + + 4yz Temporary Negative Completion reply + The command was unsuccessful but might be reattempted later. + + 5yz Permanent Negative Completion Reply + The command was unsuccessful; the client should not try exactly + that sequence of commands again. + + 6yz Asynchronous Reply + Sent out-of-order in response to an earlier SETEVENTS command. + + The following second characters are used: + + x0z Syntax + Sent in response to ill-formed or nonsensical commands. + + x1z Protocol + Refers to operations of the Tor Control protocol. + + x5z Tor + Refers to actual operations of Tor system. + + The following codes are defined: + + 250 OK + 251 Operation was unnecessary + [Tor has declined to perform the operation, but no harm was done.] + + 451 Resource exhausted + + 500 Syntax error: protocol + + 510 Unrecognized command + 511 Unimplemented command + 512 Syntax error in command argument + 513 Unrecognized command argument + 514 Authentication required + 515 Bad authentication + + 550 Unspecified Tor error + + 551 Internal error + [Something went wrong inside Tor, so that the client's + request couldn't be fulfilled.] + + 552 Unrecognized entity + [A configuration key, a stream ID, circuit ID, event, + mentioned in the command did not actually exist.] + + 553 Invalid configuration value + [The client tried to set a configuration option to an + incorrect, ill-formed, or impossible value.] + + 554 Invalid descriptor + + 555 Unmanaged entity + + 650 Asynchronous event notification +``` + +Unless specified to have specific contents, the human-readable messages +in error replies should not be relied upon to match those in this document. + + +## Asynchronous events + +These replies can be sent after a corresponding SETEVENTS command has been +received. They will not be interleaved with other Reply elements, but they +can appear between a command and its corresponding reply. For example, +this sequence is possible: + +```text + C: SETEVENTS CIRC + S: 250 OK + C: GETCONF SOCKSPORT ORPORT + S: 650 CIRC 1000 EXTENDED moria1,moria2 + S: 250-SOCKSPORT=9050 + S: 250 ORPORT=0 + + But this sequence is disallowed: + + C: SETEVENTS CIRC + S: 250 OK + C: GETCONF SOCKSPORT ORPORT + S: 250-SOCKSPORT=9050 + S: 650 CIRC 1000 EXTENDED moria1,moria2 + S: 250 ORPORT=0 +``` + +Clients MUST tolerate more arguments in an asynchronous reply than +expected, and MUST tolerate more lines in an asynchronous reply than +expected. For instance, a client that expects a CIRC message like: + +650 CIRC 1000 EXTENDED moria1,moria2 + +must tolerate: + +```text + 650-CIRC 1000 EXTENDED moria1,moria2 0xBEEF + 650-EXTRAMAGIC=99 + 650 ANONYMITY=high +``` + +If clients receives extended events (selected by USEFEATUERE +EXTENDED_EVENTS in Tor 0.1.2.2-alpha..Tor-0.2.1.x, and always-on in +Tor 0.2.2.x and later), then each event line as specified below may be +followed by additional arguments and additional lines. Additional +lines will be of the form: + +"650" ("-"/" ") KEYWORD ["=" ARGUMENTS] CRLF + +Additional arguments will be of the form + +SP KEYWORD ["=" ( QuotedString / * NonSpDquote ) ] + +Clients MUST tolerate events with arguments and keywords they do not +recognize, and SHOULD process those events as if any unrecognized +arguments and keywords were not present. + +Clients SHOULD NOT depend on the order of keyword=value arguments, +and SHOULD NOT depend on there being no new keyword=value arguments +appearing between existing keyword=value arguments, though as of this +writing (Jun 2011) some do. Thus, extensions to this protocol should +add new keywords only after the existing keywords, until all +controllers have been fixed. At some point this "SHOULD NOT" might +become a "MUST NOT". + + +### Circuit status changed + +The syntax is: + +```text + "650" SP "CIRC" SP CircuitID SP CircStatus [SP Path] + [SP "BUILD_FLAGS=" BuildFlags] [SP "PURPOSE=" Purpose] + [SP "HS_STATE=" HSState] [SP "REND_QUERY=" HSAddress] + [SP "TIME_CREATED=" TimeCreated] + [SP "REASON=" Reason [SP "REMOTE_REASON=" Reason]] + [SP "SOCKS_USERNAME=" EscapedUsername] + [SP "SOCKS_PASSWORD=" EscapedPassword] + [SP "HS_POW=" HSPoW ] + CRLF + + CircStatus = + "LAUNCHED" / ; circuit ID assigned to new circuit + "BUILT" / ; all hops finished, can now accept streams + "GUARD_WAIT" / ; all hops finished, waiting to see if a + ; circuit with a better guard will be usable. + "EXTENDED" / ; one more hop has been completed + "FAILED" / ; circuit closed (was not built) + "CLOSED" ; circuit closed (was built) + + Path = LongName *("," LongName) + ; In Tor versions 0.1.2.2-alpha through 0.2.2.1-alpha with feature + ; VERBOSE_NAMES turned off and before version 0.1.2.2-alpha, Path + ; is as follows: + ; Path = ServerID *("," ServerID) + + BuildFlags = BuildFlag *("," BuildFlag) + BuildFlag = "ONEHOP_TUNNEL" / "IS_INTERNAL" / + "NEED_CAPACITY" / "NEED_UPTIME" + + Purpose = "GENERAL" / "HS_CLIENT_INTRO" / "HS_CLIENT_REND" / + "HS_SERVICE_INTRO" / "HS_SERVICE_REND" / "TESTING" / + "CONTROLLER" / "MEASURE_TIMEOUT" / + "HS_VANGUARDS" / "PATH_BIAS_TESTING" / + "CIRCUIT_PADDING" + + HSState = "HSCI_CONNECTING" / "HSCI_INTRO_SENT" / "HSCI_DONE" / + "HSCR_CONNECTING" / "HSCR_ESTABLISHED_IDLE" / + "HSCR_ESTABLISHED_WAITING" / "HSCR_JOINED" / + "HSSI_CONNECTING" / "HSSI_ESTABLISHED" / + "HSSR_CONNECTING" / "HSSR_JOINED" + + HSPoWType = "v1" + HSPoWEffort = 1*DIGIT + HSPoW = HSPoWType "," HSPoWEffort + + EscapedUsername = QuotedString + EscapedPassword = QuotedString + + HSAddress = 16*Base32Character / 56*Base32Character + Base32Character = ALPHA / "2" / "3" / "4" / "5" / "6" / "7" + + TimeCreated = ISOTime2Frac + Seconds = 1*DIGIT + Microseconds = 1*DIGIT + + Reason = "NONE" / "TORPROTOCOL" / "INTERNAL" / "REQUESTED" / + "HIBERNATING" / "RESOURCELIMIT" / "CONNECTFAILED" / + "OR_IDENTITY" / "OR_CONN_CLOSED" / "TIMEOUT" / + "FINISHED" / "DESTROYED" / "NOPATH" / "NOSUCHSERVICE" / + "MEASUREMENT_EXPIRED" + + The path is provided only when the circuit has been extended at least one + hop. + + The "BUILD_FLAGS" field is provided only in versions 0.2.3.11-alpha + and later. Clients MUST accept build flags not listed above. + Build flags are defined as follows: + + ONEHOP_TUNNEL (one-hop circuit, used for tunneled directory conns) + IS_INTERNAL (internal circuit, not to be used for exiting streams) + NEED_CAPACITY (this circuit must use only high-capacity nodes) + NEED_UPTIME (this circuit must use only high-uptime nodes) + + The "PURPOSE" field is provided only in versions 0.2.1.6-alpha and + later, and only if extended events are enabled (see 3.19). Clients + MUST accept purposes not listed above. Purposes are defined as + follows: + + GENERAL (circuit for AP and/or directory request streams) + HS_CLIENT_INTRO (HS client-side introduction-point circuit) + HS_CLIENT_REND (HS client-side rendezvous circuit; carries AP streams) + HS_SERVICE_INTRO (HS service-side introduction-point circuit) + HS_SERVICE_REND (HS service-side rendezvous circuit) + TESTING (reachability-testing circuit; carries no traffic) + CONTROLLER (circuit built by a controller) + MEASURE_TIMEOUT (circuit being kept around to see how long it takes) + HS_VANGUARDS (circuit created ahead of time when using + HS vanguards, and later repurposed as needed) + PATH_BIAS_TESTING (circuit used to probe whether our circuits are + being deliberately closed by an attacker) + CIRCUIT_PADDING (circuit that is being held open to disguise its + true close time) + + The "HS_STATE" field is provided only for hidden-service circuits, + and only in versions 0.2.3.11-alpha and later. Clients MUST accept + hidden-service circuit states not listed above. Hidden-service + circuit states are defined as follows: + + HSCI_* (client-side introduction-point circuit states) + HSCI_CONNECTING (connecting to intro point) + HSCI_INTRO_SENT (sent INTRODUCE1; waiting for reply from IP) + HSCI_DONE (received reply from IP relay; closing) + + HSCR_* (client-side rendezvous-point circuit states) + HSCR_CONNECTING (connecting to or waiting for reply from RP) + HSCR_ESTABLISHED_IDLE (established RP; waiting for introduction) + HSCR_ESTABLISHED_WAITING (introduction sent to HS; waiting for rend) + HSCR_JOINED (connected to HS) + + HSSI_* (service-side introduction-point circuit states) + HSSI_CONNECTING (connecting to intro point) + HSSI_ESTABLISHED (established intro point) + + HSSR_* (service-side rendezvous-point circuit states) + HSSR_CONNECTING (connecting to client's rend point) + HSSR_JOINED (connected to client's RP circuit) + + The "SOCKS_USERNAME" and "SOCKS_PASSWORD" fields indicate the credentials + that were used by a SOCKS client to connect to Tor's SOCKS port and + initiate this circuit. (Streams for SOCKS clients connected with different + usernames and/or passwords are isolated on separate circuits if the + IsolateSOCKSAuth flag is active; see Proposal 171.) [Added in Tor + 0.4.3.1-alpha.] + + The "REND_QUERY" field is provided only for hidden-service-related + circuits, and only in versions 0.2.3.11-alpha and later. Clients + MUST accept hidden service addresses in formats other than that + specified above. [Added in Tor 0.4.3.1-alpha.] + + The "TIME_CREATED" field is provided only in versions 0.2.3.11-alpha and + later. TIME_CREATED is the time at which the circuit was created or + cannibalized. [Added in Tor 0.4.3.1-alpha.] + + The "REASON" field is provided only for FAILED and CLOSED events, and only + if extended events are enabled (see 3.19). Clients MUST accept reasons + not listed above. [Added in Tor 0.4.3.1-alpha.] Reasons are as given in + tor-spec.txt, except for: + + NOPATH (Not enough nodes to make circuit) + MEASUREMENT_EXPIRED (As "TIMEOUT", except that we had left the circuit + open for measurement purposes to see how long it + would take to finish.) + IP_NOW_REDUNDANT (Closing a circuit to an introduction point that + has become redundant, since some other circuit + opened in parallel with it has succeeded.) + + The "REMOTE_REASON" field is provided only when we receive a DESTROY or + TRUNCATE cell, and only if extended events are enabled. It contains the + actual reason given by the remote OR for closing the circuit. Clients MUST + accept reasons not listed above. Reasons are as listed in tor-spec.txt. + [Added in Tor 0.4.3.1-alpha.] +``` + + +### Stream status changed + +The syntax is: + +```text + "650" SP "STREAM" SP StreamID SP StreamStatus SP CircuitID SP Target + [SP "REASON=" Reason [ SP "REMOTE_REASON=" Reason ]] + [SP "SOURCE=" Source] [ SP "SOURCE_ADDR=" Address ":" Port ] + [SP "PURPOSE=" Purpose] [SP "SOCKS_USERNAME=" EscapedUsername] + [SP "SOCKS_PASSWORD=" EscapedPassword] + [SP "CLIENT_PROTOCOL=" ClientProtocol] [SP "NYM_EPOCH=" NymEpoch] + [SP "SESSION_GROUP=" SessionGroup] [SP "ISO_FIELDS=" IsoFields] + CRLF + + StreamStatus = + "NEW" / ; New request to connect + "NEWRESOLVE" / ; New request to resolve an address + "REMAP" / ; Address re-mapped to another + "SENTCONNECT" / ; Sent a connect cell along a circuit + "SENTRESOLVE" / ; Sent a resolve cell along a circuit + "SUCCEEDED" / ; Received a reply; stream established + "FAILED" / ; Stream failed and not retriable + "CLOSED" / ; Stream closed + "DETACHED" / ; Detached from circuit; still retriable + "CONTROLLER_WAIT" ; Waiting for controller to use ATTACHSTREAM + ; (new in 0.4.5.1-alpha) + "XOFF_SENT" ; XOFF has been sent for this stream + ; (new in 0.4.7.5-alpha) + "XOFF_RECV" ; XOFF has been received for this stream + ; (new in 0.4.7.5-alpha) + "XON_SENT" ; XON has been sent for this stream + ; (new in 0.4.7.5-alpha) + "XON_RECV" ; XON has been received for this stream + ; (new in 0.4.7.5-alpha) + + Target = TargetAddress ":" Port + Port = an integer from 0 to 65535 inclusive + TargetAddress = Address / "(Tor_internal)" + + EscapedUsername = QuotedString + EscapedPassword = QuotedString + + ClientProtocol = + "SOCKS4" / + "SOCKS5" / + "TRANS" / + "NATD" / + "DNS" / + "HTTPCONNECT" / + "UNKNOWN" + + NymEpoch = a nonnegative integer + SessionGroup = an integer + + IsoFields = a comma-separated list of IsoField values + + IsoField = + "CLIENTADDR" / + "CLIENTPORT" / + "DESTADDR" / + "DESTPORT" / + the name of a field that is valid for STREAM events +``` + +The circuit ID designates which circuit this stream is attached to. If +the stream is unattached, the circuit ID "0" is given. The target +indicates the address which the stream is meant to resolve or connect to; +it can be "(Tor_internal)" for a virtual stream created by the Tor program +to talk to itself. + +```text + Reason = "MISC" / "RESOLVEFAILED" / "CONNECTREFUSED" / + "EXITPOLICY" / "DESTROY" / "DONE" / "TIMEOUT" / + "NOROUTE" / "HIBERNATING" / "INTERNAL"/ "RESOURCELIMIT" / + "CONNRESET" / "TORPROTOCOL" / "NOTDIRECTORY" / "END" / + "PRIVATE_ADDR" + + The "REASON" field is provided only for FAILED, CLOSED, and DETACHED + events, and only if extended events are enabled (see 3.19). Clients MUST + accept reasons not listed above. Reasons are as given in tor-spec.txt, + except for: + + END (We received a RELAY_END cell from the other side of this + stream.) + PRIVATE_ADDR (The client tried to connect to a private address like + 127.0.0.1 or 10.0.0.1 over Tor.) + [XXXX document more. -NM] + + The "REMOTE_REASON" field is provided only when we receive a RELAY_END + cell, and only if extended events are enabled. It contains the actual + reason given by the remote OR for closing the stream. Clients MUST accept + reasons not listed above. Reasons are as listed in tor-spec.txt. + + "REMAP" events include a Source if extended events are enabled: + + Source = "CACHE" / "EXIT" + + Clients MUST accept sources not listed above. "CACHE" is given if + the Tor client decided to remap the address because of a cached + answer, and "EXIT" is given if the remote node we queried gave us + the new address as a response. + + The "SOURCE_ADDR" field is included with NEW and NEWRESOLVE events if + extended events are enabled. It indicates the address and port + that requested the connection, and can be (e.g.) used to look up the + requesting program. + + Purpose = "DIR_FETCH" / "DIR_UPLOAD" / "DNS_REQUEST" / + "USER" / "DIRPORT_TEST" + + The "PURPOSE" field is provided only for NEW and NEWRESOLVE events, and + only if extended events are enabled (see 3.19). Clients MUST accept + purposes not listed above. The purposes above are defined as: + + "DIR_FETCH" -- This stream is generated internally to Tor for + fetching directory information. + "DIR_UPLOAD" -- An internal stream for uploading information to + a directory authority. + "DIRPORT_TEST" -- A stream we're using to test our own directory + port to make sure it's reachable. + "DNS_REQUEST" -- A user-initiated DNS request. + "USER" -- This stream is handling user traffic, OR it's internal + to Tor, but it doesn't match one of the purposes above. + + The "SOCKS_USERNAME" and "SOCKS_PASSWORD" fields indicate the credentials + that were used by a SOCKS client to connect to Tor's SOCKS port and + initiate this stream. (Streams for SOCKS clients connected with different + usernames and/or passwords are isolated on separate circuits if the + IsolateSOCKSAuth flag is active; see Proposal 171.) + + The "CLIENT_PROTOCOL" field indicates the protocol that was used by a client + to initiate this stream. (Streams for clients connected with different + protocols are isolated on separate circuits if the IsolateClientProtocol + flag is active.) Controllers MUST tolerate unrecognized client protocols. + + The "NYM_EPOCH" field indicates the nym epoch that was active when a client + initiated this stream. The epoch increments when the NEWNYM signal is + received. (Streams with different nym epochs are isolated on separate + circuits.) + + The "SESSION_GROUP" field indicates the session group of the listener port + that a client used to initiate this stream. By default, the session group is + different for each listener port, but this can be overridden for a listener + via the "SessionGroup" option in torrc. (Streams with different session + groups are isolated on separate circuits.) + + The "ISO_FIELDS" field indicates the set of STREAM event fields for which + stream isolation is enabled for the listener port that a client used to + initiate this stream. The special values "CLIENTADDR", "CLIENTPORT", + "DESTADDR", and "DESTPORT", if their correspondingly named fields are not + present, refer to the Address and Port components of the "SOURCE_ADDR" and + Target fields. +``` + + +### OR Connection status changed + +The syntax is: + +```text + "650" SP "ORCONN" SP (LongName / Target) SP ORStatus [ SP "REASON=" + Reason ] [ SP "NCIRCS=" NumCircuits ] [ SP "ID=" ConnID ] CRLF + + ORStatus = "NEW" / "LAUNCHED" / "CONNECTED" / "FAILED" / "CLOSED" + + ; In Tor versions 0.1.2.2-alpha through 0.2.2.1-alpha with feature + ; VERBOSE_NAMES turned off and before version 0.1.2.2-alpha, OR + ; Connection is as follows: + "650" SP "ORCONN" SP (ServerID / Target) SP ORStatus [ SP "REASON=" + Reason ] [ SP "NCIRCS=" NumCircuits ] CRLF +``` + +NEW is for incoming connections, and LAUNCHED is for outgoing +connections. CONNECTED means the TLS handshake has finished (in +either direction). FAILED means a connection is being closed that +hasn't finished its handshake, and CLOSED is for connections that +have handshaked. + +A LongName or ServerID is specified unless it's a NEW connection, in +which case we don't know what server it is yet, so we use Address:Port. + +If extended events are enabled (see 3.19), optional reason and +circuit counting information is provided for CLOSED and FAILED +events. + +```text + Reason = "MISC" / "DONE" / "CONNECTREFUSED" / + "IDENTITY" / "CONNECTRESET" / "TIMEOUT" / "NOROUTE" / + "IOERROR" / "RESOURCELIMIT" / "PT_MISSING" + + NumCircuits counts both established and pending circuits. + + The ORStatus values are as follows: + + NEW -- We have received a new incoming OR connection, and are starting + the server-side handshake. + LAUNCHED -- We have launched a new outgoing OR connection, and are + starting the client-side handshake. + CONNECTED -- The OR connection has been connected and the handshake is + done. + FAILED -- Our attempt to open the OR connection failed. + CLOSED -- The OR connection closed in an unremarkable way. + + The Reason values for closed/failed OR connections are: + + DONE -- The OR connection has shut down cleanly. + CONNECTREFUSED -- We got an ECONNREFUSED while connecting to the target + OR. + IDENTITY -- We connected to the OR, but found that its identity was + not what we expected. + CONNECTRESET -- We got an ECONNRESET or similar IO error from the + connection with the OR. + TIMEOUT -- We got an ETIMEOUT or similar IO error from the connection + with the OR, or we're closing the connection for being idle for too + long. + NOROUTE -- We got an ENOTCONN, ENETUNREACH, ENETDOWN, EHOSTUNREACH, or + similar error while connecting to the OR. + IOERROR -- We got some other IO error on our connection to the OR. + RESOURCELIMIT -- We don't have enough operating system resources (file + descriptors, buffers, etc) to connect to the OR. + PT_MISSING -- No pluggable transport was available. + MISC -- The OR connection closed for some other reason. + + [First added ID parameter in 0.2.5.2-alpha] +``` + + +### Bandwidth used in the last second + +The syntax is: + +```text + "650" SP "BW" SP BytesRead SP BytesWritten *(SP Type "=" Num) CRLF + BytesRead = 1*DIGIT + BytesWritten = 1*DIGIT + Type = "DIR" / "OR" / "EXIT" / "APP" / ... + Num = 1*DIGIT +``` + +BytesRead and BytesWritten are the totals. [In a future Tor version, +we may also include a breakdown of the connection types that used +bandwidth this second (not implemented yet).] + + +### Log messages + +The syntax is: + +"650" SP Severity SP ReplyText CRLF + +or + +"650+" Severity CRLF Data 650 SP "OK" CRLF + +Severity = "DEBUG" / "INFO" / "NOTICE" / "WARN"/ "ERR" + +Some low-level logs may be sent from signal handlers, so their destination +logs must be signal-safe. These low-level logs include backtraces, +logging function errors, and errors in code called by logging functions. +Signal-safe logs are never sent as control port log events. + +Control port message trace debug logs are never sent as control port log +events, to avoid modifying control output when debugging. + + +### New descriptors available + +This event is generated when new router descriptors (not microdescs or +extrainfos or anything else) are received. + +Syntax: + +```text + "650" SP "NEWDESC" 1*(SP LongName) CRLF + ; In Tor versions 0.1.2.2-alpha through 0.2.2.1-alpha with feature + ; VERBOSE_NAMES turned off and before version 0.1.2.2-alpha, it + ; is as follows: + "650" SP "NEWDESC" 1*(SP ServerID) CRLF +``` + + +### New Address mapping + +These events are generated when a new address mapping is entered in +Tor's address map cache, or when the answer for a RESOLVE command is +found. Entries can be created by a successful or failed DNS lookup, +a successful or failed connection attempt, a RESOLVE command, +a MAPADDRESS command, the AutomapHostsOnResolve feature, or the +TrackHostExits feature. + +Syntax: + +```text + "650" SP "ADDRMAP" SP Address SP NewAddress SP Expiry + [SP "error=" ErrorCode] [SP "EXPIRES=" UTCExpiry] [SP "CACHED=" Cached] + [SP "STREAMID=" StreamId] CRLF + + NewAddress = Address / "" + Expiry = DQUOTE ISOTime DQUOTE / "NEVER" + + ErrorCode = "yes" / "internal" / "Unable to launch resolve request" + UTCExpiry = DQUOTE IsoTime DQUOTE + + Cached = DQUOTE "YES" DQUOTE / DQUOTE "NO" DQUOTE + StreamId = DQUOTE StreamId DQUOTE +``` + +Error and UTCExpiry are only provided if extended events are enabled. +The values for Error are mostly useless. Future values will be +chosen to match 1*(ALNUM / "_"); the "Unable to launch resolve request" +value is a bug in Tor before 0.2.4.7-alpha. + +Expiry is expressed as the local time (rather than UTC). This is a bug, +left in for backward compatibility; new code should look at UTCExpiry +instead. (If Expiry is "NEVER", UTCExpiry is omitted.) + +Cached indicates whether the mapping will be stored until it expires, or if +it is just a notification in response to a RESOLVE command. + +StreamId is the global stream identifier of the stream or circuit from which +the address was resolved. + + +### Descriptors uploaded to us in our role as authoritative dirserver + +[NOTE: This feature was removed in Tor 0.3.2.1-alpha.] + +Tor generates this event when it's a directory authority, and +somebody has just uploaded a server descriptor. + +Syntax: + +```text + "650" "+" "AUTHDIR_NEWDESCS" CRLF Action CRLF Message CRLF + Descriptor CRLF "." CRLF "650" SP "OK" CRLF + Action = "ACCEPTED" / "DROPPED" / "REJECTED" + Message = Text +``` + +The Descriptor field is the text of the server descriptor; the Action +field is "ACCEPTED" if we're accepting the descriptor as the new +best valid descriptor for its router, "REJECTED" if we aren't taking +the descriptor and we're complaining to the uploading relay about +it, and "DROPPED" if we decide to drop the descriptor without +complaining. The Message field is a human-readable string +explaining why we chose the Action. (It doesn't contain newlines.) + + +### Our descriptor changed + +Syntax: + +"650" SP "DESCCHANGED" CRLF + +[First added in 0.1.2.2-alpha.] + + +### Status events + +Status events (STATUS_GENERAL, STATUS_CLIENT, and STATUS_SERVER) are sent +based on occurrences in the Tor process pertaining to the general state of +the program. Generally, they correspond to log messages of severity Notice +or higher. They differ from log messages in that their format is a +specified interface. + +Syntax: + +```text + "650" SP StatusType SP StatusSeverity SP StatusAction + [SP StatusArguments] CRLF + + StatusType = "STATUS_GENERAL" / "STATUS_CLIENT" / "STATUS_SERVER" + StatusSeverity = "NOTICE" / "WARN" / "ERR" + StatusAction = 1*ALPHA + StatusArguments = StatusArgument *(SP StatusArgument) + StatusArgument = StatusKeyword '=' StatusValue + StatusKeyword = 1*(ALNUM / "_") + StatusValue = 1*(ALNUM / '_') / QuotedString + + StatusAction is a string, and StatusArguments is a series of + keyword=value pairs on the same line. Values may be space-terminated + strings, or quoted strings. + + These events are always produced with EXTENDED_EVENTS and + VERBOSE_NAMES; see the explanations in the USEFEATURE section + for details. + + Controllers MUST tolerate unrecognized actions, MUST tolerate + unrecognized arguments, MUST tolerate missing arguments, and MUST + tolerate arguments that arrive in any order. + + Each event description below is accompanied by a recommendation for + controllers. These recommendations are suggestions only; no controller + is required to implement them. +``` + +Compatibility note: versions of Tor before 0.2.0.22-rc incorrectly +generated "STATUS_SERVER" as "STATUS_SEVER". To be compatible with those +versions, tools should accept both. + +Actions for STATUS_GENERAL events can be as follows: + +```text + CLOCK_JUMPED + "TIME=NUM" + Tor spent enough time without CPU cycles that it has closed all + its circuits and will establish them anew. This typically + happens when a laptop goes to sleep and then wakes up again. It + also happens when the system is swapping so heavily that Tor is + starving. The "time" argument specifies the number of seconds Tor + thinks it was unconscious for (or alternatively, the number of + seconds it went back in time). + + This status event is sent as NOTICE severity normally, but WARN + severity if Tor is acting as a server currently. + + {Recommendation for controller: ignore it, since we don't really + know what the user should do anyway. Hm.} + + DANGEROUS_VERSION + "CURRENT=version" + "REASON=NEW/OBSOLETE/UNRECOMMENDED" + "RECOMMENDED=\"version, version, ...\"" + Tor has found that directory servers don't recommend its version of + the Tor software. RECOMMENDED is a comma-and-space-separated string + of Tor versions that are recommended. REASON is NEW if this version + of Tor is newer than any recommended version, OBSOLETE if + this version of Tor is older than any recommended version, and + UNRECOMMENDED if some recommended versions of Tor are newer and + some are older than this version. (The "OBSOLETE" reason was called + "OLD" from Tor 0.1.2.3-alpha up to and including 0.2.0.12-alpha.) + + {Controllers may want to suggest that the user upgrade OLD or + UNRECOMMENDED versions. NEW versions may be known-insecure, or may + simply be development versions.} + + TOO_MANY_CONNECTIONS + "CURRENT=NUM" + Tor has reached its ulimit -n or whatever the native limit is on file + descriptors or sockets. CURRENT is the number of sockets Tor + currently has open. The user should really do something about + this. The "current" argument shows the number of connections currently + open. + + {Controllers may recommend that the user increase the limit, or + increase it for them. Recommendations should be phrased in an + OS-appropriate way and automated when possible.} + + BUG + "REASON=STRING" + Tor has encountered a situation that its developers never expected, + and the developers would like to learn that it happened. Perhaps + the controller can explain this to the user and encourage her to + file a bug report? + + {Controllers should log bugs, but shouldn't annoy the user in case a + bug appears frequently.} + + CLOCK_SKEW + SKEW="+" / "-" SECONDS + MIN_SKEW="+" / "-" SECONDS. + SOURCE="DIRSERV:" IP ":" Port / + "NETWORKSTATUS:" IP ":" Port / + "OR:" IP ":" Port / + "CONSENSUS" + If "SKEW" is present, it's an estimate of how far we are from the + time declared in the source. (In other words, if we're an hour in + the past, the value is -3600.) "MIN_SKEW" is present, it's a lower + bound. If the source is a DIRSERV, we got the current time from a + connection to a dirserver. If the source is a NETWORKSTATUS, we + decided we're skewed because we got a v2 networkstatus from far in + the future. If the source is OR, the skew comes from a NETINFO + cell from a connection to another relay. If the source is + CONSENSUS, we decided we're skewed because we got a networkstatus + consensus from the future. + + {Tor should send this message to controllers when it thinks the + skew is so high that it will interfere with proper Tor operation. + Controllers shouldn't blindly adjust the clock, since the more + accurate source of skew info (DIRSERV) is currently + unauthenticated.} + + BAD_LIBEVENT + "METHOD=" libevent method + "VERSION=" libevent version + "BADNESS=" "BROKEN" / "BUGGY" / "SLOW" + "RECOVERED=" "NO" / "YES" + Tor knows about bugs in using the configured event method in this + version of libevent. "BROKEN" libevents won't work at all; + "BUGGY" libevents might work okay; "SLOW" libevents will work + fine, but not quickly. If "RECOVERED" is YES, Tor managed to + switch to a more reliable (but probably slower!) libevent method. + + {Controllers may want to warn the user if this event occurs, though + generally it's the fault of whoever built the Tor binary and there's + not much the user can do besides upgrade libevent or upgrade the + binary.} + + DIR_ALL_UNREACHABLE + Tor believes that none of the known directory servers are + reachable -- this is most likely because the local network is + down or otherwise not working, and might help to explain for the + user why Tor appears to be broken. + + {Controllers may want to warn the user if this event occurs; further + action is generally not possible.} + + Actions for STATUS_CLIENT events can be as follows: + + BOOTSTRAP + "PROGRESS=" num + "TAG=" Keyword + "SUMMARY=" String + ["WARNING=" String] + ["REASON=" Keyword] + ["COUNT=" num] + ["RECOMMENDATION=" Keyword] + ["HOST=" QuotedString] + ["HOSTADDR=" QuotedString] + + Tor has made some progress at establishing a connection to the + Tor network, fetching directory information, or making its first + circuit; or it has encountered a problem while bootstrapping. This + status event is especially useful for users with slow connections + or with connectivity problems. + + "Progress" gives a number between 0 and 100 for how far through + the bootstrapping process we are. "Summary" is a string that can + be displayed to the user to describe the *next* task that Tor + will tackle, i.e., the task it is working on after sending the + status event. "Tag" is a string that controllers can use to + recognize bootstrap phases, if they want to do something smarter + than just blindly displaying the summary string; see Section 5 + for the current tags that Tor issues. + + The StatusSeverity describes whether this is a normal bootstrap + phase (severity notice) or an indication of a bootstrapping + problem (severity warn). + + For bootstrap problems, we include the same progress, tag, and + summary values as we would for a normal bootstrap event, but we + also include "warning", "reason", "count", and "recommendation" + key/value combos. The "count" number tells how many bootstrap + problems there have been so far at this phase. The "reason" + string lists one of the reasons allowed in the ORCONN event. The + "warning" argument string with any hints Tor has to offer about + why it's having troubles bootstrapping. + + The "reason" values are long-term-stable controller-facing tags to + identify particular issues in a bootstrapping step. The warning + strings, on the other hand, are human-readable. Controllers + SHOULD NOT rely on the format of any warning string. Currently + the possible values for "recommendation" are either "ignore" or + "warn" -- if ignore, the controller can accumulate the string in + a pile of problems to show the user if the user asks; if warn, + the controller should alert the user that Tor is pretty sure + there's a bootstrapping problem. + + The "host" value is the identity digest (in hex) of the node we're + trying to connect to; the "hostaddr" is an address:port combination, + where 'address' is an ipv4 or ipv6 address. + + Currently Tor uses recommendation=ignore for the first + nine bootstrap problem reports for a given phase, and then + uses recommendation=warn for subsequent problems at that + phase. Hopefully this is a good balance between tolerating + occasional errors and reporting serious problems quickly. + + ENOUGH_DIR_INFO + Tor now knows enough network-status documents and enough server + descriptors that it's going to start trying to build circuits now. + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build + both exit and internal circuits. If not, Tor will only build internal + circuits.] + + {Controllers may want to use this event to decide when to indicate + progress to their users, but should not interrupt the user's browsing + to tell them so.} + + NOT_ENOUGH_DIR_INFO + We discarded expired statuses and server descriptors to fall + below the desired threshold of directory information. We won't + try to build any circuits until ENOUGH_DIR_INFO occurs again. + + {Controllers may want to use this event to decide when to indicate + progress to their users, but should not interrupt the user's browsing + to tell them so.} + + CIRCUIT_ESTABLISHED + Tor is able to establish circuits for client use. This event will + only be sent if we just built a circuit that changed our mind -- + that is, prior to this event we didn't know whether we could + establish circuits. + + {Suggested use: controllers can notify their users that Tor is + ready for use as a client once they see this status event. [Perhaps + controllers should also have a timeout if too much time passes and + this event hasn't arrived, to give tips on how to troubleshoot. + On the other hand, hopefully Tor will send further status events + if it can identify the problem.]} + + CIRCUIT_NOT_ESTABLISHED + "REASON=" "EXTERNAL_ADDRESS" / "DIR_ALL_UNREACHABLE" / "CLOCK_JUMPED" + We are no longer confident that we can build circuits. The "reason" + keyword provides an explanation: which other status event type caused + our lack of confidence. + + {Controllers may want to use this event to decide when to indicate + progress to their users, but should not interrupt the user's browsing + to do so.} + [Note: only REASON=CLOCK_JUMPED is implemented currently.] + + CONSENSUS_ARRIVED + Tor has received and validated a new consensus networkstatus. + (This event can be delayed a little while after the consensus + is received, if Tor needs to fetch certificates.) + + DANGEROUS_PORT + "PORT=" port + "RESULT=" "REJECT" / "WARN" + A stream was initiated to a port that's commonly used for + vulnerable-plaintext protocols. If the Result is "reject", we + refused the connection; whereas if it's "warn", we allowed it. + + {Controllers should warn their users when this occurs, unless they + happen to know that the application using Tor is in fact doing so + correctly (e.g., because it is part of a distributed bundle). They + might also want some sort of interface to let the user configure + their RejectPlaintextPorts and WarnPlaintextPorts config options.} + + DANGEROUS_SOCKS + "PROTOCOL=" "SOCKS4" / "SOCKS5" + "ADDRESS=" IP:port + A connection was made to Tor's SOCKS port using one of the SOCKS + approaches that doesn't support hostnames -- only raw IP addresses. + If the client application got this address from gethostbyname(), + it may be leaking target addresses via DNS. + + {Controllers should warn their users when this occurs, unless they + happen to know that the application using Tor is in fact doing so + correctly (e.g., because it is part of a distributed bundle).} + + SOCKS_UNKNOWN_PROTOCOL + "DATA=string" + A connection was made to Tor's SOCKS port that tried to use it + for something other than the SOCKS protocol. Perhaps the user is + using Tor as an HTTP proxy? The DATA is the first few characters + sent to Tor on the SOCKS port. + + {Controllers may want to warn their users when this occurs: it + indicates a misconfigured application.} + + SOCKS_BAD_HOSTNAME + "HOSTNAME=QuotedString" + Some application gave us a funny-looking hostname. Perhaps + it is broken? In any case it won't work with Tor and the user + should know. + + {Controllers may want to warn their users when this occurs: it + usually indicates a misconfigured application.} + + Actions for STATUS_SERVER can be as follows: + + EXTERNAL_ADDRESS + "ADDRESS=IP" + "HOSTNAME=NAME" + "METHOD=CONFIGURED/CONFIGURED_ORPORT/DIRSERV/RESOLVED/ + INTERFACE/GETHOSTNAME" + Our best idea for our externally visible IP has changed to 'IP'. If + 'HOSTNAME' is present, we got the new IP by resolving 'NAME'. If the + method is 'CONFIGURED', the IP was given verbatim as the Address + configuration option. If the method is 'CONFIGURED_ORPORT', the IP was + given verbatim in the ORPort configuration option. If the method is + 'RESOLVED', we resolved the Address configuration option to get the IP. + If the method is 'GETHOSTNAME', we resolved our hostname to get the IP. + If the method is 'INTERFACE', we got the address of one of our network + interfaces to get the IP. If the method is 'DIRSERV', a directory + server told us a guess for what our IP might be. + + {Controllers may want to record this info and display it to the user.} + + CHECKING_REACHABILITY + "ORADDRESS=IP:port" + "DIRADDRESS=IP:port" + We're going to start testing the reachability of our external OR port + or directory port. + + {This event could affect the controller's idea of server status, but + the controller should not interrupt the user to tell them so.} + + REACHABILITY_SUCCEEDED + "ORADDRESS=IP:port" + "DIRADDRESS=IP:port" + We successfully verified the reachability of our external OR port or + directory port (depending on which of ORADDRESS or DIRADDRESS is + given.) + + {This event could affect the controller's idea of server status, but + the controller should not interrupt the user to tell them so.} + + GOOD_SERVER_DESCRIPTOR + We successfully uploaded our server descriptor to at least one + of the directory authorities, with no complaints. + + {Originally, the goal of this event was to declare "every authority + has accepted the descriptor, so there will be no complaints + about it." But since some authorities might be offline, it's + harder to get certainty than we had thought. As such, this event + is equivalent to ACCEPTED_SERVER_DESCRIPTOR below. Controllers + should just look at ACCEPTED_SERVER_DESCRIPTOR and should ignore + this event for now.} + + SERVER_DESCRIPTOR_STATUS + "STATUS=" "LISTED" / "UNLISTED" + We just got a new networkstatus consensus, and whether we're in + it or not in it has changed. Specifically, status is "listed" + if we're listed in it but previous to this point we didn't know + we were listed in a consensus; and status is "unlisted" if we + thought we should have been listed in it (e.g. we were listed in + the last one), but we're not. + + {Moving from listed to unlisted is not necessarily cause for + alarm. The relay might have failed a few reachability tests, + or the Internet might have had some routing problems. So this + feature is mainly to let relay operators know when their relay + has successfully been listed in the consensus.} + + [Not implemented yet. We should do this in 0.2.2.x. -RD] + + NAMESERVER_STATUS + "NS=addr" + "STATUS=" "UP" / "DOWN" + "ERR=" message + One of our nameservers has changed status. + + {This event could affect the controller's idea of server status, but + the controller should not interrupt the user to tell them so.} + + NAMESERVER_ALL_DOWN + All of our nameservers have gone down. + + {This is a problem; if it happens often without the nameservers + coming up again, the user needs to configure more or better + nameservers.} + + DNS_HIJACKED + Our DNS provider is providing an address when it should be saying + "NOTFOUND"; Tor will treat the address as a synonym for "NOTFOUND". + + {This is an annoyance; controllers may want to tell admins that their + DNS provider is not to be trusted.} + + DNS_USELESS + Our DNS provider is giving a hijacked address instead of well-known + websites; Tor will not try to be an exit node. + + {Controllers could warn the admin if the relay is running as an + exit node: the admin needs to configure a good DNS server. + Alternatively, this happens a lot in some restrictive environments + (hotels, universities, coffeeshops) when the user hasn't registered.} + + BAD_SERVER_DESCRIPTOR + "DIRAUTH=addr:port" + "REASON=string" + A directory authority rejected our descriptor. Possible reasons + include malformed descriptors, incorrect keys, highly skewed clocks, + and so on. + + {Controllers should warn the admin, and try to cope if they can.} + + ACCEPTED_SERVER_DESCRIPTOR + "DIRAUTH=addr:port" + A single directory authority accepted our descriptor. + // actually notice + + {This event could affect the controller's idea of server status, but + the controller should not interrupt the user to tell them so.} + + REACHABILITY_FAILED + "ORADDRESS=IP:port" + "DIRADDRESS=IP:port" + We failed to connect to our external OR port or directory port + successfully. + + {This event could affect the controller's idea of server status. The + controller should warn the admin and suggest reasonable steps to take.} + + HIBERNATION_STATUS + "STATUS=" "AWAKE" | "SOFT" | "HARD" + Our bandwidth based accounting status has changed, and we are now + relaying traffic/rejecting new connections/hibernating. + + {This event could affect the controller's idea of server status. The + controller MAY inform the admin, though presumably the accounting was + explicitly enabled for a reason.} + + [This event was added in tor 0.2.9.0-alpha.] +``` + + +### Our set of guard nodes has changed + +Syntax: + +```text + "650" SP "GUARD" SP Type SP Name SP Status ... CRLF + Type = "ENTRY" + Name = ServerSpec + (Identifies the guard affected) + Status = "NEW" | "UP" | "DOWN" | "BAD" | "GOOD" | "DROPPED" +``` + +The ENTRY type indicates a guard used for connections to the Tor +network. + +The Status values are: + +```text + "NEW" -- This node was not previously used as a guard; now we have + picked it as one. + "DROPPED" -- This node is one we previously picked as a guard; we + no longer consider it to be a member of our guard list. + "UP" -- The guard now seems to be reachable. + "DOWN" -- The guard now seems to be unreachable. + "BAD" -- Because of flags set in the consensus and/or values in the + configuration, this node is now unusable as a guard. + "BAD_L2" -- This layer2 guard has expired or got removed from the + consensus. This node is removed from the layer2 guard set. + "GOOD" -- Because of flags set in the consensus and/or values in the + configuration, this node is now usable as a guard. + + Controllers must accept unrecognized types and unrecognized statuses. +``` + + +### Network status has changed + +Syntax: + +"650" "+" "NS" CRLF 1*NetworkStatus "." CRLF "650" SP "OK" CRLF + +The event is used whenever our local view of a relay status changes. +This happens when we get a new v3 consensus (in which case the entries +we see are a duplicate of what we see in the NEWCONSENSUS event, +below), but it also happens when we decide to mark a relay as up or +down in our local status, for example based on connection attempts. + +[First added in 0.1.2.3-alpha] + + +### Bandwidth used on an application stream + +The syntax is: + +```text + "650" SP "STREAM_BW" SP StreamID SP BytesWritten SP BytesRead SP + Time CRLF + BytesWritten = 1*DIGIT + BytesRead = 1*DIGIT + Time = ISOTime2Frac +``` + +BytesWritten and BytesRead are the number of bytes written and read +by the application since the last STREAM_BW event on this stream. + +Note that from Tor's perspective, *reading* a byte on a stream means +that the application *wrote* the byte. That's why the order of "written" +vs "read" is opposite for stream_bw events compared to bw events. + +The Time field is provided only in versions 0.3.2.1-alpha and later. It +records when Tor created the bandwidth event. + +These events are generated about once per second per stream; no events +are generated for streams that have not written or read. These events +apply only to streams entering Tor (such as on a SOCKSPort, TransPort, +or so on). They are not generated for exiting streams. + + +### Per-country client stats + +The syntax is: + +```text + "650" SP "CLIENTS_SEEN" SP TimeStarted SP CountrySummary SP + IPVersions CRLF +``` + +We just generated a new summary of which countries we've seen clients +from recently. The controller could display this for the user, e.g. +in their "relay" configuration window, to give them a sense that they +are actually being useful. + +Currently only bridge relays will receive this event, but once we figure +out how to sufficiently aggregate and sanitize the client counts on +main relays, we might start sending these events in other cases too. + +TimeStarted is a quoted string indicating when the reported summary +counts from (in UTCS). + +The CountrySummary keyword has as its argument a comma-separated, +possibly empty set of "countrycode=count" pairs. For example (without +linebreak), +650-CLIENTS_SEEN TimeStarted="2008-12-25 23:50:43" +CountrySummary=us=16,de=8,uk=8 + +The IPVersions keyword has as its argument a comma-separated set of +"protocol-family=count" pairs. For example, +IPVersions=v4=16,v6=40 + +Note that these values are rounded, not exact. The rounding +algorithm is specified in the description of "geoip-client-origins" +in dir-spec.txt. + + +### New consensus networkstatus has arrived + +The syntax is: + +```text + "650" "+" "NEWCONSENSUS" CRLF 1*NetworkStatus "." CRLF "650" SP + "OK" CRLF +``` + +A new consensus networkstatus has arrived. We include NS-style lines for +every relay in the consensus. NEWCONSENSUS is a separate event from the +NS event, because the list here represents every usable relay: so any +relay *not* mentioned in this list is implicitly no longer recommended. + +[First added in 0.2.1.13-alpha] + + +### New circuit buildtime has been set + +The syntax is: + +```text + "650" SP "BUILDTIMEOUT_SET" SP Type SP "TOTAL_TIMES=" Total SP + "TIMEOUT_MS=" Timeout SP "XM=" Xm SP "ALPHA=" Alpha SP + "CUTOFF_QUANTILE=" Quantile SP "TIMEOUT_RATE=" TimeoutRate SP + "CLOSE_MS=" CloseTimeout SP "CLOSE_RATE=" CloseRate + CRLF + Type = "COMPUTED" / "RESET" / "SUSPENDED" / "DISCARD" / "RESUME" + Total = Integer count of timeouts stored + Timeout = Integer timeout in milliseconds + Xm = Estimated integer Pareto parameter Xm in milliseconds + Alpha = Estimated floating point Paredo parameter alpha + Quantile = Floating point CDF quantile cutoff point for this timeout + TimeoutRate = Floating point ratio of circuits that timeout + CloseTimeout = How long to keep measurement circs in milliseconds + CloseRate = Floating point ratio of measurement circuits that are closed +``` + +A new circuit build timeout time has been set. If Type is "COMPUTED", +Tor has computed the value based on historical data. If Type is "RESET", +initialization or drastic network changes have caused Tor to reset +the timeout back to the default, to relearn again. If Type is +"SUSPENDED", Tor has detected a loss of network connectivity and has +temporarily changed the timeout value to the default until the network +recovers. If type is "DISCARD", Tor has decided to discard timeout +values that likely happened while the network was down. If type is +"RESUME", Tor has decided to resume timeout calculation. + +The Total value is the count of circuit build times Tor used in +computing this value. It is capped internally at the maximum number +of build times Tor stores (NCIRCUITS_TO_OBSERVE). + +The Timeout itself is provided in milliseconds. Internally, Tor rounds +this value to the nearest second before using it. + +[First added in 0.2.2.7-alpha] + + +### Signal received + +The syntax is: + +"650" SP "SIGNAL" SP Signal CRLF + +Signal = "RELOAD" / "DUMP" / "DEBUG" / "NEWNYM" / "CLEARDNSCACHE" + +A signal has been received and actions taken by Tor. The meaning of each +signal, and the mapping to Unix signals, is as defined in section 3.7. +Future versions of Tor MAY generate signals other than those listed here; +controllers MUST be able to accept them. + +If Tor chose to ignore a signal (such as NEWNYM), this event will not be +sent. Note that some options (like ReloadTorrcOnSIGHUP) may affect the +semantics of the signals here. + +Note that the HALT (SIGTERM) and SHUTDOWN (SIGINT) signals do not currently +generate any event. + +[First added in 0.2.3.1-alpha] + + +### Configuration changed + +The syntax is: + +StartReplyLine *(MidReplyLine) EndReplyLine + +```text + StartReplyLine = "650-CONF_CHANGED" CRLF + MidReplyLine = "650-" KEYWORD ["=" VALUE] CRLF + EndReplyLine = "650 OK" +``` + +Tor configuration options have changed (such as via a SETCONF or RELOAD +signal). KEYWORD and VALUE specify the configuration option that was changed. +Undefined configuration options contain only the KEYWORD. + + +### Circuit status changed slightly + +The syntax is: + +```text + "650" SP "CIRC_MINOR" SP CircuitID SP CircEvent [SP Path] + [SP "BUILD_FLAGS=" BuildFlags] [SP "PURPOSE=" Purpose] + [SP "HS_STATE=" HSState] [SP "REND_QUERY=" HSAddress] + [SP "TIME_CREATED=" TimeCreated] + [SP "OLD_PURPOSE=" Purpose [SP "OLD_HS_STATE=" HSState]] CRLF + + CircEvent = + "PURPOSE_CHANGED" / ; circuit purpose or HS-related state changed + "CANNIBALIZED" ; circuit cannibalized + + Clients MUST accept circuit events not listed above. +``` + +The "OLD_PURPOSE" field is provided for both PURPOSE_CHANGED and +CANNIBALIZED events. The "OLD_HS_STATE" field is provided whenever +the "OLD_PURPOSE" field is provided and is a hidden-service-related +purpose. + +Other fields are as specified in section 4.1.1 above. + +[First added in 0.2.3.11-alpha] + + +### Pluggable transport launched + +The syntax is: + +```text + "650" SP "TRANSPORT_LAUNCHED" SP Type SP Name SP TransportAddress SP Port + Type = "server" | "client" + Name = The name of the pluggable transport + TransportAddress = An IPv4 or IPv6 address on which the pluggable + transport is listening for connections + Port = The TCP port on which it is listening for connections. + + A pluggable transport called 'Name' of type 'Type' was launched + successfully and is now listening for connections on 'Address':'Port'. +``` + + +### Bandwidth used on an OR or DIR or EXIT connection + +The syntax is: + +```text + "650" SP "CONN_BW" SP "ID=" ConnID SP "TYPE=" ConnType + SP "READ=" BytesRead SP "WRITTEN=" BytesWritten CRLF + + ConnType = "OR" / ; Carrying traffic within the tor network. This can + either be our own (client) traffic or traffic we're + relaying within the network. + "DIR" / ; Fetching tor descriptor data, or transmitting + descriptors we're mirroring. + "EXIT" ; Carrying traffic between the tor network and an + external destination. + + BytesRead = 1*DIGIT + BytesWritten = 1*DIGIT + + Controllers MUST tolerate unrecognized connection types. +``` + +BytesWritten and BytesRead are the number of bytes written and read +by Tor since the last CONN_BW event on this connection. + +These events are generated about once per second per connection; no +events are generated for connections that have not read or written. +These events are only generated if TestingTorNetwork is set. + +[First added in 0.2.5.2-alpha] + + +### Bandwidth used by all streams attached to a circuit + +The syntax is: + +```text + "650" SP "CIRC_BW" SP "ID=" CircuitID SP "READ=" BytesRead SP + "WRITTEN=" BytesWritten SP "TIME=" Time SP + "DELIVERED_READ=" DeliveredBytesRead SP + "OVERHEAD_READ=" OverheadBytesRead SP + "DELIVERED_WRITTEN=" DeliveredBytesWritten SP + "OVERHEAD_WRITTEN=" OverheadBytesWritten SP + "SS=" SlowStartState SP + "CWND=" CWNDCells SP + "RTT=" RTTMilliseconds SP + "MIN_RTT=" RTTMilliseconds CRLF + BytesRead = 1*DIGIT + BytesWritten = 1*DIGIT + OverheadBytesRead = 1*DIGIT + OverheadBytesWritten = 1*DIGIT + DeliveredBytesRead = 1*DIGIT + DeliveredBytesWritten = 1*DIGIT + SlowStartState = 0 or 1 + CWNDCells = 1*DIGIT + RTTMilliseconds= 1*DIGIT + Time = ISOTime2Frac +``` + +BytesRead and BytesWritten are the number of bytes read and written +on this circuit since the last CIRC_BW event. These bytes have not +necessarily been validated by Tor, and can include invalid cells, +dropped cells, and ignored cells (such as padding cells). These +values include the relay headers, but not circuit headers. + +Circuit data that has been validated and processed by Tor is further +broken down into two categories: delivered payloads and overhead. +DeliveredBytesRead and DeliveredBytesWritten are the total relay cell +payloads transmitted since the last CIRC_BW event, not counting relay +cell headers or circuit headers. OverheadBytesRead and +OverheadBytesWritten are the extra unused bytes at the end of each +cell in order for it to be the fixed CELL_LEN bytes long. + +The sum of DeliveredBytesRead and OverheadBytesRead MUST be less than +BytesRead, and the same is true for their written counterparts. This +sum represents the total relay cell bytes on the circuit that +have been validated by Tor, not counting relay headers and cell headers. +Subtracting this sum (plus relay cell headers) from the BytesRead +(or BytesWritten) value gives the byte count that Tor has decided to +reject due to protocol errors, or has otherwise decided to ignore. + +The Time field is provided only in versions 0.3.2.1-alpha and later. It +records when Tor created the bandwidth event. + +The SS, CWND, RTT, and MIN_RTT fields are present only if the circuit +has negotiated congestion control to an onion service or Exit hop (any +intermediate leaky pipe congestion control hops are not examined here). +SS provides an indication if the circuit is in slow start (1), or not (0). +CWND is the size of the congestion window in terms of number of cells. +RTT is the N_EWMA smoothed current RTT value, and MIN_RTT is the minimum +RTT value of the circuit. The SS and CWND fields apply only to the +upstream direction of the circuit. The slow start state and CWND values +of the other endpoint may be different. + +These events are generated about once per second per circuit; no events +are generated for circuits that had no attached stream writing or +reading. + +[First added in 0.2.5.2-alpha] + +[DELIVERED_READ, OVERHEAD_READ, DELIVERED_WRITTEN, and OVERHEAD_WRITTEN +were added in Tor 0.3.4.0-alpha] + +[SS, CWND, RTT, and MIN_RTT were added in Tor 0.4.7.5-alpha] + + +### Per-circuit cell stats + +The syntax is: + +```text + "650" SP "CELL_STATS" + [ SP "ID=" CircuitID ] + [ SP "InboundQueue=" QueueID SP "InboundConn=" ConnID ] + [ SP "InboundAdded=" CellsByType ] + [ SP "InboundRemoved=" CellsByType SP + "InboundTime=" MsecByType ] + [ SP "OutboundQueue=" QueueID SP "OutboundConn=" ConnID ] + [ SP "OutboundAdded=" CellsByType ] + [ SP "OutboundRemoved=" CellsByType SP + "OutboundTime=" MsecByType ] CRLF + CellsByType, MsecByType = CellType ":" 1*DIGIT + 0*( "," CellType ":" 1*DIGIT ) + CellType = 1*( "a" - "z" / "0" - "9" / "_" ) + + Examples are: + + 650 CELL_STATS ID=14 OutboundQueue=19403 OutboundConn=15 + OutboundAdded=create_fast:1,relay_early:2 + OutboundRemoved=create_fast:1,relay_early:2 + OutboundTime=create_fast:0,relay_early:0 + 650 CELL_STATS InboundQueue=19403 InboundConn=32 + InboundAdded=relay:1,created_fast:1 + InboundRemoved=relay:1,created_fast:1 + InboundTime=relay:0,created_fast:0 + OutboundQueue=6710 OutboundConn=18 + OutboundAdded=create:1,relay_early:1 + OutboundRemoved=create:1,relay_early:1 + OutboundTime=create:0,relay_early:0 +``` + +ID is the locally unique circuit identifier that is only included if the +circuit originates at this node. + +Inbound and outbound refer to the direction of cell flow through the +circuit which is either to origin (inbound) or from origin (outbound). + +InboundQueue and OutboundQueue are identifiers of the inbound and +outbound circuit queues of this circuit. These identifiers are only +unique per OR connection. OutboundQueue is chosen by this node and +matches InboundQueue of the next node in the circuit. + +InboundConn and OutboundConn are locally unique IDs of inbound and +outbound OR connection. OutboundConn does not necessarily match +InboundConn of the next node in the circuit. + +InboundQueue and InboundConn are not present if the circuit originates +at this node. OutboundQueue and OutboundConn are not present if the +circuit (currently) ends at this node. + +InboundAdded and OutboundAdded are total number of cells by cell type +added to inbound and outbound queues. Only present if at least one cell +was added to a queue. + +InboundRemoved and OutboundRemoved are total number of cells by +cell type processed from inbound and outbound queues. InboundTime and +OutboundTime are total waiting times in milliseconds of all processed +cells by cell type. Only present if at least one cell was removed from +a queue. + +These events are generated about once per second per circuit; no +events are generated for circuits that have not added or processed any +cell. These events are only generated if TestingTorNetwork is set. + +[First added in 0.2.5.2-alpha] + + +### Token buckets refilled + +The syntax is: + +```text + "650" SP "TB_EMPTY" SP BucketName [ SP "ID=" ConnID ] SP + "READ=" ReadBucketEmpty SP "WRITTEN=" WriteBucketEmpty SP + "LAST=" LastRefill CRLF + + BucketName = "GLOBAL" / "RELAY" / "ORCONN" + ReadBucketEmpty = 1*DIGIT + WriteBucketEmpty = 1*DIGIT + LastRefill = 1*DIGIT + + Examples are: + + 650 TB_EMPTY ORCONN ID=16 READ=0 WRITTEN=0 LAST=100 + 650 TB_EMPTY GLOBAL READ=93 WRITTEN=93 LAST=100 + 650 TB_EMPTY RELAY READ=93 WRITTEN=93 LAST=100 +``` + +This event is generated when refilling a previously empty token +bucket. BucketNames "GLOBAL" and "RELAY" keywords are used for the +global or relay token buckets, BucketName "ORCONN" is used for the +token buckets of an OR connection. Controllers MUST tolerate +unrecognized bucket names. + +ConnID is only included if the BucketName is "ORCONN". + +If both global and relay buckets and/or the buckets of one or more OR +connections run out of tokens at the same time, multiple separate +events are generated. + +ReadBucketEmpty (WriteBucketEmpty) is the time in millis that the read +(write) bucket was empty since the last refill. LastRefill is the +time in millis since the last refill. + +If a bucket went negative and if refilling tokens didn't make it go +positive again, there will be multiple consecutive TB_EMPTY events for +each refill interval during which the bucket contained zero tokens or +less. In such a case, ReadBucketEmpty or WriteBucketEmpty are capped +at LastRefill in order not to report empty times more than once. + +These events are only generated if TestingTorNetwork is set. + +[First added in 0.2.5.2-alpha] + + +### HiddenService descriptors + +The syntax is: + +```text + "650" SP "HS_DESC" SP Action SP HSAddress SP AuthType SP HsDir + [SP DescriptorID] [SP "REASON=" Reason] [SP "REPLICA=" Replica] + [SP "HSDIR_INDEX=" HSDirIndex] + + Action = "REQUESTED" / "UPLOAD" / "RECEIVED" / "UPLOADED" / "IGNORE" / + "FAILED" / "CREATED" + HSAddress = 16*Base32Character / 56*Base32Character / "UNKNOWN" + AuthType = "NO_AUTH" / "BASIC_AUTH" / "STEALTH_AUTH" / "UNKNOWN" + HsDir = LongName / Fingerprint / "UNKNOWN" + DescriptorID = 32*Base32Character / 43*Base64Character + Reason = "BAD_DESC" / "QUERY_REJECTED" / "UPLOAD_REJECTED" / "NOT_FOUND" / + "UNEXPECTED" / "QUERY_NO_HSDIR" / "QUERY_RATE_LIMITED" + Replica = 1*DIGIT + HSDirIndex = 64*HEXDIG + + These events will be triggered when required HiddenService descriptor is + not found in the cache and a fetch or upload with the network is performed. + + If the fetch was triggered with only a DescriptorID (using the HSFETCH + command for instance), the HSAddress only appears in the Action=RECEIVED + since there is no way to know the HSAddress from the DescriptorID thus + the value will be "UNKNOWN". + + If we already had the v0 descriptor, the newly fetched v2 descriptor + will be ignored and a "HS_DESC" event with "IGNORE" action will be + generated. + + For HsDir, LongName is always preferred. If HsDir cannot be found in node + list at the time event is sent, Fingerprint will be used instead. + + If Action is "FAILED", Tor SHOULD send Reason field as well. Possible + values of Reason are: + - "BAD_DESC" - descriptor was retrieved, but found to be unparsable. + - "QUERY_REJECTED" - query was rejected by HS directory. + - "UPLOAD_REJECTED" - descriptor was rejected by HS directory. + - "NOT_FOUND" - HS descriptor with given identifier was not found. + - "UNEXPECTED" - nature of failure is unknown. + - "QUERY_NO_HSDIR" - No suitable HSDir were found for the query. + - "QUERY_RATE_LIMITED" - query for this service is rate-limited + + For "QUERY_NO_HSDIR" or "QUERY_RATE_LIMITED", the HsDir will be set to + "UNKNOWN" which was introduced in tor 0.3.1.0-alpha and 0.4.1.0-alpha + respectively. + + If Action is "CREATED", Tor SHOULD send Replica field as well. The Replica + field contains the replica number of the generated descriptor. The Replica + number is specified in rend-spec.txt section 1.3 and determines the + descriptor ID of the descriptor. + + For hidden service v3, the following applies: + + The "HSDIR_INDEX=" is an optional field that is only for version 3 + which contains the computed index of the HsDir the descriptor was + uploaded to or fetched from. + + The "DescriptorID" key is the descriptor blinded key used for the index + value at the "HsDir". + + The "REPLICA=" field is not used for the "CREATED" event because v3 + doesn't use the replica number in the descriptor ID computation. + + Because client authentication is not yet implemented, the "AuthType" + field is always "NO_AUTH". + + [HS v3 support added 0.3.3.1-alpha] +``` + + +### HiddenService descriptors content + +The syntax is: + +```text + "650" "+" "HS_DESC_CONTENT" SP HSAddress SP DescId SP HsDir CRLF + Descriptor CRLF "." CRLF "650" SP "OK" CRLF + + HSAddress = 16*Base32Character / 56*Base32Character / "UNKNOWN" + DescId = 32*Base32Character / 32*Base64Character + HsDir = LongName / "UNKNOWN" + Descriptor = The text of the descriptor formatted as specified in + rend-spec.txt section 1.3 (v2) or rend-spec-v3.txt + section 2.4 (v3) or empty string on failure. +``` + +This event is triggered when a successfully fetched HS descriptor is +received. The text of that descriptor is then replied. If the HS_DESC +event is enabled, it is replied just after the RECEIVED action. + +If a fetch fails, the Descriptor is an empty string and HSAddress is set +to "UNKNOWN". The HS_DESC event should be used to get more information on +the failed request. + +If the fetch fails for the QUERY_NO_HSDIR or QUERY_RATE_LIMITED reason from +the HS_DESC event, the HsDir is set to "UNKNOWN". This was introduced in +0.3.1.0-alpha and 0.4.1.0-alpha respectively. + +It's expected to receive a reply relatively fast as in it's the time it +takes to fetch something over the Tor network. This can be between a +couple of seconds up to 60 seconds (not a hard limit). But, in any cases, +this event will reply either the descriptor's content or an empty one. + +[HS_DESC_CONTENT was added in Tor 0.2.7.1-alpha] +[HS v3 support added 0.3.3.1-alpha] + + +### Network liveness has changed + +Syntax: + +```text + "650" SP "NETWORK_LIVENESS" SP Status CRLF + Status = "UP" / ; The network now seems to be reachable. + "DOWN" / ; The network now seems to be unreachable. + + Controllers MUST tolerate unrecognized status types. + + [NETWORK_LIVENESS was added in Tor 0.2.7.2-alpha] +``` + + +### Pluggable Transport Logs + +Syntax: + +"650" SP "PT_LOG" SP PT=Program SP Message + +```text + Program = The program path as defined in the *TransportPlugin + configuration option. Tor accepts relative and full path. + Message = The log message that the PT sends back to the tor parent + process minus the "LOG" string prefix. Formatted as + specified in pt-spec.txt section "3.3.4. Pluggable + Transport Log Message". + + This event is triggered when tor receives a log message from the PT. + + Example: + + PT (obfs4): LOG SEVERITY=debug MESSAGE="Connected to bridge A" + + the resulting control port event would be: + + Tor: 650 PT_LOG PT=/usr/bin/obs4proxy SEVERITY=debug MESSAGE="Connected to bridge A" + + [PT_LOG was added in Tor 0.4.0.1-alpha] +``` + + +### Pluggable Transport Status + +Syntax: + +"650" SP "PT_STATUS" SP PT=Program SP TRANSPORT=Transport SP Message + +```text + Program = The program path as defined in the *TransportPlugin + configuration option. Tor accepts relative and full path. + Transport = This value indicates a hint on what the PT is such as the + name or the protocol used for instance. + Message = The status message that the PT sends back to the tor parent + process minus the "STATUS" string prefix. Formatted as + specified in pt-spec.txt section "3.3.5 Pluggable + Transport Status Message". + + This event is triggered when tor receives a log message from the PT. + + Example: + + PT (obfs4): STATUS TRANSPORT=obfs4 CONNECT=Success + + the resulting control port event would be: + + Tor: 650 PT_STATUS PT=/usr/bin/obs4proxy TRANSPORT=obfs4 CONNECT=Success + + [PT_STATUS was added in Tor 0.4.0.1-alpha] +``` + diff --git a/spec/control-spec/scope.md b/spec/control-spec/scope.md new file mode 100644 index 0000000..8684e3e --- /dev/null +++ b/spec/control-spec/scope.md @@ -0,0 +1,22 @@ + +# Scope + +This document describes an implementation-specific protocol that is used +for other programs (such as frontend user-interfaces) to communicate with a +locally running Tor process. It is not part of the Tor onion routing +protocol. + +This protocol replaces version 0 of TC, which is now deprecated. For +reference, TC is described in "control-spec-v0.txt". Implementors are +recommended to avoid using TC directly, but instead to use a library that +can easily be updated to use the newer protocol. (Version 0 is used by Tor +versions 0.1.0.x; the protocol in this document only works with Tor +versions in the 0.1.1.x series and later.) + +```text + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL + NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + RFC 2119. +``` + diff --git a/spec/dir-list-spec.md b/spec/dir-list-spec.md new file mode 100644 index 0000000..e6c1e0c --- /dev/null +++ b/spec/dir-list-spec.md @@ -0,0 +1,568 @@ +```text + Tor Directory List Format + Tim Wilson-Brown (teor) + +Table of Contents + + 1. Scope and Preliminaries + 1.1. Format Overview + 1.2. Acknowledgements + 1.3. Format Versions + 1.4. Future Plans + 2. Format Details + 2.1. Nonterminals + 2.2. List Header + 2.2.1. List Header Format + 2.3. List Generation + 2.3.1. List Generation Format + 2.4. Directory Entry + 2.4.1. Directory Entry Format + 3. Usage Considerations + 3.1. Caching + 3.2. Retrieving Directory Information + 3.3. Fallback Reliability + A.1. Sample Data + A.1.1. Sample Fallback List Header + A.1.2. Sample Fallback List Generation + A.1.3. Sample Fallback Entries +``` + + +# Scope and Preliminaries + +This document describes the format of Tor's directory lists, which are +compiled and hard-coded into the tor binary. There is currently one +list: the fallback directory mirrors. This list is also parsed by other +libraries, like stem and metrics-lib. Alternate Tor implementations can +use this list to bootstrap from the latest public Tor directory +information. + +The FallbackDir feature was introduced by proposal 210, and was first +supported by Tor in Tor version 0.2.4.7-alpha. The first hard-coded +list was shipped in 0.2.8.1-alpha. + +The hard-coded fallback directory list is located in the tor source +repository at: + +src/app/config/fallback_dirs.inc + +In Tor 0.3.4 and earlier, the list is located at: + +src/or/fallback_dirs.inc + +This document describes version 2.0.0 and later of the directory list +format. + +Legacy, semi-structured versions of the fallback list were released with +Tor 0.2.8.1-alpha through Tor 0.3.1.9. We call this format version 1. +Stem and Relay Search have parsers for this legacy format. + + +## Format Overview + +A directory list is a C code fragment containing an array of C string +constants. Each double-quoted C string constant is a valid torrc +FallbackDir entry. Each entry contains various data fields. + +Directory lists do not include the C array's declaration, or the array's +terminating NULL. Entries in directory lists do not include the +FallbackDir torrc option. These are handled by the including C code. + +Directory lists also include C-style comments and whitespace. The +presence of whitespace may be significant, but the amount of whitespace +is never significant. The type of whitespace is not significant to the +C compiler or Tor C string parser. However, other parsers MAY rely on +the distinction between newlines and spaces. (And that the only +whitespace characters in the list are newlines and spaces.) + +The directory entry C string constants are split over multiple lines for +readability. Structured C-style comments are used to provide additional +data fields. This information is not used by Tor, but may be of interest +to other libraries. + +The order of directory entries and data fields is not significant, +except where noted below. + + +## Acknowledgements + +The original fallback directory script and format was created by +weasel. The current script uses code written by gsathya & karsten. + +This specification was revised after feedback from: + +```text + Damian Johnson ("atagar") + Iain R. Learmonth ("irl") +``` + + +## Format Versions + +The directory list format uses semantic versioning: https://semver.org + +```text + In particular: + * major versions are used for incompatible changes, like + removing non-optional fields + * minor versions are used for compatible changes, like adding + fields + * patch versions are for bug fixes, like fixing an + incorrectly-formatted Summary item + + 1.0.0 - The legacy fallback directory list format + + 2.0.0 - Adds name and extrainfo structured comments, and section separator + comments to make the list easier to parses. Also adds a source list + comment to the header. + + 3.0.0 - Modifies the format of the source list comment. +``` + + +## Future Plans + +Tor also has an auth_dirs.inc file, but it is not yet in this format. +Tor uses slightly different formats for authorities and fallback +directory mirrors, so we will need to make some changes to tor so that +it parses this format. (We will also need to add authority-specific +information to this format.) See #24818 for details. + +We want to add a torrc option so operators can opt-in their relays as +fallback directory mirrors. This gives us a signed opt-in confirmation. +(We can also continue to accept whitelist entries, and do other checks.) +We need to write a short proposal, and make some changes to tor and the +fallback update script. See #24839 for details. + + +# Format Details + +Directory lists contain the following sections: + +```text + - List Header (exactly once) + - List Generation (exactly once, may be empty) + - Directory Entry (zero or more times) + + Each section (or entry) ends with a separator. +``` + + +## Nonterminals + +The following nonterminals are defined in the Onionoo details document +specification: + +```text + dir_address + fingerprint + nickname + + See https://metrics.torproject.org/onionoo.html#details +``` + +The following nonterminals are defined in the "Tor directory protocol" +specification in dir-spec.txt: + +```text + Keyword + ArgumentChar + NL (newline) + SP (space) + bool (must not be confused with Onionoo's JSON "boolean") + + We derive the following nonterminals from Onionoo and dir-spec.txt: + + ipv4_or_port ::= port from an IPv4 or_addresses item + + The ipv4_or_port is the port part of an IPv4 address from the + Onionoo or_addresses list. + + ipv6_or_address ::= an IPv6 or_addresses item + + The ipv6_or_address is an IPv6 address and port from the Onionoo + or_addresses list. The address MAY be in the canonical RFC 5952 + IPv6 address format. + + A key-value pair: + + value ::= Zero or more ArgumentChar, excluding the following strings: + * a double quotation mark (DQUOTE), and + * the C comment terminators ("/*" and "*/"). + + Note that the C++ comment ("//") and equals sign ("=") are + not excluded, because they are reserved for future use in + base64 values. + + key_value ::= Keyword "=" value + + We also define these additional nonterminals: + + number ::= An optional negative sign ("-"), followed by one or more + numeric characters ([0-9]), with an optional decimal part + (".", followed by one or more numeric characters). + + separator ::= "/*" SP+ "=====" SP+ "*/" +``` + + +## List Header + +The list header consists of a number of key-value pairs, embedded in +C-style comments. + + +### List Header Format + +"/*" SP+ "type=" Keyword SP+ "*/" SP* NL + +[At start, exactly once.] + +```text + The type of directory entries in the list. Parsers SHOULD exit with + an error if this is not the first line of the list, or if the value + is anything other than "fallback". + + "/*" SP+ "version=" version_number SP+ "*/" SP* NL + + [In second position, exactly once.] + + The version of the directory list format. + + version_number is a semantic version, see the "Format Versions" + section for details. + + Version 1.0.0 represents the undocumented, legacy fallback list + format(s). Version 2.0.0 and later are documented by this + specification. + + "/*" SP+ "timestamp=" number SP+ "*/" SP* NL + + [Exactly once.] + + A positive integer that indicates when this directory list was + generated. This timestamp is guaranteed to increase for every + version 2.0.0 and later directory list. + + The current timestamp format is YYYYMMDDHHMMSS, as an integer. + + "/*" SP+ "source=" Keyword ("," Keyword)* SP+ "*/" SP* NL + + [Zero or one time.] + + A list of the sources of the directory entries in the list. + + As of version 3.0.0, the possible sources are: + * "offer-list" - the fallback_offer_list file in the fallback-scripts + repository. + * "descriptor" - one or more signed descriptors, each containing an + "offer-fallback-dir" line. This feature will be + implemented in ticket #24839. + * "fallback" - a fallback_dirs.inc file from a tor repository. + Used in check_existing mode. + + Before #24839 is implemented, the default is "offer-list". During the + transition to signed offers, it will be "descriptor,offer-list". + Afterwards, it will be "descriptor". + + In version 2.0.0, only one source name was allowed after "source=", + and the deprecated "whitelist" source name was used instead of + "offer-list". + + This line was added in version 2.0.0 of this specification. The format + of this line was modified in version 3.0.0 of this specification. + + "/*" SP+ key_value SP+ "*/" SP* NL + + [Zero or more times.] + + Future releases may include additional header fields. Parsers MUST NOT + rely on the order of these additional fields. Additional header fields + will be accompanied by a minor version increment. + + separator SP* NL + + The list header ends with the section separator. +``` + + +## List Generation + +The list generation information consists of human-readable prose +describing the content and origin of this directory list. It is contained +in zero or more C-style comments, and may contain multi-line comments and +uncommented C code. + +In particular, this section may contain C-style comments that contain +an equals ("=") character. It may also be entirely empty. + +Future releases may arbitrarily change the content of this section. +Parsers MUST NOT rely on a version increment when the format changes. + + +### List Generation Format + +In general, parsers MUST NOT rely on the format of this section. + +Parsers MAY rely on the following details: + +The list generation section MUST NOT be a valid directory entry. + +The list generation summary MUST end with a section separator: + +separator SP* NL + +There MUST NOT be any section separators in the list generation +section, other than the terminating section separator. + + +## Directory Entry + +A directory entry consists of a C string constant, and one or more +C-style comments. The C string constant is a valid argument to the +DirAuthority or FallbackDir torrc option. The section also contains +additional key-value fields in C-style comments. + +The list of fallback entries does not include the directory +authorities: they are in a separate list. (The Tor implementation combines +these lists after parsing them, and applies the DirAuthorityFallbackRate +to their weights.) + + +### Directory Entry Format + +```text + If a directory entry does not conform to this format, the entry SHOULD + be ignored by parsers. + + DQUOTE dir_address SP+ "orport=" ipv4_or_port SP+ + "id=" fingerprint DQUOTE SP* NL + + [At start, exactly once, on a single line.] + + This line consists of the following fields: + + dir_address + + An IPv4 address and DirPort for this directory, as defined by + Onionoo. In this format version, all IPv4 addresses and DirPorts + are guaranteed to be non-zero. (For IPv4 addresses, this means + that they are not equal to "0.0.0.0".) + + ipv4_or_port + + An IPv4 ORPort for this directory, derived from Onionoo. In this + format version, all IPv4 ORPorts are guaranteed to be non-zero. + + fingerprint + + The relay fingerprint of this directory, as defined by Onionoo. + All relay fingerprints are guaranteed to have one or more non-zero + digits. + + Note: + + Each double-quoted C string line that occurs after the first line, + starts with space inside the quotes. This is a requirement of the + Tor implementation. + + DQUOTE SP+ "ipv6=" ipv6_or_address DQUOTE SP* NL + + [Zero or one time.] + + The IPv6 address and ORPort for this directory, as defined by + Onionoo. If present, IPv6 addresses and ORPorts are guaranteed to be + non-zero. (For IPv6 addresses, this means that they are not equal to + "[::]".) + + DQUOTE SP+ "weight=" number DQUOTE SP* NL + + [Zero or one time.] + + A non-negative, real-numbered weight for this directory. + The default fallback weight is 1.0, and the default + DirAuthorityFallbackRate is 1.0 in legacy Tor versions, and 0.1 in + recent Tor versions. + + weight was removed in version 2.0.0, but is documented because it + may be of interest to libraries implementing Tor's fallback + behaviour. + + DQUOTE SP+ key_value DQUOTE SP* NL + + [Zero or more times.] + + Future releases may include additional data fields in double-quoted + C string constants. Parsers MUST NOT rely on the order of these + additional fields. Additional data fields will be accompanied by a + minor version increment. + + "/*" SP+ "nickname=" nickname* SP+ "*/" SP* NL + + [Exactly once.] + + The nickname for this directory, as defined by Onionoo. An + empty nickname indicates that the nickname is unknown. + + The first fallback list in the 2.0.0 format had nickname lines, but + they were all empty. + + "/*" SP+ "extrainfo=" bool SP+ "*/" SP* NL + + [Exactly once.] + + An integer flag that indicates whether this directory caches + extra-info documents. Set to 1 if the directory claimed that it + cached extra-info documents in its descriptor when the list was + created. 0 indicates that it did not, or its descriptor was not + available. + + The first fallback list in the 2.0.0 format had extrainfo lines, but + they were all zero. + + "/*" SP+ key_value SP+ "*/" SP* NL + + [Zero or more times.] + + Future releases may include additional data fields in C-style + comments. Parsers MUST NOT rely on the order of these additional + fields. Additional data fields will be accompanied by a minor version + increment. + + separator SP* NL + + [Exactly once.] + + Each directory entry ends with the section separator. + + "," SP* NL + + [Exactly once.] + + The comma terminates the C string constant. (Multiple C string + constants separated by whitespace or comments are coalesced by + the C compiler.) +``` + + +# Usage Considerations + +This section contains recommended library behaviours. It does not affect +the format of directory lists. + + +## Caching + +The fallback list typically changes once every 6-12 months. The data in +the list represents the state of the fallback directory entries when the +list was created. Fallbacks can and do change their details over time. + +Libraries SHOULD parse and cache the most recent version of these lists +during their build or release processes. Libraries MUST NOT retrieve the +lists by default every time they are deployed or executed. + +The latest fallback list can be retrieved from: + +https://gitweb.torproject.org/tor.git/plain/src/or/fallback_dirs.inc + +Libraries MUST NOT rely on the availability of the server that hosts +these lists. + +The list can also be retrieved using: + +git clone https://git.torproject.org/tor.git + +If you just want the latest list, you may wish to perform a shallow +clone. + + +## Retrieving Directory Information + +Some libraries retrieve directory documents directly from the Tor +Directory Authorities. The directory authorities are designed to support +Tor relay and client bootstrap, and MAY choose to rate-limit library +access. Libraries MAY provide a user-agent in their requests, if they +are not intended to support anonymous operation. (User agents are a +fingerprinting vector.) + +Libraries SHOULD consider the potential load on the authorities, and +whether other sources can meet their needs. + +Libraries that require high-uptime availability of Tor directory +information should investigate the following options: + +```text + * OnionOO: https://metrics.torproject.org/onionoo.html + * Third-party OnionOO mirrors are also available + * CollecTor: https://collector.torproject.org/ + * Fallback Directory Mirrors +``` + +Onionoo and CollecTor are typically updated every hour on a regular +schedule. Fallbacks update their own directory information at random +intervals, see dir-spec for details. + + +## Fallback Reliability + +The fallback list is typically regenerated when the fallback failure +rate exceeds 25%. Libraries SHOULD NOT rely on any particular fallback +being available, or some proportion of fallbacks being available. + +Libraries that use fallbacks MAY wish to query an authority after a +few fallback queries fail. For example, Tor clients try 3-4 fallbacks +before trying an authority. + + +## Sample Data + +A sample version 2.0.0 fallback list is available here: + +https://trac.torproject.org/projects/tor/raw-attachment/ticket/22759/fallback_dirs_new_format_version.4.inc + +A sample transitional version 2.0.0 fallback list is available here: + +https://raw.githubusercontent.com/teor2345/tor/fallback-format-2-v4/src/or/fallback_dirs.inc + + +### Sample Fallback List Header + +/* type=fallback */ +/* version=2.0.0 */ +/* ===== */ + + +### Sample Fallback List Generation + +/* Whitelist & blacklist excluded 1326 of 1513 candidates. */ +/* Checked IPv4 DirPorts served a consensus within 15.0s. */ +/* +Final Count: 151 (Eligible 187, Target 392 (1963 * 0.20), Max 200) +Excluded: 36 (Same Operator 27, Failed/Skipped Download 9, Excess 0) +Bandwidth Range: 1.3 - 40.0 MByte/s +*/ +/* +Onionoo Source: details Date: 2017-05-16 07:00:00 Version: 4.0 +URL: https:onionoo.torproject.orgdetails?fields=fingerprint%2Cnickname%2Ccontact%2Clast_changed_address_or_port%2Cconsensus_weight%2Cadvertised_bandwidth%2Cor_addresses%2Cdir_address%2Crecommended_version%2Cflags%2Ceffective_family%2Cplatform&flag=V2Dir&type=relay&last_seen_days=-0&first_seen_days=30- +*/ +/* +Onionoo Source: uptime Date: 2017-05-16 07:00:00 Version: 4.0 +URL: https:onionoo.torproject.orguptime?first_seen_days=30-&flag=V2Dir&type=relay&last_seen_days=-0 +*/ +/* ===== */ + + +### Sample Fallback Entries + +"176.10.104.240:80 orport=443 id=0111BA9B604669E636FFD5B503F382A4B7AD6E80" +/* nickname=foo */ +/* extrainfo=1 */ +/* ===== */ +, +"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" +" ipv6=[2a01:4f8:162:51e2::2]:9001" +/* nickname= */ +/* extrainfo=0 */ +/* ===== */ +, + diff --git a/spec/dir-spec-intro.md b/spec/dir-spec-intro.md new file mode 100644 index 0000000..9a4ef78 --- /dev/null +++ b/spec/dir-spec-intro.md @@ -0,0 +1,70 @@ +Tor directory protocol, version 3 + +Table of Contents + +```text + 0. Scope and preliminaries + 0.1. History + 0.2. Goals of the version 3 protoc + 0.3. Some Remaining questions + 1. Outline + 1.1. What's different from version 2? + 1.2. Document meta-format + 1.3. Signing documents + 1.4. Voting timeline + 2. Router operation and formats + 2.1. Uploading server descriptors and extra-info documents + 2.1.1. Server descriptor format + 2.1.2. Extra-info document format + 2.1.3. Nonterminals in server descriptors + 3. Directory authority operation and formats + 3.1. Creating key certificates + 3.2. Accepting server descriptor and extra-info document uploads + 3.3. Computing microdescriptors + 3.4. Exchanging votes + 3.4.1. Vote and consensus status document formats + 3.4.2. Assigning flags in a vote + 3.4.3. Serving bandwidth list files + 3.5. Downloading missing certificates from other directory authorities + 3.6. Downloading server descriptors from other directory authorities + 3.7. Downloading extra-info documents from other directory authorities + 3.8. Computing a consensus from a set of votes + 3.8.0.1. Deciding which Ids to include. + 3.8.0.2. Deciding which descriptors to include + 3.8.1. Forward compatibility + 3.8.2. Encoding port lists + 3.8.3. Computing Bandwidth Weights + 3.9. Computing consensus flavors + 3.9.1. ns consensus + 3.9.2. Microdescriptor consensus + 3.10. Exchanging detached signatures + 3.11. Publishing the signed consensus + 4. Directory cache operation + 4.1. Downloading consensus status documents from directory authorities + 4.2. Downloading server descriptors from directory authorities + 4.3. Downloading microdescriptors from directory authorities + 4.4. Downloading extra-info documents from directory authorities + 4.5. Consensus diffs + 4.5.1. Consensus diff format + 4.5.2. Serving and requesting diff + 4.6 Retrying failed downloads + 5. Client operation + 5.1. Downloading network-status documents + 5.2. Downloading server descriptors or microdescriptors + 5.3. Downloading extra-info documents + 5.4. Using directory information + 5.4.1. Choosing routers for circuits. + 5.4.2. Managing naming + 5.4.3. Software versions + 5.4.4. Warning about a router's status. + 5.5. Retrying failed downloads + 6. Standards compliance + 6.1. HTTP headers + 6.2. HTTP status codes + A. Consensus-negotiation timeline. + B. General-use HTTP URLs + C. Converting a curve25519 public key to an ed25519 public key + D. Inferring missing proto lines. + E. Limited ed diff format +``` + diff --git a/spec/dir-spec/accepting-server-descriptor-extra-info-document.md b/spec/dir-spec/accepting-server-descriptor-extra-info-document.md new file mode 100644 index 0000000..aa69ea9 --- /dev/null +++ b/spec/dir-spec/accepting-server-descriptor-extra-info-document.md @@ -0,0 +1,52 @@ + +## Accepting server descriptor and extra-info document uploads + +When a router posts a signed descriptor to a directory authority, the +authority first checks whether it is well-formed and correctly +self-signed. If it is, the authority next verifies that the nickname +in question is not already assigned to a router with a different +public key. +Finally, the authority MAY check that the router is not blacklisted +because of its key, IP, or another reason. + +An authority also keeps a record of all the Ed25519/RSA1024 +identity key pairs that it has seen before. It rejects any +descriptor that has a known Ed/RSA identity key that it has +already seen accompanied by a different RSA/Ed identity key +in an older descriptor. + +At a future date, authorities will begin rejecting all +descriptors whose RSA key was previously accompanied by an +Ed25519 key, if the descriptor does not list an Ed25519 key. + +At a future date, authorities will begin rejecting all descriptors +that do not list an Ed25519 key. + +If the descriptor passes these tests, and the authority does not already +have a descriptor for a router with this public key, it accepts the +descriptor and remembers it. + +If the authority _does_ have a descriptor with the same public key, the +newly uploaded descriptor is remembered if its publication time is more +recent than the most recent old descriptor for that router, and either: + +```text + - There are non-cosmetic differences between the old descriptor and the + new one. + - Enough time has passed between the descriptors' publication times. + (Currently, 2 hours.) +``` + +Differences between server descriptors are "non-cosmetic" if they would be +sufficient to force an upload as described in section 2.1 above. + +Note that the "cosmetic difference" test only applies to uploaded +descriptors, not to descriptors that the authority downloads from other +authorities. + +When a router posts a signed extra-info document to a directory authority, +the authority again checks it for well-formedness and correct signature, +and checks that its matches the extra-info-digest in some router +descriptor that it believes is currently useful. If so, it accepts it and +stores it and serves it as requested. If not, it drops it. + diff --git a/spec/dir-spec/assigning-flags-vote.md b/spec/dir-spec/assigning-flags-vote.md new file mode 100644 index 0000000..309b025 --- /dev/null +++ b/spec/dir-spec/assigning-flags-vote.md @@ -0,0 +1,172 @@ + +### Assigning flags in a vote + +(This section describes how directory authorities choose which status +flags to apply to routers. Later directory authorities MAY do things +differently, so long as clients keep working well. Clients MUST NOT +depend on the exact behaviors in this section.) + +In the below definitions, a router is considered "active" if it is +running, valid, and not hibernating. + +When we speak of a router's bandwidth in this section, we mean either +its measured bandwidth, or its advertised bandwidth. If a sufficient +threshold (configurable with MinMeasuredBWsForAuthToIgnoreAdvertised, +500 by default) of routers have measured bandwidth values, then the +authority bases flags on _measured_ bandwidths, and treats nodes with +non-measured bandwidths as if their bandwidths were zero. Otherwise, +it uses measured bandwidths for nodes that have them, and advertised +bandwidths for other nodes. + +When computing thresholds based on percentiles of nodes, an authority +only considers nodes that are active, that have not been +omitted as a sybil (see below), and whose bandwidth is at least +4 KB. Nodes that don't meet these criteria do not influence any +threshold calculations (including calculation of stability and uptime +and bandwidth thresholds) and also do not have their Exit status +change. + +"Valid" -- a router is 'Valid' if it is running a version of Tor not +known to be broken, and the directory authority has not blacklisted +it as suspicious. + +```text + "Named" -- + "Unnamed" -- Directory authorities no longer assign these flags. + They were once used to determine whether a relay's nickname was + canonically linked to its public key. +``` + +"Running" -- A router is 'Running' if the authority managed to connect to +it successfully within the last 45 minutes on all its published ORPorts. +Authorities check reachability on: + +```text + * the IPv4 ORPort in the "r" line, and + * the IPv6 ORPort considered for the "a" line, if: + * the router advertises at least one IPv6 ORPort, and + * AuthDirHasIPv6Connectivity 1 is set on the authority. +``` + +A minority of voting authorities that set AuthDirHasIPv6Connectivity will +drop unreachable IPv6 ORPorts from the full consensus. Consensus method 27 +in 0.3.3.x puts IPv6 ORPorts in the microdesc consensus, so that +authorities can drop unreachable IPv6 ORPorts from all consensus flavors. +Consensus method 28 removes IPv6 ORPorts from microdescriptors. + +"Stable" -- A router is 'Stable' if it is active, and either its Weighted +MTBF is at least the median for known active routers or its Weighted MTBF +corresponds to at least 7 days. Routers are never called Stable if they are +running a version of Tor known to drop circuits stupidly. (0.1.1.10-alpha +through 0.1.1.16-rc are stupid this way.) + +To calculate weighted MTBF, compute the weighted mean of the lengths +of all intervals when the router was observed to be up, weighting +intervals by $\alpha^n$, where $n$ is the amount of time that has +passed since the interval ended, and $\alpha$ is chosen so that +measurements over approximately one month old no longer influence the +weighted MTBF much. + +[XXXX what happens when we have less than 4 days of MTBF info.] + +"Exit" -- A router is called an 'Exit' iff it allows exits to at +least one /8 address space on each of ports 80 and 443. (Up until +Tor version 0.3.2, the flag was assigned if relays exit to at least +two of the ports 80, 443, and 6667.) + +"Fast" -- A router is 'Fast' if it is active, and its bandwidth is either in +the top 7/8ths for known active routers or at least 100KB/s. + +"Guard" -- A router is a possible Guard if all of the following apply: + +```text + - It is Fast, + - It is Stable, + - Its Weighted Fractional Uptime is at least the median for "familiar" + active routers, + - It is "familiar", + - Its bandwidth is at least AuthDirGuardBWGuarantee (if set, 2 MB by + default), OR its bandwidth is among the 25% fastest relays, + - It qualifies for the V2Dir flag as described below (this + constraint was added in 0.3.3.x, because in 0.3.0.x clients + started avoiding guards that didn't also have the V2Dir flag). +``` + +To calculate weighted fractional uptime, compute the fraction +of time that the router is up in any given day, weighting so that +downtime and uptime in the past counts less. + +A node is 'familiar' if 1/8 of all active nodes have appeared more +recently than it, OR it has been around for a few weeks. + +"Authority" -- A router is called an 'Authority' if the authority +generating the network-status document believes it is an authority. + +"V2Dir" -- A router supports the v2 directory protocol or higher if it has +an open directory port OR a tunnelled-dir-server line in its router +descriptor, and it is running a version of the directory +protocol that supports the functionality clients need. (Currently, every +supported version of Tor supports the functionality that clients need, +but some relays might set "DirCache 0" or set really low rate limiting, +making them unqualified to be a directory mirror, i.e. they will omit +the tunnelled-dir-server line from their descriptor.) + +"HSDir" -- A router is a v2 hidden service directory if it stores and +serves v2 hidden service descriptors, has the Stable and Fast flag, and the +authority believes that it's been up for at least 96 hours (or the current +value of MinUptimeHidServDirectoryV2). + +"MiddleOnly" -- An authority should vote for this flag if it believes +that a relay is unsuitable for use except as a middle relay. When +voting for this flag, the authority should also vote against "Exit", +"Guard", "HsDir", and "V2Dir". When voting for this flag, if the +authority votes on the "BadExit" flag, the authority should vote in +favor of "BadExit". (This flag was added in 0.4.7.2-alpha.) + +"NoEdConsensus" -- authorities should not vote on this flag; it is +produced as part of the consensus for consensus method 22 or later. + +"StaleDesc" -- authorities should vote to assign this flag if the +published time on the descriptor is over 18 hours in the past. (This flag +was added in 0.4.0.1-alpha.) + +"Sybil" -- authorities SHOULD NOT accept more than 2 relays on a single IP. +If this happens, the authority *should* vote for the excess relays, but +should omit the Running or Valid flags and instead should assign the "Sybil" +flag. When there are more than 2 (or AuthDirMaxServersPerAddr) relays to +choose from, authorities should first prefer authorities to non-authorities, +then prefer Running to non-Running, and then prefer high-bandwidth to +low-bandwidth relays. In this comparison, measured bandwidth is used unless +it is not present for a router, in which case advertised bandwidth is used. + +Thus, the network-status vote includes all non-blacklisted, +non-expired, non-superseded descriptors. + +The bandwidth in a "w" line should be taken as the best estimate +of the router's actual capacity that the authority has. For now, +this should be the lesser of the observed bandwidth and bandwidth +rate limit from the server descriptor. It is given in kilobytes +per second, and capped at some arbitrary value (currently 10 MB/s). + +The Measured= keyword on a "w" line vote is currently computed +by multiplying the previous published consensus bandwidth by the +ratio of the measured average node stream capacity to the network +average. If 3 or more authorities provide a Measured= keyword for +a router, the authorities produce a consensus containing a "w" +Bandwidth= keyword equal to the median of the Measured= votes. + +As a special case, if the "w" line in a vote is about a relay with the +Authority flag, it should not include a Measured= keyword. The goal is +to leave such relays marked as Unmeasured, so they can reserve their +attention for authority-specific activities. "w" lines for votes about +authorities may include the bandwidth authority's measurement using +a different keyword, e.g. MeasuredButAuthority=, so it can still be +reported and recorded for posterity. + +The ports listed in a "p" line should be taken as those ports for +which the router's exit policy permits 'most' addresses, ignoring any +accept not for all addresses, ignoring all rejects for private +netblocks. "Most" addresses are permitted if no more than 2^25 +IPv4 addresses (two /8 networks) were blocked. The list is encoded +as described in section 3.8.2. + diff --git a/spec/dir-spec/client-operation.md b/spec/dir-spec/client-operation.md new file mode 100644 index 0000000..41942c6 --- /dev/null +++ b/spec/dir-spec/client-operation.md @@ -0,0 +1,292 @@ + +# Client operation + +Every Tor that is not a directory server (that is, those that do +not have a DirPort set) implements this section. + + +## Downloading network-status documents + +Each client maintains a list of directory authorities. Insofar as +possible, clients SHOULD all use the same list. + +```text + [Newer versions of Tor (0.2.8.1-alpha and later): + Each client also maintains a list of default fallback directory mirrors + (fallbacks). Each released version of Tor MAY have a different list, + depending on the mirrors that satisfy the fallback directory criteria at + release time.] +``` + +Clients try to have a live consensus network-status document at all times. +A network-status document is "live" if the time in its valid-after field +has passed, and the time in its valid-until field has not passed. + +When a client has no consensus network-status document, it downloads it +from a randomly chosen fallback directory mirror or authority. Clients +prefer fallbacks to authorities, trying them earlier and more frequently. +In all other cases, the client downloads from caches randomly chosen from +among those believed to be V3 directory servers. (This information comes +from the network-status documents.) + +After receiving any response client MUST discard any network-status +documents that it did not request. + +On failure, the client waits briefly, then tries that network-status +document again from another cache. The client does not build circuits +until it has a live network-status consensus document, and it has +descriptors for a significant proportion of the routers that it believes +are running (this is configurable using torrc options and consensus +parameters). + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. When bootstrap completes, Tor will be ready + to handle an application requesting an exit circuit to services like the + World Wide Web. +``` + +If the consensus does not contain Exits, Tor will only build internal +circuits. In this case, earlier statuses will have included "internal" +as indicated above. When bootstrap completes, Tor will be ready to handle +an application requesting an internal circuit to hidden services at +".onion" addresses. + +If a future consensus contains Exits, exit circuits may become available.] + +(Note: clients can and should pick caches based on the network-status +information they have: once they have first fetched network-status info +from an authority or fallback, they should not need to go to the authority +directly again, and should only choose the fallback at random, based on its +consensus weight in the current consensus.) + +To avoid swarming the caches whenever a consensus expires, the +clients download new consensuses at a randomly chosen time after the +caches are expected to have a fresh consensus, but before their +consensus will expire. (This time is chosen uniformly at random from +the interval between the time 3/4 into the first interval after the +consensus is no longer fresh, and 7/8 of the time remaining after +that before the consensus is invalid.) + +```text + [For example, if a client has a consensus that became valid at 1:00, + and is fresh until 2:00, and expires at 4:00, that client will fetch + a new consensus at a random time between 2:45 and 3:50, since 3/4 + of the one-hour interval is 45 minutes, and 7/8 of the remaining 75 + minutes is 65 minutes.] +``` + +Clients may choose to download the microdescriptor consensus instead +of the general network status consensus. In that case they should use +the same update strategy as for the normal consensus. They should not +download more than one consensus flavor. + +When a client does not have a live consensus, it will generally use the +most recent consensus it has if that consensus is "reasonably live". A +"reasonably live" consensus is one that expired less than 24 hours ago. + + +## Downloading server descriptors or microdescriptors + +Clients try to have the best descriptor for each router. A descriptor is +"best" if: + +* It is listed in the consensus network-status document. + +Periodically (currently every 10 seconds) clients check whether there are +any "downloadable" descriptors. A descriptor is downloadable if: + +```text + - It is the "best" descriptor for some router. + - The descriptor was published at least 10 minutes in the past. + (This prevents clients from trying to fetch descriptors that the + mirrors have probably not yet retrieved and cached.) + - The client does not currently have it. + - The client is not currently trying to download it. + - The client would not discard it immediately upon receiving it. + - The client thinks it is running and valid (see section 5.4.1 below). +``` + +If at least 16 known routers have downloadable descriptors, or if +enough time (currently 10 minutes) has passed since the last time the +client tried to download descriptors, it launches requests for all +downloadable descriptors. + +When downloading multiple server descriptors, the client chooses multiple +mirrors so that: + +```text + - At least 3 different mirrors are used, except when this would result + in more than one request for under 4 descriptors. + - No more than 128 descriptors are requested from a single mirror. + - Otherwise, as few mirrors as possible are used. + After choosing mirrors, the client divides the descriptors among them + randomly. +``` + +After receiving any response the client MUST discard any descriptors that +it did not request. + +When a descriptor download fails, the client notes it, and does not +consider the descriptor downloadable again until a certain amount of time +has passed. (Currently 0 seconds for the first failure, 60 seconds for the +second, 5 minutes for the third, 10 minutes for the fourth, and 1 day +thereafter.) Periodically (currently once an hour) clients reset the +failure count. + +Clients retain the most recent descriptor they have downloaded for each +router so long as it is listed in the consensus. If it is not listed, +they keep it so long as it is not too old (currently, ROUTER_MAX_AGE=48 +hours) and no better router descriptor has been downloaded for the same +relay. Caches retain descriptors until they are at least +OLD_ROUTER_DESC_MAX_AGE=5 days old. + +Clients which chose to download the microdescriptor consensus instead +of the general consensus must download the referenced microdescriptors +instead of server descriptors. Clients fetch and cache +microdescriptors preemptively from dir mirrors when starting up, like +they currently fetch descriptors. After bootstrapping, clients only +need to fetch the microdescriptors that have changed. + +When a client gets a new microdescriptor consensus, it looks to see if +there are any microdescriptors it needs to learn, and launches a request +for them. + +Clients maintain a cache of microdescriptors along with metadata like +when it was last referenced by a consensus, and which identity key +it corresponds to. They keep a microdescriptor until it hasn't been +mentioned in any consensus for a week. Future clients might cache them +for longer or shorter times. + + +## Downloading extra-info documents + +Any client that uses extra-info documents should implement this +section. + +Note that generally, clients don't need extra-info documents. + +Periodically, the Tor instance checks whether it is missing any extra-info +documents: in other words, if it has any server descriptors with an +extra-info-digest field that does not match any of the extra-info +documents currently held. If so, it downloads whatever extra-info +documents are missing. Clients try to download from caches. +We follow the same splitting and back-off rules as in section 5.2. + + +## Using directory information + +[XXX This subsection really belongs in path-spec.txt, not here. -KL] + +Everyone besides directory authorities uses the approaches in this section +to decide which relays to use and what their keys are likely to be. +(Directory authorities just believe their own opinions, as in section 3.4.2 +above.) + + +### Choosing routers for circuits. + +Circuits SHOULD NOT be built until the client has enough directory +information: a live consensus network status [XXXX fallback?] and +descriptors for at least 1/4 of the relays believed to be running. + +A relay is "listed" if it is included by the consensus network-status +document. Clients SHOULD NOT use unlisted relays. + +These flags are used as follows: + +```text + - Clients SHOULD NOT use non-'Valid' or non-'Running' routers unless + requested to do so. + + - Clients SHOULD NOT use non-'Fast' routers for any purpose other than + very-low-bandwidth circuits (such as introduction circuits). + + - Clients SHOULD NOT use non-'Stable' routers for circuits that are + likely to need to be open for a very long time (such as those used for + IRC or SSH connections). + + - Clients SHOULD NOT choose non-'Guard' nodes when picking entry guard + nodes. + + See the "path-spec.txt" document for more details. +``` + + +### Managing naming + +(This section is removed; authorities no longer assign the 'Named' flag.) + + +### Software versions + +An implementation of Tor SHOULD warn when it has fetched a consensus +network-status, and it is running a software version not listed. + + +### Warning about a router's status. + +(This section is removed; authorities no longer assign the 'Named' flag.) + + +## Retrying failed downloads + +This section applies to caches as well as to clients. + +When a client fails to download a resource (a consensus, a router +descriptor, a microdescriptor, etc) it waits for a certain amount of +time before retrying the download. To determine the amount of time +to wait, clients use a randomized exponential backoff algorithm. +(Specifically, they use a variation of the "decorrelated jitter" +algorithm from +https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ .) + +The specific formula used to compute the 'i+1'th delay is: + +```text + Delay_{i+1} = MIN(cap, random_between(lower_bound, upper_bound))) + where upper_bound = MAX(lower_bound+1, Delay_i * 3) + lower_bound = MAX(1, base_delay). +``` + +The value of 'cap' is set to INT_MAX; the value of 'base_delay' +depends on what is being downloaded, whether the client is fully +bootstrapped, how the client is configured, and where it is +downloading from. Current base_delay values are: + +```text + Consensus objects, as a non-bridge cache: + 0 (TestingServerConsensusDownloadInitialDelay) + + Consensus objects, as a client or bridge that has bootstrapped: + 0 (TestingClientConsensusDownloadInitialDelay) + + Consensus objects, as a client or bridge that is bootstrapping, + when connecting to an authority because no "fallback" caches are + known: + 0 (ClientBootstrapConsensusAuthorityOnlyDownloadInitialDelay) + + Consensus objects, as a client or bridge that is bootstrapping, + when "fallback" caches are known but connecting to an authority + anyway: + 6 (ClientBootstrapConsensusAuthorityDownloadInitialDelay) + + Consensus objects, as a client or bridge that is bootstrapping, + when downloading from a "fallback" cache. + 0 (ClientBootstrapConsensusFallbackDownloadInitialDelay) + + Bridge descriptors, as a bridge-using client when at least one bridge + is usable: + 10800 (TestingBridgeDownloadInitialDelay) + + Bridge descriptors, otherwise: + 0 (TestingBridgeBootstrapDownloadInitialDelay) + + Other objects, as cache or authority: + 0 (TestingServerDownloadInitialDelay) + + Other objects, as client: + 0 (TestingClientDownloadInitialDelay) +``` + diff --git a/spec/dir-spec/computing-microdescriptors.md b/spec/dir-spec/computing-microdescriptors.md new file mode 100644 index 0000000..29ddd48 --- /dev/null +++ b/spec/dir-spec/computing-microdescriptors.md @@ -0,0 +1,169 @@ + +## Computing microdescriptors + +Microdescriptors are a stripped-down version of server descriptors +generated by the directory authorities which may additionally contain +authority-generated information. Microdescriptors contain only the +most relevant parts that clients care about. Microdescriptors are +expected to be relatively static and only change about once per week. +Microdescriptors do not contain any information that clients need to +use to decide which servers to fetch information about, or which +servers to fetch information from. + +Microdescriptors are a straight transform from the server descriptor +and the consensus method. Microdescriptors have no header or footer. +Microdescriptors are identified by the hash of its concatenated +elements without a signature by the router. Microdescriptors do not +contain any version information, because their version is determined +by the consensus method. + +Starting with consensus method 8, microdescriptors contain the +following elements taken from or based on the server descriptor. Order +matters here, because different directory authorities must be able to +transform a given server descriptor and consensus method into the exact +same microdescriptor. + +"onion-key" NL a public key in PEM format + +[Exactly once, at start] +[No extra arguments] + +The "onion-key" element as specified in section 2.1.1. + +When generating microdescriptors for consensus method 30 or later, +the trailing = sign must be absent. For consensus method 29 or +earlier, the trailing = sign must be present. + +"ntor-onion-key" SP base-64-encoded-key NL + +[Exactly once] + +The "ntor-onion-key" element as specified in section 2.1.1. + +(Only included when generating microdescriptors for +consensus-method 16 or later.) + +[Before Tor 0.4.5.1-alpha, this field was optional.] + +"a" SP address ":" port NL + +[Any number] + +Additional advertised addresses for the OR. + +Present currently only if the OR advertises at least one IPv6 +address; currently, the first address is included and all others are +omitted. Any other IPv4 or IPv6 addresses should be ignored. + +Address and port are as for "or-address" as specified in +section 2.1.1. + +(Only included when generating microdescriptors for +consensus-methods 14 to 27.) + +"family" names NL + +[At most once] + +The "family" element as specified in section 2.1.1. + +When generating microdescriptors for consensus method 29 or later, +the following canonicalization algorithm is applied to improve +compression: + +```text + For all entries of the form $hexid=name or $hexid~name, + remove the =name or ~name portion. + + Remove all entries of the form $hexid, where hexid is not + 40 hexadecimal characters long. + + If an entry is a valid nickname, put it into lower case. + + If an entry is a valid $hexid, put it into upper case. + + If there are any entries, add a single $hexid entry for + the relay in question, so that it is a member of its own + family. + + Sort all entries in lexical order. + + Remove duplicate entries. +``` + +(Note that if an entry is not of the form "nickname", "$hexid", +"$hexid=nickname" or "$hexid~nickname", then it will be unchanged: +this is what makes the algorithm forward-compatible.) + +"p" SP ("accept" / "reject") SP PortList NL + +[Exactly once.] + +The exit-policy summary as specified in sections 3.4.1 and 3.8.2. + +[With microdescriptors, clients don't learn exact exit policies: +clients can only guess whether a relay accepts their request, try the +BEGIN request, and might get end-reason-exit-policy if they guessed +wrong, in which case they'll have to try elsewhere.] + +[In consensus methods before 5, this line was omitted.] + +"p6" SP ("accept" / "reject") SP PortList NL + +[At most once] + +The IPv6 exit policy summary as specified in sections 3.4.1 and +3.8.2. A missing "p6" line is equivalent to "p6 reject 1-65535". + +(Only included when generating microdescriptors for +consensus-method 15 or later.) + +"id" SP "rsa1024" SP base64-encoded-identity-digest NL + +[At most once] + +The node identity digest (as described in tor-spec.txt), base64 +encoded, without trailing =s. This line is included to prevent +collisions between microdescriptors. + +Implementations SHOULD ignore these lines: they are +added to microdescriptors only to prevent collisions. + +(Only included when generating microdescriptors for +consensus-method 18 or later.) + +"id" SP "ed25519" SP base64-encoded-ed25519-identity NL + +[At most once] + +The node's master Ed25519 identity key, base64 encoded, +without trailing =s. + +All implementations MUST ignore this key for any microdescriptor +whose corresponding entry in the consensus includes the +'NoEdConsensus' flag. + +(Only included when generating microdescriptors for +consensus-method 21 or later.) + +"id" SP keytype ... NL + +[At most once per distinct keytype.] + +Implementations MUST ignore "id" lines with unrecognized +key-types in place of "rsa1024" or "ed25519" + +"pr" SP Entries NL + +[Exactly once.] + +The "proto" element as specified in section 2.1.1. + +[Before Tor 0.4.5.1-alpha, this field was optional.] + +(Note that with microdescriptors, clients do not learn the RSA identity of +their routers: they only learn a hash of the RSA identity key. This is +all they need to confirm the actual identity key when doing a TLS +handshake, and all they need to put the identity key digest in their +CREATE cells.) + diff --git a/spec/dir-spec/consensus-negotiation-timeline.md b/spec/dir-spec/consensus-negotiation-timeline.md new file mode 100644 index 0000000..9e17724 --- /dev/null +++ b/spec/dir-spec/consensus-negotiation-timeline.md @@ -0,0 +1,16 @@ + +# Consensus-negotiation timeline. + +```text + Period begins: this is the Published time. + Everybody sends votes + Reconciliation: everybody tries to fetch missing votes. + consensus may exist at this point. + End of voting period: + everyone swaps signatures. + Now it's okay for caches to download + Now it's okay for clients to download. + + Valid-after/valid-until switchover +``` + diff --git a/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md b/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md new file mode 100644 index 0000000..7541b54 --- /dev/null +++ b/spec/dir-spec/converting-curve25519-public-key-to-ed25519-public.md @@ -0,0 +1,77 @@ + +# Converting a curve25519 public key to an ed25519 public key + +Given an X25519 key, that is, an affine point (u,v) on the +Montgomery curve defined by + +bv^2 = u(u^2 + au +1) + +where + +```text + a = 486662 + b = 1 +``` + +and comprised of the compressed form (i.e. consisting of only the +u-coordinate), we can retrieve the y-coordinate of the affine point +(x,y) on the twisted Edwards form of the curve defined by + +-x^2 + y^2 = 1 + d x^2 y^2 + +where + +d = - 121665/121666 + +by computing + +y = (u-1)/(u+1). + +and then we can apply the usual curve25519 twisted Edwards point +decompression algorithm to find _an_ x-coordinate of an affine +twisted Edwards point to check signatures with. Signing keys for +ed25519 are compressed curve points in twisted Edwards form (so a +y-coordinate and the sign of the x-coordinate), and X25519 keys are +compressed curve points in Montgomery form (i.e. a u-coordinate). + +However, note that compressed point in Montgomery form neglects to +encode what the sign of the corresponding twisted Edwards +x-coordinate would be. Thus, we need the sign of the x-coordinate +to do this operation; otherwise, we'll have two possible +x-coordinates that might have correspond to the ed25519 public key. + +To get the sign, the easiest way is to take the corresponding +private key, feed it to the ed25519 public key generation +algorithm, and see what the sign is. + +[Recomputing the sign bit from the private key every time sounds +rather strange and inefficient to me… —isis] + +Note that in addition to its coordinates, an expanded Ed25519 private key +also has a 32-byte random value, "prefix", used to compute internal `r` +values in the signature. For security, this prefix value should be +derived deterministically from the curve25519 key. The Tor +implementation derives it as SHA512(private_key | STR)[0..32], where +STR is the nul-terminated string: + +"Derive high part of ed25519 key from curve25519 key\0" + +On the client side, where there is no access to the curve25519 private +keys, one may use the curve25519 public key's Montgomery u-coordinate to +recover the Montgomery v-coordinate by computing the right-hand side of +the Montgomery curve equation: + +bv^2 = u(u^2 + au +1) + +where + +```text + a = 486662 + b = 1 +``` + +Then, knowing the intended sign of the Edwards x-coordinate, one +may recover said x-coordinate by computing: + +x = (u/v) * sqrt(-a - 2) + diff --git a/spec/dir-spec/creating-key-certificates.md b/spec/dir-spec/creating-key-certificates.md new file mode 100644 index 0000000..4c25867 --- /dev/null +++ b/spec/dir-spec/creating-key-certificates.md @@ -0,0 +1,90 @@ + +## Creating key certificates + +Key certificates consist of the following items: + +"dir-key-certificate-version" version NL + +[At start, exactly once.] + +Determines the version of the key certificate. MUST be "3" for +the protocol described in this document. Implementations MUST +reject formats they don't understand. + +```text + "dir-address" IPPort NL + [At most once] + + An IP:Port for this authority's directory port. + + "fingerprint" fingerprint NL + + [Exactly once.] +``` + +Hexadecimal encoding without spaces based on the authority's +identity key. + +"dir-identity-key" NL a public key in PEM format + +[Exactly once.] +[No extra arguments] + +The long-term authority identity key for this authority. This key +SHOULD be at least 2048 bits long; it MUST NOT be shorter than +1024 bits. + +"dir-key-published" YYYY-MM-DD HH:MM:SS NL + +[Exactly once.] + +The time (in UTC) when this document and corresponding key were +last generated. + +Implementations SHOULD reject certificates that are published +too far in the future, though they MAY tolerate some clock skew. + +"dir-key-expires" YYYY-MM-DD HH:MM:SS NL + +[Exactly once.] + +A time (in UTC) after which this key is no longer valid. + +Implementations SHOULD reject expired certificates, though they +MAY tolerate some clock skew. + +"dir-signing-key" NL a key in PEM format + +[Exactly once.] +[No extra arguments] + +The directory server's public signing key. This key MUST be at +least 1024 bits, and MAY be longer. + +"dir-key-crosscert" NL CrossSignature NL + +[Exactly once.] +[No extra arguments] + +CrossSignature is a signature, made using the certificate's signing +key, of the digest of the PKCS1-padded hash of the certificate's +identity key. For backward compatibility with broken versions of the +parser, we wrap the base64-encoded signature in -----BEGIN ID +SIGNATURE---- and -----END ID SIGNATURE----- tags. Implementations +MUST allow the "ID " portion to be omitted, however. + +Implementations MUST verify that the signature is a correct signature +of the hash of the identity key using the signing key. + +"dir-key-certification" NL Signature NL + +[At end, exactly once.] +[No extra arguments] + +A document signature as documented in section 1.3, using the +initial item "dir-key-certificate-version" and the final item +"dir-key-certification", signed with the authority identity key. + +Authorities MUST generate a new signing key and corresponding +certificate before the key expires. + diff --git a/spec/dir-spec/directory-authority-operation-formats.md b/spec/dir-spec/directory-authority-operation-formats.md new file mode 100644 index 0000000..9e171cf --- /dev/null +++ b/spec/dir-spec/directory-authority-operation-formats.md @@ -0,0 +1,10 @@ + +# Directory authority operation and formats + +Every authority has two keys used in this protocol: a signing key, and +an authority identity key. (Authorities also have a router identity +key used in their role as a router and by earlier versions of the +directory protocol.) The identity key is used from time to time to +sign new key certificates using new signing keys; it is very sensitive. +The signing key is used to sign key certificates and status documents. + diff --git a/spec/dir-spec/directory-cache-operation.md b/spec/dir-spec/directory-cache-operation.md new file mode 100644 index 0000000..6003d64 --- /dev/null +++ b/spec/dir-spec/directory-cache-operation.md @@ -0,0 +1,157 @@ + +# Directory cache operation + +All directory caches implement this section, except as noted. + + +## Downloading consensus status documents from directory authorities + +All directory caches try to keep a recent +network-status consensus document to serve to clients. A cache ALWAYS +downloads a network-status consensus if any of the following are true: + +- The cache has no consensus document. +- The cache's consensus document is no longer valid. + +Otherwise, the cache downloads a new consensus document at a randomly +chosen time in the first half-interval after its current consensus +stops being fresh. (This time is chosen at random to avoid swarming +the authorities at the start of each period. The interval size is +inferred from the difference between the valid-after time and the +fresh-until time on the consensus.) + +```text + [For example, if a cache has a consensus that became valid at 1:00, + and is fresh until 2:00, that cache will fetch a new consensus at + a random time between 2:00 and 2:30.] +``` + +Directory caches also fetch consensus flavors from the authorities. +Caches check the correctness of consensus flavors, but do not check +anything about an unrecognized consensus document beyond its digest and +length. Caches serve all consensus flavors from the same locations as +the directory authorities. + + +## Downloading server descriptors from directory authorities + +Periodically (currently, every 10 seconds), directory caches check +whether there are any specific descriptors that they do not have and that +they are not currently trying to download. Caches identify these +descriptors by hash in the recent network-status consensus documents. + +If so, the directory cache launches requests to the authorities for these +descriptors. + +If one of these downloads fails, we do not try to download that descriptor +from the authority that failed to serve it again unless we receive a newer +network-status consensus that lists the same descriptor. + +Directory caches must potentially cache multiple descriptors for each +router. Caches must not discard any descriptor listed by any recent +consensus. If there is enough space to store additional descriptors, +caches SHOULD try to hold those which clients are likely to download the +most. (Currently, this is judged based on the interval for which each +descriptor seemed newest.) + +[XXXX define recent] + + +## Downloading microdescriptors from directory authorities + +Directory mirrors should fetch, cache, and serve each microdescriptor +from the authorities. + +The microdescriptors with base64 hashes ,, are available +at: + +http:///tor/micro/d/--[.z] + + are base64 encoded with trailing =s omitted for size and for +consistency with the microdescriptor consensus format. -s are used +instead of +s to separate items, since the + character is used in +base64 encoding. + +Directory mirrors should check to make sure that the microdescriptors +they're about to serve match the right hashes (either the hashes from +the fetch URL or the hashes from the consensus, respectively). + +(NOTE: Due to squid proxy url limitations at most 92 microdescriptor hashes +can be retrieved in a single request.) + + +## Downloading extra-info documents from directory authorities + +Any cache that chooses to cache extra-info documents should implement this +section. + +Periodically, the Tor instance checks whether it is missing any extra-info +documents: in other words, if it has any server descriptors with an +extra-info-digest field that does not match any of the extra-info +documents currently held. If so, it downloads whatever extra-info +documents are missing. Caches download from authorities. We follow the +same splitting and back-off rules as in section 4.2. + + +## Consensus diffs + +Instead of downloading an entire consensus, clients may download +a "diff" document containing an ed-style diff from a previous +consensus document. Caches (and authorities) make these diffs as +they learn about new consensuses. To do so, they must store a +record of older consensuses. + +(Support for consensus diffs was added in 0.3.1.1-alpha, and is +advertised with the DirCache protocol version "2" or later.) + + +### Consensus diff format + +Consensus diffs are formatted as follows: + +The first line is "network-status-diff-version 1" NL + +The second line is + +"hash" SP FromDigest SP ToDigest NL + +where FromDigest is the hex-encoded SHA3-256 digest of the _signed +part_ of the consensus that the diff should be applied to, and +ToDigest is the hex-encoded SHA3-256 digest of the _entire_ +consensus resulting from applying the diff. (See 3.4.1 for +information on that part of a consensus is signed.) + +The third and subsequent lines encode the diff from FromDigest to +ToDigest in a limited subset of the ed diff format, as specified +in appendix E. + + +### Serving and requesting diffs. + +When downloading the current consensus, a client may include an +HTTP header of the form + +X-Or-Diff-From-Consensus: HASH1, HASH2, ... + +where the HASH values are hex-encoded SHA3-256 digests of the +_signed part_ of one or more consensuses that the client knows +about. + +If a cache knows a consensus diff from one of those consensuses +to the most recent consensus of the requested flavor, it may +send that diff instead of the specified consensus. + +Caches also serve diffs from the URIs: + +/tor/status-vote/current/consensus/diff//.z +/tor/status-vote/current/consensus-/diff//.z + +where FLAVOR is the consensus flavor, defaulting to "ns", and +FPRLIST is +-separated list of recognized authority identity +fingerprints as in appendix B. + + +## Retrying failed downloads + +See section 5.5 below; it applies to caches as well as clients. + diff --git a/spec/dir-spec/exchanging-votes.md b/spec/dir-spec/exchanging-votes.md new file mode 100644 index 0000000..7c42ad4 --- /dev/null +++ b/spec/dir-spec/exchanging-votes.md @@ -0,0 +1,53 @@ + +## Exchanging votes + +Authorities divide time into Intervals. Authority administrators SHOULD +try to all pick the same interval length, and SHOULD pick intervals that +are commonly used divisions of time (e.g., 5 minutes, 15 minutes, 30 +minutes, 60 minutes, 90 minutes). Voting intervals SHOULD be chosen to +divide evenly into a 24-hour day. + +Authorities SHOULD act according to interval and delays in the +latest consensus. Lacking a latest consensus, they SHOULD default to a +30-minute Interval, a 5 minute VotingDelay, and a 5 minute DistDelay. + +Authorities MUST take pains to ensure that their clocks remain accurate +within a few seconds. (Running NTP is usually sufficient.) + +The first voting period of each day begins at 00:00 (midnight) UTC. If +the last period of the day would be truncated by one-half or more, it is +merged with the second-to-last period. + +An authority SHOULD publish its vote immediately at the start of each voting +period (minus VoteSeconds+DistSeconds). It does this by making it +available at + +http:///tor/status-vote/next/authority.z + +and sending it in an HTTP POST request to each other authority at the URL + +http:///tor/post/vote + +If, at the start of the voting period, minus DistSeconds, an authority +does not have a current statement from another authority, the first +authority downloads the other's statement. + +Once an authority has a vote from another authority, it makes it available +at + +http:///tor/status-vote/next/.z + +where is the fingerprint of the other authority's identity key. +And at + +http:///tor/status-vote/next/d/.z + +where is the digest of the vote document. + +Also, once an authority receives a vote from another authority, it +examines it for new descriptors and fetches them from that authority. +This may be the only way for an authority to hear about relays that didn't +publish their descriptor to all authorities, and, while it's too late +for the authority to include relays in its current vote, it can include +them in its next vote. See section 3.6 below for details. + diff --git a/spec/dir-spec/extra-info-document-format.md b/spec/dir-spec/extra-info-document-format.md new file mode 100644 index 0000000..34df991 --- /dev/null +++ b/spec/dir-spec/extra-info-document-format.md @@ -0,0 +1,591 @@ + +### Extra-info document format + +Extra-info documents consist of the following items: + +```text + "extra-info" Nickname Fingerprint NL + [At start, exactly once.] +``` + +Identifies what router this is an extra-info descriptor for. +Fingerprint is encoded in hex (using upper-case letters), with +no spaces. + +```text + "identity-ed25519" + [As in router descriptors] + + "published" YYYY-MM-DD HH:MM:SS NL + + [Exactly once.] +``` + +The time, in UTC, when this document (and its corresponding router +descriptor if any) was generated. It MUST match the published time +in the corresponding server descriptor. + +```text + "read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once.] + "write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once.] +``` + +Declare how much bandwidth the OR has used recently. Usage is divided +into intervals of NSEC seconds. The YYYY-MM-DD HH:MM:SS field +defines the end of the most recent interval. The numbers are the +number of bytes used in the most recent intervals, ordered from +oldest to newest. + +These fields include both IPv4 and IPv6 traffic. + +```text + "ipv6-read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + "ipv6-write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] +``` + +Declare how much bandwidth the OR has used recently, on IPv6 +connections. See "read-history" and "write-history" for full details. + +```text + "geoip-db-digest" Digest NL + [At most once.] +``` + +SHA1 digest of the IPv4 GeoIP database file that is used to +resolve IPv4 addresses to country codes. + +```text + "geoip6-db-digest" Digest NL + [At most once.] +``` + +SHA1 digest of the IPv6 GeoIP database file that is used to +resolve IPv6 addresses to country codes. + +("geoip-start-time" YYYY-MM-DD HH:MM:SS NL) +("geoip-client-origins" CC=NUM,CC=NUM,... NL) + +Only generated by bridge routers (see blocking.pdf), and only +when they have been configured with a geoip database. +Non-bridges SHOULD NOT generate these fields. Contains a list +of mappings from two-letter country codes (CC) to the number +of clients that have connected to that bridge from that +country (approximate, and rounded up to the nearest multiple of 8 +in order to hamper traffic analysis). A country is included +only if it has at least one address. The time in +"geoip-start-time" is the time at which we began collecting geoip +statistics. + +"geoip-start-time" and "geoip-client-origins" have been replaced by +"bridge-stats-end" and "bridge-ips" in 0.2.2.4-alpha. The +reason is that the measurement interval with "geoip-stats" as +determined by subtracting "geoip-start-time" from "published" could +have had a variable length, whereas the measurement interval in +0.2.2.4-alpha and later is set to be exactly 24 hours long. In +order to clearly distinguish the new measurement intervals from +the old ones, the new keywords have been introduced. + +```text + "bridge-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "bridge-stats-end" line, as well as any other "bridge-*" line, +is only added when the relay has been running as a bridge for at +least 24 hours. + +```text + "bridge-ips" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +unique IP addresses that have connected from that country to the +bridge and which are no known relays, rounded up to the nearest +multiple of 8. + +```text + "bridge-ip-versions" FAM=NUM,FAM=NUM,... NL + [At most once.] +``` + +List of unique IP addresses that have connected to the bridge +per protocol family. + +```text + "bridge-ip-transports" PT=NUM,PT=NUM,... NL + [At most once.] +``` + +List of mappings from pluggable transport names to the number +of unique IP addresses that have connected using that +pluggable transport. Unobfuscated connections are counted +using the reserved pluggable transport name "" (without +quotes). If we received a connection from a transport proxy +but we couldn't figure out the name of the pluggable +transport, we use the reserved pluggable transport name +"". + +("" and "" are reserved because normal pluggable +transport names MUST match the following regular expression: +"[a-zA-Z_][a-zA-Z0-9_]*" ) + +The pluggable transport name list is sorted into lexically +ascending order. + +If no clients have connected to the bridge yet, we only write +"bridge-ip-transports" to the stats file. + +```text + "dirreq-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "dirreq-stats-end" line, as well as any other "dirreq-*" line, +is only added when the relay has opened its Dir port and after 24 +hours of measuring directory requests. + +```text + "dirreq-v2-ips" CC=NUM,CC=NUM,... NL + [At most once.] + "dirreq-v3-ips" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +unique IP addresses that have connected from that country to +request a v2/v3 network status, rounded up to the nearest multiple +of 8. Only those IP addresses are counted that the directory can +answer with a 200 OK status code. (Note here and below: current Tor +versions, as of 0.2.5.2-alpha, no longer cache or serve v2 +networkstatus documents.) + +```text + "dirreq-v2-reqs" CC=NUM,CC=NUM,... NL + [At most once.] + "dirreq-v3-reqs" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +requests for v2/v3 network statuses from that country, rounded up +to the nearest multiple of 8. Only those requests are counted that +the directory can answer with a 200 OK status code. + +```text + "dirreq-v2-share" NUM% NL + [At most once.] + "dirreq-v3-share" NUM% NL + [At most once.] +``` + +The share of v2/v3 network status requests that the directory +expects to receive from clients based on its advertised bandwidth +compared to the overall network bandwidth capacity. Shares are +formatted in percent with two decimal places. Shares are +calculated as means over the whole 24-hour interval. + +```text + "dirreq-v2-resp" status=NUM,... NL + [At most once.] + "dirreq-v3-resp" status=NUM,... NL + [At most once.] +``` + +List of mappings from response statuses to the number of requests +for v2/v3 network statuses that were answered with that response +status, rounded up to the nearest multiple of 4. Only response +statuses with at least 1 response are reported. New response +statuses can be added at any time. The current list of response +statuses is as follows: + +```text + "ok": a network status request is answered; this number + corresponds to the sum of all requests as reported in + "dirreq-v2-reqs" or "dirreq-v3-reqs", respectively, before + rounding up. + "not-enough-sigs: a version 3 network status is not signed by a + sufficient number of requested authorities. + "unavailable": a requested network status object is unavailable. + "not-found": a requested network status is not found. + "not-modified": a network status has not been modified since the + If-Modified-Since time that is included in the request. + "busy": the directory is busy. + + "dirreq-v2-direct-dl" key=NUM,... NL + [At most once.] + "dirreq-v3-direct-dl" key=NUM,... NL + [At most once.] + "dirreq-v2-tunneled-dl" key=NUM,... NL + [At most once.] + "dirreq-v3-tunneled-dl" key=NUM,... NL + [At most once.] +``` + +List of statistics about possible failures in the download process +of v2/v3 network statuses. Requests are either "direct" +HTTP-encoded requests over the relay's directory port, or +"tunneled" requests using a BEGIN_DIR cell over the relay's OR +port. The list of possible statistics can change, and statistics +can be left out from reporting. The current list of statistics is +as follows: + +Successful downloads and failures: + +```text + "complete": a client has finished the download successfully. + "timeout": a download did not finish within 10 minutes after + starting to send the response. + "running": a download is still running at the end of the + measurement period for less than 10 minutes after starting to + send the response. + + Download times: + + "min", "max": smallest and largest measured bandwidth in B/s. + "d[1-4,6-9]": 1st to 4th and 6th to 9th decile of measured + bandwidth in B/s. For a given decile i, i/10 of all downloads + had a smaller bandwidth than di, and (10-i)/10 of all downloads + had a larger bandwidth than di. + "q[1,3]": 1st and 3rd quartile of measured bandwidth in B/s. One + fourth of all downloads had a smaller bandwidth than q1, one + fourth of all downloads had a larger bandwidth than q3, and the + remaining half of all downloads had a bandwidth between q1 and + q3. + "md": median of measured bandwidth in B/s. Half of the downloads + had a smaller bandwidth than md, the other half had a larger + bandwidth than md. + + "dirreq-read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] + "dirreq-write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM... NL + [At most once] +``` + +Declare how much bandwidth the OR has spent on answering directory +requests. Usage is divided into intervals of NSEC seconds. The +YYYY-MM-DD HH:MM:SS field defines the end of the most recent +interval. The numbers are the number of bytes used in the most +recent intervals, ordered from oldest to newest. + +```text + "entry-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +An "entry-stats-end" line, as well as any other "entry-*" +line, is first added after the relay has been running for at least +24 hours. + +```text + "entry-ips" CC=NUM,CC=NUM,... NL + [At most once.] +``` + +List of mappings from two-letter country codes to the number of +unique IP addresses that have connected from that country to the +relay and which are no known other relays, rounded up to the +nearest multiple of 8. + +```text + "cell-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "cell-stats-end" line, as well as any other "cell-*" line, +is first added after the relay has been running for at least 24 +hours. + +```text + "cell-processed-cells" NUM,...,NUM NL + [At most once.] +``` + +Mean number of processed cells per circuit, subdivided into +deciles of circuits by the number of cells they have processed in +descending order from loudest to quietest circuits. + +```text + "cell-queued-cells" NUM,...,NUM NL + [At most once.] +``` + +Mean number of cells contained in queues by circuit decile. These +means are calculated by 1) determining the mean number of cells in +a single circuit between its creation and its termination and 2) +calculating the mean for all circuits in a given decile as +determined in "cell-processed-cells". Numbers have a precision of +two decimal places. + +Note that this statistic can be inaccurate for circuits that had +queued cells at the start or end of the measurement interval. + +```text + "cell-time-in-queue" NUM,...,NUM NL + [At most once.] +``` + +Mean time cells spend in circuit queues in milliseconds. Times are +calculated by 1) determining the mean time cells spend in the +queue of a single circuit and 2) calculating the mean for all +circuits in a given decile as determined in +"cell-processed-cells". + +Note that this statistic can be inaccurate for circuits that had +queued cells at the start or end of the measurement interval. + +```text + "cell-circuits-per-decile" NUM NL + [At most once.] +``` + +Mean number of circuits that are included in any of the deciles, +rounded up to the next integer. + +```text + "conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL + [At most once] +``` + +Number of connections, split into 10-second intervals, that are +used uni-directionally or bi-directionally as observed in the NSEC +seconds (usually 86400 seconds) before YYYY-MM-DD HH:MM:SS. Every +10 seconds, we determine for every connection whether we read and +wrote less than a threshold of 20 KiB (BELOW), read at least 10 +times more than we wrote (READ), wrote at least 10 times more than +we read (WRITE), or read and wrote more than the threshold, but +not 10 times more in either direction (BOTH). After classifying a +connection, read and write counters are reset for the next +10-second interval. + +This measurement includes both IPv4 and IPv6 connections. + +```text + "ipv6-conn-bi-direct" YYYY-MM-DD HH:MM:SS (NSEC s) BELOW,READ,WRITE,BOTH NL + [At most once] +``` + +Number of IPv6 connections that are used uni-directionally or +bi-directionally. See "conn-bi-direct" for more details. + +```text + "exit-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +An "exit-stats-end" line, as well as any other "exit-*" line, is +first added after the relay has been running for at least 24 hours +and only if the relay permits exiting (where exiting to a single +port and IP address is sufficient). + +```text + "exit-kibibytes-written" port=N,port=N,... NL + [At most once.] + "exit-kibibytes-read" port=N,port=N,... NL + [At most once.] +``` + +List of mappings from ports to the number of kibibytes that the +relay has written to or read from exit connections to that port, +rounded up to the next full kibibyte. Relays may limit the +number of listed ports and subsume any remaining kibibytes under +port "other". + +```text + "exit-streams-opened" port=N,port=N,... NL + [At most once.] +``` + +List of mappings from ports to the number of opened exit streams +to that port, rounded up to the nearest multiple of 4. Relays may +limit the number of listed ports and subsume any remaining opened +streams under port "other". + +```text + "hidserv-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] + "hidserv-v3-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). + +A "hidserv-stats-end" line, as well as any other "hidserv-*" line, +is first added after the relay has been running for at least 24 +hours. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "hidserv-rend-relayed-cells" SP NUM SP key=val SP key=val ... NL + [At most once.] + "hidserv-rend-v3-relayed-cells" SP NUM SP key=val SP key=val ... NL + [At most once.] +``` + +Approximate number of RELAY cells seen in either direction on a +circuit after receiving and successfully processing a RENDEZVOUS1 +cell. + +The original measurement value is obfuscated in several steps: +first, it is rounded up to the nearest multiple of 'bin_size' +which is reported in the key=val part of this line; second, a +(possibly negative) noise value is added to the result of the +first step by randomly sampling from a Laplace distribution with +mu = 0 and b = (delta_f / epsilon) with 'delta_f' and 'epsilon' +being reported in the key=val part, too; third, the result of the +previous obfuscation steps is truncated to the next smaller +integer and included as 'NUM'. Note that the overall reported +value can be negative. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "hidserv-dir-onions-seen" SP NUM SP key=val SP key=val ... NL + [At most once.] + "hidserv-dir-v3-onions-seen" SP NUM SP key=val SP key=val ... NL + [At most once.] +``` + +Approximate number of unique hidden-service identities seen in +descriptors published to and accepted by this hidden-service +directory. + +The original measurement value is obfuscated in the same way as +the 'NUM' value reported in "hidserv-rend-relayed-cells", but +possibly with different parameters as reported in the key=val part +of this line. Note that the overall reported value can be +negative. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "transport" transportname address:port [arglist] NL + [Any number.] +``` + +Signals that the router supports the 'transportname' pluggable +transport in IP address 'address' and TCP port 'port'. A single +descriptor MUST not have more than one transport line with the +same 'transportname'. + +Pluggable transports are only relevant to bridges, but these entries +can appear in non-bridge relays as well. + +```text + "padding-counts" YYYY-MM-DD HH:MM:SS (NSEC s) key=NUM key=NUM ... NL + [At most once.] +``` + +YYYY-MM-DD HH:MM:SS defines the end of the included measurement +interval of length NSEC seconds (86400 seconds by default). Counts +are reset to 0 at the end of this interval. + +The keyword list is currently as follows: + +```text + bin-size + - The current rounding value for cell count fields (10000 by + default) + write-drop + - The number of RELAY_DROP cells this relay sent + write-pad + - The number of CELL_PADDING cells this relay sent + write-total + - The total number of cells this relay cent + read-drop + - The number of RELAY_DROP cells this relay received + read-pad + - The number of CELL_PADDING cells this relay received + read-total + - The total number of cells this relay received + enabled-read-pad + - The number of CELL_PADDING cells this relay received on + connections that support padding + enabled-read-total + - The total number of cells this relay received on connections + that support padding + enabled-write-pad + - The total number of cells this relay received on connections + that support padding + enabled-write-total + - The total number of cells sent by this relay on connections + that support padding + max-chanpad-timers + - The maximum number of timers that this relay scheduled for + padding in the previous NSEC interval + + "overload-ratelimits" SP version SP YYYY-MM-DD SP HH:MM:SS + SP rate-limit SP burst-limit + SP read-overload-count SP write-overload-count NL + [At most once.] + + Indicates that a bandwidth limit was exhausted for this relay. +``` + +The "rate-limit" and "burst-limit" are the raw values from the +BandwidthRate and BandwidthBurst found in the torrc configuration file. + +The "{read|write}-overload-count" are the counts of how many times the +reported limits of burst/rate were exhausted and thus the maximum +between the read and write count occurrences. To make the counter more +meaningful and to avoid multiple connections saturating the counter +when a relay is overloaded, we only increment it once a minute. + +The 'version' field is set to '1' for now. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "overload-fd-exhausted" SP version YYYY-MM-DD HH:MM:SS NL + [At most once.] +``` + +Indicates that a file descriptor exhaustion was experienced by this +relay. + +The timestamp indicates that the maximum was reached between the +timestamp and the "published" timestamp of the document. + +This overload field should remain in place for 72 hours since last +triggered. If the limits are reached again in this period, the +timestamp is updated, and this 72 hour period restarts. + +The 'version' field is set to '1' for the initial implementation which +detects fd exhaustion only when a socket open fails. + +(Introduced in tor-0.4.6.1-alpha) + +```text + "router-sig-ed25519" + [As in router descriptors] + + "router-signature" NL Signature NL + [At end, exactly once.] + [No extra arguments] +``` + +A document signature as documented in section 1.3, using the +initial item "extra-info" and the final item "router-signature", +signed with the router's identity key. + diff --git a/spec/dir-spec/general-use-http-urls.md b/spec/dir-spec/general-use-http-urls.md new file mode 100644 index 0000000..d3c68ad --- /dev/null +++ b/spec/dir-spec/general-use-http-urls.md @@ -0,0 +1,132 @@ + +# General-use HTTP URLs + +"Fingerprints" in these URLs are base16-encoded SHA1 hashes. + +The most recent v3 consensus should be available at: + +http:///tor/status-vote/current/consensus.z + +Similarly, the v3 microdescriptor consensus should be available at: + +http:///tor/status-vote/current/consensus-microdesc.z + +Starting with Tor version 0.2.1.1-alpha is also available at: + +http:///tor/status-vote/current/consensus/++.z + +(NOTE: Due to squid proxy url limitations at most 96 fingerprints can be +retrieved in a single request.) + +Where F1, F2, etc. are authority identity fingerprints the client trusts. +Servers will only return a consensus if more than half of the requested +authorities have signed the document, otherwise a 404 error will be sent +back. The fingerprints can be shortened to a length of any multiple of +two, using only the leftmost part of the encoded fingerprint. Tor uses +3 bytes (6 hex characters) of the fingerprint. + +Clients SHOULD sort the fingerprints in ascending order. Server MUST +accept any order. + +Clients SHOULD use this format when requesting consensus documents from +directory authority servers and from caches running a version of Tor +that is known to support this URL format. + +A concatenated set of all the current key certificates should be available +at: + +http:///tor/keys/all.z + +The key certificate for this server should be available at: + +http:///tor/keys/authority.z + +The key certificate for an authority whose authority identity fingerprint +is should be available at: + +http:///tor/keys/fp/.z + +The key certificate whose signing key fingerprint is should be +available at: + +http:///tor/keys/sk/.z + +The key certificate whose identity key fingerprint is and whose signing +key fingerprint is should be available at: + +http:///tor/keys/fp-sk/-.z + +(As usual, clients may request multiple certificates using: + +http:///tor/keys/fp-sk/-+-.z ) + +[The above fp-sk format was not supported before Tor 0.2.1.9-alpha.] + +The most recent descriptor for a server whose identity key has a +fingerprint of should be available at: + +http:///tor/server/fp/.z + +The most recent descriptors for servers with identity fingerprints +,, should be available at: + +http:///tor/server/fp/++.z + +(NOTE: Due to squid proxy url limitations at most 96 fingerprints can be +retrieved in a single request. + +Implementations SHOULD NOT download descriptors by identity key +fingerprint. This allows a corrupted server (in collusion with a cache) to +provide a unique descriptor to a client, and thereby partition that client +from the rest of the network.) + +The server descriptor with (descriptor) digest (in hex) should be +available at: + +http:///tor/server/d/.z + +The most recent descriptors with digests ,, should be +available at: + +http:///tor/server/d/++.z + +The most recent descriptor for this server should be at: + +http:///tor/server/authority.z + +This is used for authorities, and also if a server is configured +as a bridge. The official Tor implementations (starting at +0.1.1.x) use this resource to test whether a server's own DirPort +is reachable. It is also useful for debugging purposes. + +A concatenated set of the most recent descriptors for all known servers +should be available at: + +http:///tor/server/all.z + +Extra-info documents are available at the URLS + +```text + http:///tor/extra/d/... + http:///tor/extra/fp/... + http:///tor/extra/all[.z] + http:///tor/extra/authority[.z] + (As for /tor/server/ URLs: supports fetching extra-info + documents by their digest, by the fingerprint of their servers, + or all at once. When serving by fingerprint, we serve the + extra-info that corresponds to the descriptor we would serve by + that fingerprint. Only directory authorities of version + 0.2.0.1-alpha or later are guaranteed to support the first + three classes of URLs. Caches may support them, and MUST + support them if they have advertised "caches-extra-info".) +``` + +For debugging, directories SHOULD expose non-compressed objects at +URLs like the above, but without the final ".z". If the client uses +Accept-Encodings header, it should override the presence or absence +of the ".z" (see section 6.1). + +Clients SHOULD use upper case letters (A-F) when base16-encoding +fingerprints. Servers MUST accept both upper and lower case fingerprints +in requests. + diff --git a/spec/dir-spec/inferring-missing-proto-lines.md b/spec/dir-spec/inferring-missing-proto-lines.md new file mode 100644 index 0000000..0c0d530 --- /dev/null +++ b/spec/dir-spec/inferring-missing-proto-lines.md @@ -0,0 +1,15 @@ + +# Inferring missing proto lines. + +The directory authorities no longer allow versions of Tor before +0.2.4.18-rc. But right now, there is no version of Tor in the consensus +before 0.2.4.19. Therefore, we should disallow versions of Tor earlier +than 0.2.4.19, so that we can have the protocol list for all current Tor +versions include: + +Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1-2 Link=1-4 +LinkAuth=1 Microdesc=1-2 Relay=1-2 + +For Desc, Microdesc and Cons, Tor versions before 0.2.7.stable should be +taken to only support version 1. + diff --git a/spec/dir-spec/limited-ed-diff-format.md b/spec/dir-spec/limited-ed-diff-format.md new file mode 100644 index 0000000..35b5200 --- /dev/null +++ b/spec/dir-spec/limited-ed-diff-format.md @@ -0,0 +1,39 @@ + +# Limited ed diff format + +We support the following format for consensus diffs. It's a +subset of the ed diff format, but clients MUST NOT accept other +ed commands. + +We support the following ed commands, each on a line by itself: + +```text + - "d" Delete line n1 + - ",d" Delete lines n1 through n2, inclusive + - ",$d" Delete line n1 through the end of the file, inclusive. + - "c" Replace line n1 with the following block + - ",c" Replace lines n1 through n2, inclusive, with the + following block. + - "a" Append the following block after line n1. +``` + +Note that line numbers always apply to the file after all previous +commands have already been applied. Note also that line numbers +are 1-indexed. + +The commands MUST apply to the file from back to front, such that +lines are only ever referred to by their position in the original +file. + +If there are any directory signatures on the original document, the +first command MUST be a ",$d" form to remove all of the directory +signatures. Using this format ensures that the client will +successfully apply the diff even if they have an unusual encoding for +the signatures. + +The replace and append command take blocks. These blocks are simply +appended to the diff after the line with the command. A line with +just a period (".") ends the block (and is not part of the lines +to add). Note that it is impossible to insert a line with just +a single dot. + diff --git a/spec/dir-spec/nonterminals-server-descriptors.md b/spec/dir-spec/nonterminals-server-descriptors.md new file mode 100644 index 0000000..256f759 --- /dev/null +++ b/spec/dir-spec/nonterminals-server-descriptors.md @@ -0,0 +1,32 @@ + +### Nonterminals in server descriptors + +```text + nickname ::= between 1 and 19 alphanumeric characters ([A-Za-z0-9]), + case-insensitive. + hexdigest ::= a '$', followed by 40 hexadecimal characters + ([A-Fa-f0-9]). [Represents a relay by the digest of its identity + key.] +``` + +exitpattern ::= addrspec ":" portspec +portspec ::= "*" | port | port "-" port +port ::= an integer between 1 and 65535, inclusive. + +```text + [Some implementations incorrectly generate ports with value 0. + Implementations SHOULD accept this, and SHOULD NOT generate it. + Connections to port 0 are never permitted.] +``` + +addrspec ::= "*" | ip4spec | ip6spec +ipv4spec ::= ip4 | ip4 "/" num_ip4_bits | ip4 "/" ip4mask +ip4 ::= an IPv4 address in dotted-quad format +ip4mask ::= an IPv4 mask in dotted-quad format +num_ip4_bits ::= an integer between 0 and 32 +ip6spec ::= ip6 | ip6 "/" num_ip6_bits +ip6 ::= an IPv6 address, surrounded by square brackets. +num_ip6_bits ::= an integer between 0 and 128 + +bool ::= "0" | "1" + diff --git a/spec/dir-spec/outline.md b/spec/dir-spec/outline.md new file mode 100644 index 0000000..0a5c81d --- /dev/null +++ b/spec/dir-spec/outline.md @@ -0,0 +1,249 @@ + +# Outline + +There is a small set (say, around 5-10) of semi-trusted directory +authorities. A default list of authorities is shipped with the Tor +software. Users can change this list, but are encouraged not to do so, +in order to avoid partitioning attacks. + +Every authority has a very-secret, long-term "Authority Identity Key". +This is stored encrypted and/or offline, and is used to sign "key +certificate" documents. Every key certificate contains a medium-term +(3-12 months) "authority signing key", that is used by the authority to +sign other directory information. (Note that the authority identity +key is distinct from the router identity key that the authority uses +in its role as an ordinary router.) + +Routers periodically upload signed "routers descriptors" to the +directory authorities describing their keys, capabilities, and other +information. Routers may also upload signed "extra-info documents" +containing information that is not required for the Tor protocol. +Directory authorities serve server descriptors indexed by router +identity, or by hash of the descriptor. + +Routers may act as directory caches to reduce load on the directory +authorities. They announce this in their descriptors. + +Periodically, each directory authority generates a view of +the current descriptors and status for known routers. They send a +signed summary of this view (a "status vote") to the other +authorities. The authorities compute the result of this vote, and sign +a "consensus status" document containing the result of the vote. + +Directory caches download, cache, and re-serve consensus documents. + +Clients, directory caches, and directory authorities all use consensus +documents to find out when their list of routers is out-of-date. +(Directory authorities also use vote statuses.) If it is, they download +any missing server descriptors. Clients download missing descriptors +from caches; caches and authorities download from authorities. +Descriptors are downloaded by the hash of the descriptor, not by the +relay's identity key: this prevents directory servers from attacking +clients by giving them descriptors nobody else uses. + +All directory information is uploaded and downloaded with HTTP. + + +## What's different from version 2? + +Clients used to download multiple network status documents, +corresponding roughly to "status votes" above. They would compute the +result of the vote on the client side. + +Authorities used to sign documents using the same private keys they used +for their roles as routers. This forced them to keep these extremely +sensitive keys in memory unencrypted. + +All of the information in extra-info documents used to be kept in the +main descriptors. + + +## Document meta-format + +Server descriptors, directories, and running-routers documents all obey the +following lightweight extensible information format. + +The highest level object is a Document, which consists of one or more +Items. Every Item begins with a KeywordLine, followed by zero or more +Objects. A KeywordLine begins with a Keyword, optionally followed by +whitespace and more non-newline characters, and ends with a newline. A +Keyword is a sequence of one or more characters in the set [A-Za-z0-9-], +but may not start with -. +An Object is a block of encoded data in pseudo-Privacy-Enhanced-Mail (PEM) +style format: that is, lines of encoded data MAY be wrapped by inserting +an ascii linefeed ("LF", also called newline, or "NL" here) character +(cf. RFC 4648 §3.1). When line wrapping, implementations MUST wrap lines +at 64 characters. Upon decoding, implementations MUST ignore and discard +all linefeed characters. + +More formally: + +NL = The ascii LF character (hex value 0x0a). +Document ::= (Item | NL)+ +Item ::= KeywordLine Object? +KeywordLine ::= Keyword (WS Argument)* NL +Keyword = KeywordStart KeywordChar* +KeywordStart ::= 'A' ... 'Z' | 'a' ... 'z' | '0' ... '9' +KeywordChar ::= KeywordStart | '-' +Argument := ArgumentChar+ +ArgumentChar ::= any graphical printing ASCII character. +WS = (SP | TAB)+ +Object ::= BeginLine Base64-encoded-data EndLine +BeginLine ::= "-----BEGIN " Keyword (" " Keyword)* "-----" NL +EndLine ::= "-----END " Keyword (" " Keyword)* "-----" NL + +A Keyword may not be "-----BEGIN". + +The BeginLine and EndLine of an Object must use the same keyword. + +When interpreting a Document, software MUST ignore any KeywordLine that +starts with a keyword it doesn't recognize; future implementations MUST NOT +require current clients to understand any KeywordLine not currently +described. + +Other implementations that want to extend Tor's directory format MAY +introduce their own items. The keywords for extension items SHOULD start +with the characters "x-" or "X-", to guarantee that they will not conflict +with keywords used by future versions of Tor. + +In our document descriptions below, we tag Items with a multiplicity in +brackets. Possible tags are: + +```text + "At start, exactly once": These items MUST occur in every instance of + the document type, and MUST appear exactly once, and MUST be the + first item in their documents. + + "Exactly once": These items MUST occur exactly one time in every + instance of the document type. + + "At end, exactly once": These items MUST occur in every instance of + the document type, and MUST appear exactly once, and MUST be the + last item in their documents. + + "At most once": These items MAY occur zero or one times in any + instance of the document type, but MUST NOT occur more than once. + + "Any number": These items MAY occur zero, one, or more times in any + instance of the document type. + + "Once or more": These items MUST occur at least once in any instance + of the document type, and MAY occur more. +``` + +For forward compatibility, each item MUST allow extra arguments at the +end of the line unless otherwise noted. So if an item's description below +is given as: + +"thing" int int int NL + +then implementations SHOULD accept this string as well: + +"thing 5 9 11 13 16 12" NL + +but not this string: + +"thing 5" NL + +and not this string: + +```text + "thing 5 10 thing" NL + . +``` + +Whenever an item DOES NOT allow extra arguments, we will tag it with +"no extra arguments". + + +## Signing documents + +Every signable document below is signed in a similar manner, using a +given "Initial Item", a final "Signature Item", a digest algorithm, and +a signing key. + +The Initial Item must be the first item in the document. + +The Signature Item has the following format: + + [arguments] NL SIGNATURE NL + +The "SIGNATURE" Object contains a signature (using the signing key) of +the PKCS#1 1.5 padded digest of the entire document, taken from the +beginning of the Initial item, through the newline after the Signature +Item's keyword and its arguments. + +The signature does not include the algorithmIdentifier specified in PKCS #1. + +Unless specified otherwise, the digest algorithm is SHA-1. + +All documents are invalid unless signed with the correct signing key. + +The "Digest" of a document, unless stated otherwise, is its digest *as +signed by this signature scheme*. + + +## Voting timeline + +Every consensus document has a "valid-after" (VA) time, a "fresh-until" +(FU) time and a "valid-until" (VU) time. VA MUST precede FU, which MUST +in turn precede VU. Times are chosen so that every consensus will be +"fresh" until the next consensus becomes valid, and "valid" for a while +after. At least 3 consensuses should be valid at any given time. + +The timeline for a given consensus is as follows: + +VA-DistSeconds-VoteSeconds: The authorities exchange votes. Each authority +uploads their vote to all other authorities. + +VA-DistSeconds-VoteSeconds/2: The authorities try to download any +votes they don't have. + +Authorities SHOULD also reject any votes that other authorities try to +upload after this time. (0.4.4.1-alpha was the first version to reject votes +in this way.) + +Note: Refusing late uploaded votes minimizes the chance of a consensus +split, particular when authorities are under bandwidth pressure. If an +authority is struggling to upload its vote, and finally uploads to a +fraction of authorities after this period, they will compute a consensus +different from the others. By refusing uploaded votes after this time, +we increase the likelihood that most authorities will use the same vote +set. + +Rejecting late uploaded votes does not fix the problem entirely. If +some authorities are able to download a specific vote, but others fail +to do so, then there may still be a consensus split. However, this +change does remove one common cause of consensus splits. + +VA-DistSeconds: The authorities calculate the consensus and exchange +signatures. (This is the earliest point at which anybody can +possibly get a given consensus if they ask for it.) + +VA-DistSeconds/2: The authorities try to download any signatures +they don't have. + +VA: All authorities have a multiply signed consensus. + +```text + VA ... FU: Caches download the consensus. (Note that since caches have + no way of telling what VA and FU are until they have downloaded + the consensus, they assume that the present consensus's VA is + equal to the previous one's FU, and that its FU is one interval after + that.) + + FU: The consensus is no longer the freshest consensus. + + FU ... (the current consensus's VU): Clients download the consensus. + (See note above: clients guess that the next consensus's FU will be + two intervals after the current VA.) +``` + +VU: The consensus is no longer valid; clients should continue to try to +download a new consensus if they have not done so already. + +VU + 24 hours: Clients will no longer use the consensus at all. + +VoteSeconds and DistSeconds MUST each be at least 20 seconds; FU-VA and +VU-FU MUST each be at least 5 minutes. + diff --git a/spec/dir-spec/router-operation-formats.md b/spec/dir-spec/router-operation-formats.md new file mode 100644 index 0000000..7757a32 --- /dev/null +++ b/spec/dir-spec/router-operation-formats.md @@ -0,0 +1,3 @@ + +# Router operation and formats + diff --git a/spec/dir-spec/scope-preliminaries.md b/spec/dir-spec/scope-preliminaries.md new file mode 100644 index 0000000..b513c6d --- /dev/null +++ b/spec/dir-spec/scope-preliminaries.md @@ -0,0 +1,139 @@ + +# Scope and preliminaries + +This directory protocol is used by Tor version 0.2.0.x-alpha and later. +See dir-spec-v1.txt for information on the protocol used up to the +0.1.0.x series, and dir-spec-v2.txt for information on the protocol +used by the 0.1.1.x and 0.1.2.x series. + +This document merges and supersedes the following proposals: + +101 Voting on the Tor Directory System +103 Splitting identity key from regularly used signing key +104 Long and Short Router Descriptors + +XXX timeline +XXX fill in XXXXs + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL +NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in +RFC 2119. + + +## History + +The earliest versions of Onion Routing shipped with a list of known +routers and their keys. When the set of routers changed, users needed to +fetch a new list. + +The Version 1 Directory protocol +-------------------------------- + +Early versions of Tor (0.0.2) introduced "Directory authorities": servers +that served signed "directory" documents containing a list of signed +"server descriptors", along with short summary of the status of each +router. Thus, clients could get up-to-date information on the state of +the network automatically, and be certain that the list they were getting +was attested by a trusted directory authority. + +Later versions (0.0.8) added directory caches, which download +directories from the authorities and serve them to clients. Non-caches +fetch from the caches in preference to fetching from the authorities, thus +distributing bandwidth requirements. + +Also added during the version 1 directory protocol were "router status" +documents: short documents that listed only the up/down status of the +routers on the network, rather than a complete list of all the +descriptors. Clients and caches would fetch these documents far more +frequently than they would fetch full directories. + +The Version 2 Directory Protocol +-------------------------------- + +During the Tor 0.1.1.x series, Tor revised its handling of directory +documents in order to address two major problems: + +```text + * Directories had grown quite large (over 1MB), and most directory + downloads consisted mainly of server descriptors that clients + already had. + + * Every directory authority was a trust bottleneck: if a single + directory authority lied, it could make clients believe for a time + an arbitrarily distorted view of the Tor network. (Clients + trusted the most recent signed document they downloaded.) Thus, + adding more authorities would make the system less secure, not + more. +``` + +To address these, we extended the directory protocol so that +authorities now published signed "network status" documents. Each +network status listed, for every router in the network: a hash of its +identity key, a hash of its most recent descriptor, and a summary of +what the authority believed about its status. Clients would download +the authorities' network status documents in turn, and believe +statements about routers iff they were attested to by more than half of +the authorities. + +Instead of downloading all server descriptors at once, clients +downloaded only the descriptors that they did not have. Descriptors +were indexed by their digests, in order to prevent malicious caches +from giving different versions of a server descriptor to different +clients. + +Routers began working harder to upload new descriptors only when their +contents were substantially changed. + + +## Goals of the version 3 protocol + +Version 3 of the Tor directory protocol tries to solve the following +issues: + +```text + * A great deal of bandwidth used to transmit server descriptors was + used by two fields that are not actually used by Tor routers + (namely read-history and write-history). We save about 60% by + moving them into a separate document that most clients do not + fetch or use. + + * It was possible under certain perverse circumstances for clients + to download an unusual set of network status documents, thus + partitioning themselves from clients who have a more recent and/or + typical set of documents. Even under the best of circumstances, + clients were sensitive to the ages of the network status documents + they downloaded. Therefore, instead of having the clients + correlate multiple network status documents, we have the + authorities collectively vote on a single consensus network status + document. + + * The most sensitive data in the entire network (the identity keys + of the directory authorities) needed to be stored unencrypted so + that the authorities can sign network-status documents on the fly. + Now, the authorities' identity keys are stored offline, and used + to certify medium-term signing keys that can be rotated. +``` + + +## Some Remaining questions + +Things we could solve on a v3 timeframe: + +The SHA-1 hash is showing its age. We should do something about our +dependency on it. We could probably future-proof ourselves here in +this revision, at least so far as documents from the authorities are +concerned. + +Too many things about the authorities are hardcoded by IP. + +Perhaps we should start accepting longer identity keys for routers +too. + +Things to solve eventually: + +Requiring every client to know about every router won't scale forever. + +Requiring every directory cache to know every router won't scale +forever. + diff --git a/spec/dir-spec/server-descriptor-format.md b/spec/dir-spec/server-descriptor-format.md new file mode 100644 index 0000000..9866076 --- /dev/null +++ b/spec/dir-spec/server-descriptor-format.md @@ -0,0 +1,508 @@ + +### Server descriptor format + +Server descriptors consist of the following items. + +In lines that take multiple arguments, extra arguments SHOULD be +accepted and ignored. Many of the nonterminals below are defined in +section 2.1.3. + +Note that many versions of Tor will generate an extra newline at the +end of their descriptors. Implementations MUST tolerate one or +more blank lines at the end of a single descriptor or a list of +concatenated descriptors. New implementations SHOULD NOT generate +such blank lines. + +"router" nickname address ORPort SOCKSPort DirPort NL + +[At start, exactly once.] + +Indicates the beginning of a server descriptor. "nickname" must be a +valid router nickname as specified in section 2.1.3. "address" must +be an IPv4 +address in dotted-quad format. The last three numbers indicate the +TCP ports at which this OR exposes functionality. ORPort is a port at +which this OR accepts TLS connections for the main OR protocol; +SOCKSPort is deprecated and should always be 0; and DirPort is the +port at which this OR accepts directory-related HTTP connections. If +any port is not supported, the value 0 is given instead of a port +number. (At least one of DirPort and ORPort SHOULD be set; +authorities MAY reject any descriptor with both DirPort and ORPort of +0.) + +```text + "identity-ed25519" NL "-----BEGIN ED25519 CERT-----" NL certificate + "-----END ED25519 CERT-----" NL +``` + +[Exactly once, in second position in document.] +[No extra arguments] + +The certificate is a base64-encoded Ed25519 certificate (see +cert-spec.txt) with terminating =s removed. When this element +is present, it MUST appear as the first or second element in +the router descriptor. + +The certificate has CERT_TYPE of [04]. It must include a +signed-with-ed25519-key extension (see cert-spec.txt, +section 2.2.1), so that we can extract the master identity key. + +[Before Tor 0.4.5.1-alpha, this field was optional.] + +"master-key-ed25519" SP MasterKey NL + +[Exactly once] + +Contains the base-64 encoded ed25519 master key as a single +argument. If it is present, it MUST match the identity key +in the identity-ed25519 entry. + +[Before Tor 0.4.5.1-alpha, this field was optional.] + +"bandwidth" bandwidth-avg bandwidth-burst bandwidth-observed NL + +[Exactly once] + +Estimated bandwidth for this router, in bytes per second. The +"average" bandwidth is the volume per second that the OR is willing to +sustain over long periods; the "burst" bandwidth is the volume that +the OR is willing to sustain in very short intervals. The "observed" +value is an estimate of the capacity this relay can handle. The +relay remembers the max bandwidth sustained output over any ten +second period in the past 5 days, and another sustained input. The +"observed" value is the lesser of these two numbers. + +Tor versions released before 2018 only kept bandwidth-observed for one +day. These versions are no longer supported or recommended. + +"platform" string NL + +[At most once] + +A human-readable string describing the system on which this OR is +running. This MAY include the operating system, and SHOULD include +the name and version of the software implementing the Tor protocol. + +"published" YYYY-MM-DD HH:MM:SS NL + +[Exactly once] + +The time, in UTC, when this descriptor (and its corresponding +extra-info document if any) was generated. + +"fingerprint" fingerprint NL + +[At most once] + +A fingerprint (a HASH_LEN-byte of asn1 encoded public key, encoded in +hex, with a single space after every 4 characters) for this router's +identity key. A descriptor is considered invalid (and MUST be +rejected) if the fingerprint line does not match the public key. + +```text + [We didn't start parsing this line until Tor 0.1.0.6-rc; it should + be marked with "opt" until earlier versions of Tor are obsolete.] + + "hibernating" bool NL + + [At most once] +``` + +If the value is 1, then the Tor relay was hibernating when the +descriptor was published, and shouldn't be used to build circuits. + +```text + [We didn't start parsing this line until Tor 0.1.0.6-rc; it should be + marked with "opt" until earlier versions of Tor are obsolete.] + + "uptime" number NL + + [At most once] + + The number of seconds that this OR process has been running. + + "onion-key" NL a public key in PEM format +``` + +[Exactly once] +[No extra arguments] + +This key is used to encrypt CREATE cells for this OR. The key MUST be +accepted for at least 1 week after any new key is published in a +subsequent descriptor. It MUST be 1024 bits. + +The key encoding is the encoding of the key as a PKCS#1 RSAPublicKey +structure, encoded in base64, and wrapped in "-----BEGIN RSA PUBLIC +KEY-----" and "-----END RSA PUBLIC KEY-----". + +"onion-key-crosscert" NL a RSA signature in PEM format. + +[Exactly once] +[No extra arguments] + +This element contains an RSA signature, generated using the +onion-key, of the following: + +```text + A SHA1 hash of the RSA identity key, + i.e. RSA key from "signing-key" (see below) [20 bytes] + The Ed25519 identity key, + i.e. Ed25519 key from "master-key-ed25519" [32 bytes] +``` + +If there is no Ed25519 identity key, or if in some future version +there is no RSA identity key, the corresponding field must be +zero-filled. + +Parties verifying this signature MUST allow additional data +beyond the 52 bytes listed above. + +This signature proves that the party creating the descriptor +had control over the secret key corresponding to the +onion-key. + +[Before Tor 0.4.5.1-alpha, this field was optional whenever +identity-ed25519 was absent.] + +"ntor-onion-key" base-64-encoded-key + +[Exactly once] + +A curve25519 public key used for the ntor circuit extended +handshake. It's the standard encoding of the OR's curve25519 +public key, encoded in base 64. The trailing '=' sign MAY be +omitted from the base64 encoding. The key MUST be accepted +for at least 1 week after any new key is published in a +subsequent descriptor. + +[Before Tor 0.4.5.1-alpha, this field was optional.] + +```text + "ntor-onion-key-crosscert" SP Bit NL + "-----BEGIN ED25519 CERT-----" NL certificate + "-----END ED25519 CERT-----" NL +``` + +[Exactly once] +[No extra arguments] + +A signature created with the ntor-onion-key, using the +certificate format documented in cert-spec.txt, with type +[0a]. The signed key here is the master identity key. + +Bit must be "0" or "1". It indicates the sign of the ed25519 +public key corresponding to the ntor onion key. If Bit is "0", +then implementations MUST guarantee that the x-coordinate of +the resulting ed25519 public key is positive. Otherwise, if +Bit is "1", then the sign of the x-coordinate MUST be negative. + +To compute the ed25519 public key corresponding to a curve25519 +key, and for further explanation on key formats, see appendix C. + +This signature proves that the party creating the descriptor +had control over the secret key corresponding to the +ntor-onion-key. + +[Before Tor 0.4.5.1-alpha, this field was optional whenever +identity-ed25519 was absent.] + +"signing-key" NL a public key in PEM format + +[Exactly once] +[No extra arguments] + +The OR's long-term RSA identity key. It MUST be 1024 bits. + +The encoding is as for "onion-key" above. + +"accept" exitpattern NL +"reject" exitpattern NL + +[Any number] + +These lines describe an "exit policy": the rules that an OR follows +when deciding whether to allow a new stream to a given address. The +'exitpattern' syntax is described below. There MUST be at least one +such entry. The rules are considered in order; if no rule matches, +the address will be accepted. For clarity, the last such entry SHOULD +be accept *:* or reject *:*. + +"ipv6-policy" SP ("accept" / "reject") SP PortList NL + +[At most once.] + +An exit-policy summary as specified in sections 3.4.1 and 3.8.2, +summarizing +the router's rules for connecting to IPv6 addresses. A missing +"ipv6-policy" line is equivalent to "ipv6-policy reject +1-65535". + +"overload-general" SP version SP YYYY-MM-DD HH:MM:SS NL + +[At most once.] + +Indicates that a relay has reached an "overloaded state" which can be +one or many of the following load metrics: + +```text + - Any OOM invocation due to memory pressure + - Any ntor onionskins are dropped + - TCP port exhaustion +``` + +The timestamp is when at least one metrics was detected. It should always +be at the hour and thus, as an example, "2020-01-10 13:00:00" is an +expected timestamp. Because this is a binary state, if the line is +present, we consider that it was hit at the very least once somewhere +between the provided timestamp and the "published" timestamp of the +document which is when the document was generated. + +The overload-general line should remain in place for 72 hours since last +triggered. If the limits are reached again in this period, the timestamp +is updated, and this 72 hour period restarts. + +The 'version' field is set to '1' for now. + +```text + (Introduced in tor-0.4.6.1-alpha, but moved from extra-info to general + descriptor in tor-0.4.6.2-alpha) + + "router-sig-ed25519" SP Signature NL + + [Exactly once.] +``` + +It MUST be the next-to-last element in the descriptor, appearing +immediately before the RSA signature. It MUST contain an Ed25519 +signature of a SHA256 digest of the entire document. This digest is +taken from the first character up to and including the first space +after the "router-sig-ed25519" string. Before computing the digest, +the string "Tor router descriptor signature v1" is prefixed to the +document. + +The signature is encoded in Base64, with terminating =s removed. + +The signing key in the identity-ed25519 certificate MUST +be the one used to sign the document. + +[Before Tor 0.4.5.1-alpha, this field was optional whenever +identity-ed25519 was absent.] + +"router-signature" NL Signature NL + +[At end, exactly once] +[No extra arguments] + +The "SIGNATURE" object contains a signature of the PKCS1-padded +hash of the entire server descriptor, taken from the beginning of the +"router" line, through the newline after the "router-signature" line. +The server descriptor is invalid unless the signature is performed +with the router's identity key. + +"contact" info NL + +[At most once] + +Describes a way to contact the relay's administrator, preferably +including an email address and a PGP key fingerprint. + +"bridge-distribution-request" SP Method NL + +[At most once, bridges only.] + +The "Method" describes how a Bridge address is distributed by +BridgeDB. Recognized methods are: "none", "any", "https", "email", +"moat". If set to "none", BridgeDB will avoid distributing your bridge +address. If set to "any", BridgeDB will choose how to distribute your +bridge address. Choosing any of the other methods will tell BridgeDB to +distribute your bridge via a specific method: + +```text + - "https" specifies distribution via the web interface at + https://bridges.torproject.org; + - "email" specifies distribution via the email autoresponder at + bridges@torproject.org; + - "moat" specifies distribution via an interactive menu inside Tor + Browser; and + + Potential future "Method" specifiers must be as follows: + Method = (KeywordChar | "_") + +``` + +All bridges SHOULD include this line. Non-bridges MUST NOT include +it. + +BridgeDB SHOULD treat unrecognized Method values as if they were +"none". + +(Default: "any") + +[This line was introduced in 0.3.2.3-alpha, with a minimal backport +to 0.2.5.16, 0.2.8.17, 0.2.9.14, 0.3.0.13, 0.3.1.9, and later.] + +"family" names NL + +[At most once] + +'Names' is a space-separated list of relay nicknames or +hexdigests. If two ORs list one another in their "family" entries, +then OPs should treat them as a single OR for the purpose of path +selection. + +For example, if node A's descriptor contains "family B", and node B's +descriptor contains "family A", then node A and node B should never +be used on the same circuit. + +```text + "read-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once] + "write-history" YYYY-MM-DD HH:MM:SS (NSEC s) NUM,NUM,NUM,NUM,NUM... NL + [At most once] +``` + +(These fields once appeared in router descriptors, but have +appeared in extra-info descriptors since 0.2.0.x.) + +"eventdns" bool NL + +[At most once] + +Declare whether this version of Tor is using the newer enhanced +dns logic. Versions of Tor with this field set to false SHOULD NOT +be used for reverse hostname lookups. + +```text + [This option is obsolete. All Tor current relays should be presumed + to have the evdns backend.] + + "caches-extra-info" NL +``` + +[At most once.] +[No extra arguments] + +Present only if this router is a directory cache that provides +extra-info documents. + +[Versions before 0.2.0.1-alpha don't recognize this] + +"extra-info-digest" SP sha1-digest [SP sha256-digest] NL + +[At most once] + +"sha1-digest" is a hex-encoded SHA1 digest (using upper-case characters) +of the router's extra-info document, as signed in the router's +extra-info (that is, not including the signature). (If this field is +absent, the router is not uploading a corresponding extra-info +document.) + +"sha256-digest" is a base64-encoded SHA256 digest of the extra-info +document. Unlike the "sha1-digest", this digest is calculated over the +entire document, including the signature. This difference is due to +a long-lived bug in the tor implementation that it would be difficult +to roll out an incremental fix for, not a design choice. Future digest +algorithms specified should not include the signature in the data used +to compute the digest. + +[Versions before 0.2.7.2-alpha did not include a SHA256 digest.] +[Versions before 0.2.0.1-alpha don't recognize this field at all.] + +"hidden-service-dir" NL + +[At most once.] + +Present only if this router stores and serves hidden service +descriptors. This router supports the descriptor versions declared +in the HSDir "proto" entry. If there is no "proto" entry, this +router supports version 2 descriptors. + +```text + "protocols" SP "Link" SP LINK-VERSION-LIST SP "Circuit" SP + CIRCUIT-VERSION-LIST NL + + [At most once.] +``` + +An obsolete list of protocol versions, superseded by the "proto" +entry. This list was never parsed, and has not been emitted +since Tor 0.2.9.4-alpha. New code should neither generate nor +parse this line. + +"allow-single-hop-exits" NL + +[At most once.] +[No extra arguments] + +Present only if the router allows single-hop circuits to make exit +connections. Most Tor relays do not support this: this is +included for specialized controllers designed to support perspective +access and such. This is obsolete in tor version >= 0.3.1.0-alpha. + +"or-address" SP ADDRESS ":" PORT NL + +[Any number] + +ADDRESS = IP6ADDR | IP4ADDR +IPV6ADDR = an ipv6 address, surrounded by square brackets. +IPV4ADDR = an ipv4 address, represented as a dotted quad. +PORT = a number between 1 and 65535 inclusive. + +An alternative for the address and ORPort of the "router" line, but with +two added capabilities: + +```text + * or-address can be either an IPv4 or IPv6 address + * or-address allows for multiple ORPorts and addresses +``` + +A descriptor SHOULD NOT include an or-address line that does nothing but +duplicate the address:port pair from its "router" line. + +The ordering of or-address lines and their PORT entries matter because +Tor MAY accept a limited number of address/port pairs. As of +Tor 0.2.3.x only the first address/port pair is advertised and used. + +"tunnelled-dir-server" NL + +[At most once.] +[No extra arguments] + +```text + Present if the router accepts "tunneled" directory requests using a + BEGIN_DIR cell over the router's OR port. + (Added in 0.2.8.1-alpha. Before this, Tor relays accepted + tunneled directory requests only if they had a DirPort open, + or if they were bridges.) + + "proto" SP Entries NL + + [Exactly once.] +``` + +Entries = +Entries = Entry +Entries = Entry SP Entries + +Entry = Keyword "=" Values + +Values = +Values = Value +Values = Value "," Values + +Value = Int +Value = Int "-" Int + +Int = NON_ZERO_DIGIT +Int = Int DIGIT + +Each 'Entry' in the "proto" line indicates that the Tor relay supports +one or more versions of the protocol in question. Entries should be +sorted by keyword. Values should be numerically ascending within each +entry. (This implies that there should be no overlapping ranges.) +Ranges should be represented as compactly as possible. Ints must be no +larger than 63. + +This field was first added in Tor 0.2.9.x. + +[Before Tor 0.4.5.1-alpha, this field was optional.] + diff --git a/spec/dir-spec/serving-bandwidth-list-files.md b/spec/dir-spec/serving-bandwidth-list-files.md new file mode 100644 index 0000000..8e7f970 --- /dev/null +++ b/spec/dir-spec/serving-bandwidth-list-files.md @@ -0,0 +1,868 @@ + +### Serving bandwidth list files + +If an authority has used a bandwidth list file to generate a vote +document it SHOULD make it available at + +http:///tor/status-vote/next/bandwidth.z + +at the start of each voting period. + +It MUST NOT attempt to send its bandwidth list file in a HTTP POST to +other authorities and it SHOULD NOT make bandwidth list files from other +authorities available. + +If an authority makes this file available, it MUST be the bandwidth file +used to create the vote document available at + +http:///tor/status-vote/next/authority.z + +To avoid inconsistent reads, authorities SHOULD only read the bandwidth +file once per voting period. Further processing and serving SHOULD use a +cached copy. + +The bandwidth list format is described in bandwidth-file-spec.txt. + +The standard URLs for bandwidth list files first-appeared in +Tor 0.4.0.4-alpha. + + +## Downloading missing certificates from other directory authorities + +XXX when to download certificates. + + +## Downloading server descriptors from other directory authorities + +Periodically (currently, every 10 seconds), directory authorities check +whether there are any specific descriptors that they do not have and that +they are not currently trying to download. +Authorities identify them by hash in vote (if publication date is more +recent than the descriptor we currently have). + +[XXXX need a way to fetch descriptors ahead of the vote? v2 status docs can +do that for now.] + +If so, the directory authority launches requests to the authorities for these +descriptors, such that each authority is only asked for descriptors listed +in its most recent vote. If more +than one authority lists the descriptor, we choose which to ask at random. + +If one of these downloads fails, we do not try to download that descriptor +from the authority that failed to serve it again unless we receive a newer +network-status (consensus or vote) from that authority that lists the same +descriptor. + +```text + Directory authorities must potentially cache multiple descriptors for each + router. Authorities must not discard any descriptor listed by any recent + consensus. If there is enough space to store additional descriptors, + authorities SHOULD try to hold those which clients are likely to download the + most. (Currently, this is judged based on the interval for which each + descriptor seemed newest.) +[XXXX define recent] +``` + +Authorities SHOULD NOT download descriptors for routers that they would +immediately reject for reasons listed in section 3.2. + + +## Downloading extra-info documents from other directory authorities + +Periodically, an authority checks whether it is missing any extra-info +documents: in other words, if it has any server descriptors with an +extra-info-digest field that does not match any of the extra-info +documents currently held. If so, it downloads whatever extra-info +documents are missing. We follow the same splitting and back-off rules +as in section 3.6. + + +## Computing a consensus from a set of votes + +Given a set of votes, authorities compute the contents of the consensus. + +The consensus status, along with as many signatures as the server +currently knows (see section 3.10 below), should be available at + +http:///tor/status-vote/next/consensus.z + +The contents of the consensus document are as follows: + +The "valid-after", "valid-until", and "fresh-until" times are taken as +the median of the respective values from all the votes. + +The times in the "voting-delay" line are taken as the median of the +VoteSeconds and DistSeconds times in the votes. + +Known-flags is the union of all flags known by any voter. + +Entries are given on the "params" line for every keyword on which a +majority of authorities (total authorities, not just those +participating in this vote) voted on, or if at least three +authorities voted for that parameter. The values given are the +low-median of all votes on that keyword. + +(In consensus methods 7 to 11 inclusive, entries were given on +the "params" line for every keyword on which *any* authority voted, +the value given being the low-median of all votes on that keyword.) + +```text + "client-versions" and "server-versions" are sorted in ascending + order; A version is recommended in the consensus if it is recommended + by more than half of the voting authorities that included a + client-versions or server-versions lines in their votes. +``` + +With consensus methods 19 through 33, a package line is generated for a +given PACKAGENAME/VERSION pair if at least three authorities list such a +package in their votes. (Call these lines the "input" lines for +PACKAGENAME.) The consensus will contain every "package" line that is +listed verbatim by more than half of the authorities listing a line for +the PACKAGENAME/VERSION pair, and no others. + +The authority item groups (dir-source, contact, fingerprint, +vote-digest) are taken from the votes of the voting +authorities. These groups are sorted by the digests of the +authorities identity keys, in ascending order. If the consensus +method is 3 or later, a dir-source line must be included for +every vote with legacy-key entry, using the legacy-key's +fingerprint, the voter's ordinary nickname with the string +"-legacy" appended, and all other fields as from the original +vote's dir-source line. + +```text + A router status entry: + * is included in the result if some router status entry with the same + identity is included by more than half of the authorities (total + authorities, not just those whose votes we have). + (Consensus method earlier than 21) + + * is included according to the rules in section 3.8.0.1 and + 3.8.0.2 below. (Consensus method 22 or later) + + * For any given RSA identity digest, we include at most + one router status entry. + + * For any given Ed25519 identity, we include at most one router + status entry. + + * A router entry has a flag set if that is included by more than half + of the authorities who care about that flag. + + * Two router entries are "the same" if they have the same + tuple. + We choose the tuple for a given router as whichever tuple appears + for that router in the most votes. We break ties first in favor of + the more recently published, then in favor of smaller server + descriptor digest. + + [ + * The Named flag appears if it is included for this routerstatus by + _any_ authority, and if all authorities that list it list the same + nickname. However, if consensus-method 2 or later is in use, and + any authority calls this identity/nickname pair Unnamed, then + this routerstatus does not get the Named flag. + + * If consensus-method 2 or later is in use, the Unnamed flag is + set for a routerstatus if any authorities have voted for a different + identities to be Named with that nickname, or if any authority + lists that nickname/ID pair as Unnamed. + + (With consensus-method 1, Unnamed is set like any other flag.) + + [But note that authorities no longer vote for the Named flag, + and the above two bulletpoints are now irrelevant.] + ] + + * The version is given as whichever version is listed by the most + voters, with ties decided in favor of more recent versions. + + * If consensus-method 4 or later is in use, then routers that + do not have the Running flag are not listed at all. + + * If consensus-method 5 or later is in use, then the "w" line + is generated using a low-median of the bandwidth values from + the votes that included "w" lines for this router. + + * If consensus-method 5 or later is in use, then the "p" line + is taken from the votes that have the same policy summary + for the descriptor we are listing. (They should all be the + same. If they are not, we pick the most commonly listed + one, breaking ties in favor of the lexicographically larger + vote.) The port list is encoded as specified in section 3.8.2. + + * If consensus-method 6 or later is in use and if 3 or more + authorities provide a Measured= keyword in their votes for + a router, the authorities produce a consensus containing a + Bandwidth= keyword equal to the median of the Measured= votes. + + * If consensus-method 7 or later is in use, the params line is + included in the output. + + * If the consensus method is under 11, bad exits are considered as + possible exits when computing bandwidth weights. Otherwise, if + method 11 or later is in use, any router that is determined to get + the BadExit flag doesn't count when we're calculating weights. + + * If consensus method 12 or later is used, only consensus + parameters that more than half of the total number of + authorities voted for are included in the consensus. + + [ As of 0.2.6.1-alpha, authorities no longer advertise or negotiate + any consensus methods lower than 13. ] + + * If consensus method 13 or later is used, microdesc consensuses + omit any router for which no microdesc was agreed upon. + + * If consensus method 14 or later is used, the ns consensus and + microdescriptors may include an "a" line for each router, listing + an IPv6 OR port. + + * If consensus method 15 or later is used, microdescriptors + include "p6" lines including IPv6 exit policies. + + * If consensus method 16 or later is used, ntor-onion-key + are included in microdescriptors + + * If consensus method 17 or later is used, authorities impose a + maximum on the Bandwidth= values that they'll put on a 'w' + line for any router that doesn't have at least 3 measured + bandwidth values in votes. They also add an "Unmeasured=1" + flag to such 'w' lines. + + * If consensus method 18 or later is used, authorities include + "id" lines in microdescriptors. This method adds RSA ids. + + * If consensus method 19 or later is used, authorities may include + "package" lines in consensuses. + + * If consensus method 20 or later is used, authorities may include + GuardFraction information in microdescriptors. + + * If consensus method 21 or later is used, authorities may include + an "id" line for ed25519 identities in microdescriptors. + + [ As of 0.2.8.2-alpha, authorities no longer advertise or negotiate + consensus method 21, because it contains bugs. ] + + * If consensus method 22 or later is used, and the votes do not + produce a majority consensus about a relay's Ed25519 key (see + 3.8.0.1 below), the consensus must include a NoEdConsensus flag on + the "s" line for every relay whose listed Ed key does not reflect + consensus. + + * If consensus method 23 or later is used, authorities include + shared randomness protocol data on their votes and consensus. + + * If consensus-method 24 or later is in use, then routers that + do not have the Valid flag are not listed at all. + + [ As of 0.3.4.1-alpha, authorities no longer advertise or negotiate + any consensus methods lower than 25. ] + + * If consensus-method 25 or later is in use, then we vote + on recommended-protocols and required-protocols lines in the + consensus. We also include protocols lines in routerstatus + entries. + + * If consensus-method 26 or later is in use, then we initialize + bandwidth weights to 1 in our calculations, to avoid + division-by-zero errors on unusual networks. + + * If consensus method 27 or later is used, the microdesc consensus + may include an "a" line for each router, listing an IPv6 OR port. + + [ As of 0.4.3.1-alpha, authorities no longer advertise or negotiate + any consensus methods lower than 28. ] + + * If consensus method 28 or later is used, microdescriptors no longer + include "a" lines. + + * If consensus method 29 or later is used, microdescriptor "family" + lines are canonicalized to improve compression. + + * If consensus method 30 or later is used, the base64 encoded + ntor-onion-key does not include the trailing = sign. + + * If consensus method 31 or later is used, authorities parse the + "bwweightscale" and "maxunmeasuredbw" parameters correctly when + computing votes. + + * If consensus method 32 or later is used, authorities handle the + "MiddleOnly" flag specially when computing a consensus. When the + voters agree to include "MiddleOnly" in a routerstatus, they + automatically remove "Exit", "Guard", "V2Dir", and "HSDir". If + the BadExit flag is included in the consensus, they automatically + add it to the routerstatus. + + * If consensus method 33 or later is used, and the consensus + flavor is "microdesc", then the "Publication" field in the "r" + line is set to "2038-01-01 00:00:00". + + * If consensus method 34 or later is used, the consensus + does not include any "package" lines. +``` + +The signatures at the end of a consensus document are sorted in +ascending order by identity digest. + +All ties in computing medians are broken in favor of the smaller or +earlier item. + + +#### Deciding which Ids to include. + +This sorting algorithm is used for consensus-method 22 and later. + +```text + First, consider each listing by tuple of identities, where 'Ed' + may be "None" if the voter included "id ed25519 none" to indicate that + the authority knows what ed25519 identities are, and thinks that the RSA + key doesn't have one. + + For each such tuple that is listed by more than half of the + total authorities (not just total votes), include it. (It is not + possible for any other to have as many votes.) If more + than half of the authorities list a single pair of this type, we + consider that Ed key to be "consensus"; see description of the + NoEdConsensus flag. + + Log any other id-RSA values corresponding to an id-Ed we included, and any + other id-Ed values corresponding to an id-RSA we included. + + For each that is not yet included, if it is listed by more than + half of the total authorities, and we do not already have it listed with + some , include it, but do not consider its Ed identity canonical. +``` + + +#### Deciding which descriptors to include + +Deciding which descriptors to include. + +A tuple belongs to an identity if it is a new tuple that +matches both ID parts, or if it is an old tuple (one with no Ed opinion) +that matches the RSA part. A tuple belongs to an identity if its +RSA identity matches. + +A tuple matches another tuple if all the fields that are present in both +tuples are the same. + +For every included identity, consider the tuples belonging to that +identity. Group them into sets of matching tuples. Include the tuple +that matches the largest set, breaking ties in favor of the most recently +published, and then in favor of the smaller server descriptor digest. + + +### Forward compatibility + +Future versions of Tor will need to include new information in the +consensus documents, but it is important that all authorities (or at least +half) generate and sign the same signed consensus. + +To achieve this, authorities list in their votes their supported methods +for generating consensuses from votes. Later methods will be assigned +higher numbers. Currently specified methods: + +```text + "1" -- The first implemented version. + "2" -- Added support for the Unnamed flag. + "3" -- Added legacy ID key support to aid in authority ID key rollovers + "4" -- No longer list routers that are not running in the consensus + "5" -- adds support for "w" and "p" lines. + "6" -- Prefers measured bandwidth values rather than advertised + "7" -- Provides keyword=integer pairs of consensus parameters + "8" -- Provides microdescriptor summaries + "9" -- Provides weights for selecting flagged routers in paths + "10" -- Fixes edge case bugs in router flag selection weights + "11" -- Don't consider BadExits when calculating bandwidth weights + "12" -- Params are only included if enough auths voted for them + "13" -- Omit router entries with missing microdescriptors. + "14" -- Adds support for "a" lines in ns consensuses and microdescriptors. + "15" -- Adds support for "p6" lines. + "16" -- Adds ntor keys to microdescriptors + "17" -- Adds "Unmeasured=1" flags to "w" lines + "18" -- Adds 'id' to microdescriptors. + "19" -- Adds "package" lines to consensuses + "20" -- Adds GuardFraction information to microdescriptors. + "21" -- Adds Ed25519 keys to microdescriptors. + "22" -- Instantiates Ed25519 voting algorithm correctly. + "23" -- Adds shared randomness protocol data. + "24" -- No longer lists routers that are not Valid in the consensus. + "25" -- Vote on recommended-protocols and required-protocols. + "26" -- Initialize bandwidth weights to 1 to avoid division-by-zero. + "27" -- Adds support for "a" lines in microdescriptor consensuses. + "28" -- Removes "a" lines from microdescriptors. + "29" -- Canonicalizes families in microdescriptors. + "30" -- Removes padding from ntor-onion-key. + "31" -- Uses correct parsing for bwweightscale and maxunmeasuredbw + when computing weights + "32" -- Adds special handling for MiddleOnly flag. + "33" -- Sets "publication" field in microdesc consensus "r" lines + to a meaningless value. + "34" -- Removes "package" lines from consensus. +``` + +Before generating a consensus, an authority must decide which consensus +method to use. To do this, it looks for the highest version number +supported by more than 2/3 of the authorities voting. If it supports this +method, then it uses it. Otherwise, it falls back to the newest consensus +method that it supports (which will probably not result in a sufficiently +signed consensus). + +All authorities MUST support method 25; authorities SHOULD support +more recent methods as well. Authorities SHOULD NOT support or +advertise support for any method before 25. Clients MAY assume that +they will never see a current valid signed consensus for any method +before method 25. + +(The consensuses generated by new methods must be parsable by +implementations that only understand the old methods, and must not cause +those implementations to compromise their anonymity. This is a means for +making changes in the contents of consensus; not for making +backward-incompatible changes in their format.) + +The following methods have incorrect implementations; authorities SHOULD +NOT advertise support for them: + +"21" -- Did not correctly enable support for ed25519 key collation. + + +### Encoding port lists + +Whether the summary shows the list of accepted ports or the list of +rejected ports depends on which list is shorter (has a shorter string +representation). In case of ties we choose the list of accepted +ports. As an exception to this rule an allow-all policy is +represented as "accept 1-65535" instead of "reject " and a reject-all +policy is similarly given as "reject 1-65535". + +Summary items are compressed, that is instead of "80-88,89-100" there +only is a single item of "80-100", similarly instead of "20,21" a +summary will say "20-21". + +Port lists are sorted in ascending order. + +The maximum allowed length of a policy summary (including the "accept " +or "reject ") is 1000 characters. If a summary exceeds that length we +use an accept-style summary and list as much of the port list as is +possible within these 1000 bytes. [XXXX be more specific.] + + +### Computing Bandwidth Weights + +Let weight_scale = 10000, or the value of the "bwweightscale" parameter. +(Before consensus method 31 there was a bug in parsing bwweightscale, so +that if there were any consensus parameters after it alphabetically, it +would always be treated as 10000. A similar bug existed for +"maxunmeasuredbw".) + +Starting with consensus method 26, G, M, E, and D are initialized to 1 and +T to 4. Prior consensus methods initialize them all to 0. With this change, +test tor networks that are small or new are much more likely to produce +bandwidth-weights in their consensus. The extra bandwidth has a negligible +impact on the bandwidth weights in the public tor network. + +Let G be the total bandwidth for Guard-flagged nodes. +Let M be the total bandwidth for non-flagged nodes. +Let E be the total bandwidth for Exit-flagged nodes. +Let D be the total bandwidth for Guard+Exit-flagged nodes. +Let T = G+M+E+D + +Let Wgd be the weight for choosing a Guard+Exit for the guard position. +Let Wmd be the weight for choosing a Guard+Exit for the middle position. +Let Wed be the weight for choosing a Guard+Exit for the exit position. + +Let Wme be the weight for choosing an Exit for the middle position. +Let Wmg be the weight for choosing a Guard for the middle position. + +Let Wgg be the weight for choosing a Guard for the guard position. +Let Wee be the weight for choosing an Exit for the exit position. + +Balanced network conditions then arise from solutions to the following +system of equations: + +Wgg*G + Wgd*D == M + Wmd*D + Wme*E + Wmg*G (guard bw = middle bw) +Wgg*G + Wgd*D == Wee*E + Wed*D (guard bw = exit bw) +Wed*D + Wmd*D + Wgd*D == D (aka: Wed+Wmd+Wdg = weight_scale) +Wmg*G + Wgg*G == G (aka: Wgg = weight_scale-Wmg) +Wme*E + Wee*E == E (aka: Wee = weight_scale-Wme) + +We are short 2 constraints with the above set. The remaining constraints +come from examining different cases of network load. The following +constraints are used in consensus method 10 and above. There are another +incorrect and obsolete set of constraints used for these same cases in +consensus method 9. For those, see dir-spec.txt in Tor 0.2.2.10-alpha +to 0.2.2.16-alpha. + +Case 1: E >= T/3 && G >= T/3 (Neither Exit nor Guard Scarce) + +In this case, the additional two constraints are: Wmg == Wmd, +Wed == 1/3. + +```text + This leads to the solution: + Wgd = weight_scale/3 + Wed = weight_scale/3 + Wmd = weight_scale/3 + Wee = (weight_scale*(E+G+M))/(3*E) + Wme = weight_scale - Wee + Wmg = (weight_scale*(2*G-E-M))/(3*G) + Wgg = weight_scale - Wmg + + Case 2: E < T/3 && G < T/3 (Both are scarce) +``` + +Let R denote the more scarce class (Rare) between Guard vs Exit. +Let S denote the less scarce class. + +Subcase a: R+D < S + +In this subcase, we simply devote all of D bandwidth to the +scarce class. + +```text + Wgg = Wee = weight_scale + Wmg = Wme = Wmd = 0; + if E < G: + Wed = weight_scale + Wgd = 0 + else: + Wed = 0 + Wgd = weight_scale + + Subcase b: R+D >= S +``` + +In this case, if M <= T/3, we have enough bandwidth to try to achieve +a balancing condition. + +Add constraints Wgg = weight_scale, Wmd == Wgd to maximize bandwidth in +the guard position while still allowing exits to be used as middle nodes: + +Wee = (weight_scale*(E - G + M))/E +Wed = (weight_scale*(D - 2*E + 4*G - 2*M))/(3*D) +Wme = (weight_scale*(G-M))/E +Wmg = 0 +Wgg = weight_scale +Wmd = (weight_scale - Wed)/2 +Wgd = (weight_scale - Wed)/2 + +If this system ends up with any values out of range (ie negative, or +above weight_scale), use the constraints Wgg == weight_scale and Wee == +weight_scale, since both those positions are scarce: + +```text + Wgg = weight_scale + Wee = weight_scale + Wed = (weight_scale*(D - 2*E + G + M))/(3*D) + Wmd = (weight_Scale*(D - 2*M + G + E))/(3*D) + Wme = 0 + Wmg = 0 + Wgd = weight_scale - Wed - Wmd + + If M > T/3, then the Wmd weight above will become negative. Set it to 0 + in this case: + Wmd = 0 + Wgd = weight_scale - Wed + + Case 3: One of E < T/3 or G < T/3 + + Let S be the scarce class (of E or G). + + Subcase a: (S+D) < T/3: + if G=S: + Wgg = Wgd = weight_scale; + Wmd = Wed = Wmg = 0; + // Minor subcase, if E is more scarce than M, + // keep its bandwidth in place. + if (E < M) Wme = 0; + else Wme = (weight_scale*(E-M))/(2*E); + Wee = weight_scale-Wme; + if E=S: + Wee = Wed = weight_scale; + Wmd = Wgd = Wme = 0; + // Minor subcase, if G is more scarce than M, + // keep its bandwidth in place. + if (G < M) Wmg = 0; + else Wmg = (weight_scale*(G-M))/(2*G); + Wgg = weight_scale-Wmg; + + Subcase b: (S+D) >= T/3 + if G=S: + Add constraints Wgg = weight_scale, Wmd == Wed to maximize bandwidth + in the guard position, while still allowing exits to be + used as middle nodes: + Wgg = weight_scale + Wgd = (weight_scale*(D - 2*G + E + M))/(3*D) + Wmg = 0 + Wee = (weight_scale*(E+M))/(2*E) + Wme = weight_scale - Wee + Wmd = (weight_scale - Wgd)/2 + Wed = (weight_scale - Wgd)/2 + if E=S: + Add constraints Wee == weight_scale, Wmd == Wgd to maximize bandwidth + in the exit position: + Wee = weight_scale; + Wed = (weight_scale*(D - 2*E + G + M))/(3*D); + Wme = 0; + Wgg = (weight_scale*(G+M))/(2*G); + Wmg = weight_scale - Wgg; + Wmd = (weight_scale - Wed)/2; + Wgd = (weight_scale - Wed)/2; +``` + +To ensure consensus, all calculations are performed using integer math +with a fixed precision determined by the bwweightscale consensus +parameter (defaults at 10000, Min: 1, Max: INT32_MAX). (See note above +about parsing bug in bwweightscale before consensus method 31.) + +For future balancing improvements, Tor clients support 11 additional weights +for directory requests and middle weighting. These weights are currently +set at weight_scale, with the exception of the following groups of +assignments: + +Directory requests use middle weights: + +Wbd=Wmd, Wbg=Wmg, Wbe=Wme, Wbm=Wmm + +Handle bridges and strange exit policies: + +Wgm=Wgg, Wem=Wee, Weg=Wed + + +## Computing consensus flavors + +Consensus flavors are variants of the consensus that clients can choose +to download and use instead of the unflavored consensus. The purpose +of a consensus flavor is to remove or replace information in the +unflavored consensus without forcing clients to download information +they would not use anyway. + +Directory authorities can produce and serve an arbitrary number of +flavors of the same consensus. A downside of creating too many new +flavors is that clients will be distinguishable based on which flavor +they download. A new flavor should not be created when adding a field +instead wouldn't be too onerous. + +Examples for consensus flavors include: + +```text + - Publishing hashes of microdescriptors instead of hashes of + full descriptors (see section 3.9.2). + - Including different digests of descriptors, instead of the + perhaps-soon-to-be-totally-broken SHA1. +``` + +Consensus flavors are derived from the unflavored consensus once the +voting process is complete. This is to avoid consensus synchronization +problems. + +Every consensus flavor has a name consisting of a sequence of one +or more alphanumeric characters and dashes. For compatibility, +the original (unflavored) consensus type is called "ns". + +The supported consensus flavors are defined as part of the +authorities' consensus method. + +All consensus flavors have in common that their first line is +"network-status-version" where version is 3 or higher, and the flavor +is a string consisting of alphanumeric characters and dashes: + +"network-status-version" SP version [SP flavor] NL + + +### ns consensus + +The ns consensus flavor is equivalent to the unflavored consensus. +When the flavor is omitted from the "network-status-version" line, +it should be assumed to be "ns". Some implementations may explicitly +state that the flavor is "ns" when generating consensuses, but should +accept consensuses where the flavor is omitted. + + +### Microdescriptor consensus + +The microdescriptor consensus is a consensus flavor that contains +microdescriptor hashes instead of descriptor hashes and that omits +exit-policy summaries which are contained in microdescriptors. The +microdescriptor consensus was designed to contain elements that are +small and frequently changing. Clients use the information in the +microdescriptor consensus to decide which servers to fetch information +about and which servers to fetch information from. + +The microdescriptor consensus is based on the unflavored consensus with +the exceptions as follows: + +"network-status-version" SP version SP "microdesc" NL + +[At start, exactly once.] + +The flavor name of a microdescriptor consensus is "microdesc". + +Changes to router status entries are as follows: + +```text + "r" SP nickname SP identity SP publication SP IP SP ORPort + SP DirPort NL + + [At start, exactly once.] + + Similar to "r" lines in section 3.4.1, but without the digest element. + + "a" SP address ":" port NL + + [Any number] + + Identical to the "r" lines in section 3.4.1. +``` + +(Only included when the vote is generated with consensus-method 14 +or later, and the consensus is generated with consensus-method 27 or +later.) + +"p" ... NL + +[At most once] + +Not currently generated. + +Exit policy summaries are contained in microdescriptors and +therefore omitted in the microdescriptor consensus. + +"m" SP digest NL + +[Exactly once.*] + +"digest" is the base64 of the SHA256 hash of the router's +microdescriptor with trailing =s omitted. For a given router +descriptor digest and consensus method there should only be a +single microdescriptor digest in the "m" lines of all votes. +If different votes have different microdescriptor digests for +the same descriptor digest and consensus method, at least one +of the authorities is broken. If this happens, the microdesc +consensus should contain whichever microdescriptor digest is +most common. If there is no winner, we break ties in the favor +of the lexically earliest. + +[*Before consensus method 13, this field was sometimes erroneously +omitted.] + +Additionally, a microdescriptor consensus SHOULD use the sha256 digest +algorithm for its signatures. + + +## Exchanging detached signatures + +Once an authority has computed and signed a consensus network status, it +should send its detached signature to each other authority in an HTTP POST +request to the URL: + +http:///tor/post/consensus-signature + +[XXX Note why we support push-and-then-pull.] + +All of the detached signatures it knows for consensus status should be +available at: + +http:///tor/status-vote/next/consensus-signatures.z + +Assuming full connectivity, every authority should compute and sign the +same consensus including any flavors in each period. Therefore, it +isn't necessary to download the consensus or any flavors of it computed +by each authority; instead, the authorities only push/fetch each +others' signatures. A "detached signature" document contains items as +follows: + +"consensus-digest" SP Digest NL + +[At start, at most once.] + +The digest of the consensus being signed. + +"valid-after" SP YYYY-MM-DD SP HH:MM:SS NL +"fresh-until" SP YYYY-MM-DD SP HH:MM:SS NL +"valid-until" SP YYYY-MM-DD SP HH:MM:SS NL + +[As in the consensus] + +"additional-digest" SP flavor SP algname SP digest NL + +[Any number.] + +For each supported consensus flavor, every directory authority +adds one or more "additional-digest" lines. "flavor" is the name +of the consensus flavor, "algname" is the name of the hash +algorithm that is used to generate the digest, and "digest" is the +hex-encoded digest. + +The hash algorithm for the microdescriptor consensus flavor is +defined as SHA256 with algname "sha256". + +```text + "additional-signature" SP flavor SP algname SP identity SP + signing-key-digest NL signature. + + [Any number.] +``` + +For each supported consensus flavor and defined digest algorithm, +every directory authority adds an "additional-signature" line. +"flavor" is the name of the consensus flavor. "algname" is the +name of the algorithm that was used to hash the identity and +signing keys, and to compute the signature. "identity" is the +hex-encoded digest of the authority identity key of the signing +authority, and "signing-key-digest" is the hex-encoded digest of +the current authority signing key of the signing authority. + +The "sha256" signature format is defined as the RSA signature of +the OAEP+-padded SHA256 digest of the item to be signed. When +checking signatures, the signature MUST be treated as valid if the +signature material begins with SHA256(document), so that other +data can get added later. +[To be honest, I didn't fully understand the previous paragraph +and only copied it from the proposals. Review carefully. -KL] + +"directory-signature" + +[As in the consensus; the signature object is the same as in the +consensus document.] + + +## Publishing the signed consensus + +The voting period ends at the valid-after time. If the consensus has +been signed by a majority of authorities, these documents are made +available at + +http:///tor/status-vote/current/consensus.z + +and + +http:///tor/status-vote/current/consensus-signatures.z + +```text + [XXX current/consensus-signatures is not currently implemented, as it + is not used in the voting protocol.] + + [XXX possible future features include support for downloading old + consensuses.] + + The other vote documents are analogously made available under +``` + +http:///tor/status-vote/current/authority.z +http:///tor/status-vote/current/.z +http:///tor/status-vote/current/d/.z +http:///tor/status-vote/current/bandwidth.z + +once the voting period ends, regardless of the number of signatures. + +The authorities serve another consensus of each flavor "F" from the +locations + +/tor/status-vote/(current|next)/consensus-F.z. and +/tor/status-vote/(current|next)/consensus-F/+....z. + +The standard URLs for bandwidth list files first-appeared in Tor 0.3.5. + diff --git a/spec/dir-spec/standards-compliance.md b/spec/dir-spec/standards-compliance.md new file mode 100644 index 0000000..f20125e --- /dev/null +++ b/spec/dir-spec/standards-compliance.md @@ -0,0 +1,79 @@ + +# Standards compliance + +All clients and servers MUST support HTTP 1.0. Clients and servers MAY +support later versions of HTTP as well. + + +## HTTP headers + +Servers SHOULD set Content-Encoding to the algorithm used to compress the +document(s) being served. Recognized algorithms are: + +```text + - "identity" -- RFC2616 section 3.5 + - "deflate" -- RFC2616 section 3.5 + - "gzip" -- RFC2616 section 3.5 + - "x-zstd" -- The zstandard compression algorithm (www.zstd.net) + - "x-tor-lzma" -- The lzma compression algorithm, with a "preset" + value no higher than 6. +``` + +Clients SHOULD use Accept-Encoding on most directory requests to indicate +which of the above compression algorithms they support. If they omit it +(as Tor clients did before 0.3.1.1-alpha), then the server should serve +only "deflate" or "identity" encoded documents, based on the presence or +absence of the ".z" suffix on the requested URL. + +Note that for anonymous directory requests (that is, requests made over +multi-hop circuits, like those for onion service lookups) implementations +SHOULD NOT advertise any Accept-Encoding values other than deflate. To do +so would be to create a fingerprinting opportunity. + +When receiving multiple documents, clients MUST accept compressed +concatenated documents and concatenated compressed documents as +equivalent. + +Servers MAY set the Content-Length: header. When they do, it should +match the number of compressed bytes that they are sending. + +Servers MAY include an X-Your-Address-Is: header, whose value is the +apparent IP address of the client connecting to them (as a dotted quad). +For directory connections tunneled over a BEGIN_DIR stream, servers SHOULD +report the IP from which the circuit carrying the BEGIN_DIR stream reached +them. + +Servers SHOULD disable caching of multiple network statuses or multiple +server descriptors. Servers MAY enable caching of single descriptors, +single network statuses, the list of all server descriptors, a v1 +directory, or a v1 running routers document. XXX mention times. + + +## HTTP status codes + +Tor delivers the following status codes. Some were chosen without much +thought; other code SHOULD NOT rely on specific status codes yet. + +```text + 200 -- the operation completed successfully + -- the user requested statuses or serverdescs, and none of the ones we + requested were found (0.2.0.4-alpha and earlier). + + 304 -- the client specified an if-modified-since time, and none of the + requested resources have changed since that time. + + 400 -- the request is malformed, or + -- the URL is for a malformed variation of one of the URLs we support, + or + -- the client tried to post to a non-authority, or + -- the authority rejected a malformed posted document, or + + 404 -- the requested document was not found. + -- the user requested statuses or serverdescs, and none of the ones + requested were found (0.2.0.5-alpha and later). + + 503 -- we are declining the request in order to save bandwidth + -- user requested some items that we ordinarily generate or store, + but we do not have any available. +``` + diff --git a/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md b/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md new file mode 100644 index 0000000..f58796e --- /dev/null +++ b/spec/dir-spec/uploading-server-descriptors-extra-info-documents.md @@ -0,0 +1,43 @@ + +## Uploading server descriptors and extra-info documents + +ORs SHOULD generate a new server descriptor and a new extra-info +document whenever any of the following events have occurred: + +```text + - A period of time (18 hrs by default) has passed since the last + time a descriptor was generated. + + - A descriptor field other than bandwidth or uptime has changed. + + - Its uptime is less than 24h and bandwidth has changed by a factor of 2 + from the last time a descriptor was generated, and at least a given + interval of time (3 hours by default) has passed since then. + + - Its uptime has been reset (by restarting). + + - It receives a networkstatus consensus in which it is not listed. + + - It receives a networkstatus consensus in which it is listed + with the StaleDesc flag. + + [XXX this list is incomplete; see router_differences_are_cosmetic() + in routerlist.c for others] +``` + +ORs SHOULD NOT publish a new server descriptor or extra-info document +if none of the above events have occurred and not much time has passed +(12 hours by default). + +Tor versions older than 0.3.5.1-alpha ignore uptime when checking for +bandwidth changes. + +After generating a descriptor, ORs upload them to every directory +authority they know, by posting them (in order) to the URL + +http:///tor/ + +Server descriptors may not exceed 20,000 bytes in length; extra-info +documents may not exceed 50,000 bytes in length. If they do, the +authorities SHOULD reject them. + diff --git a/spec/dir-spec/vote-consensus-status-document-formats.md b/spec/dir-spec/vote-consensus-status-document-formats.md new file mode 100644 index 0000000..4ed6db6 --- /dev/null +++ b/spec/dir-spec/vote-consensus-status-document-formats.md @@ -0,0 +1,706 @@ + +### Vote and consensus status document formats + +Votes and consensuses are more strictly formatted than other documents +in this specification, since different authorities must be able to +generate exactly the same consensus given the same set of votes. + +The procedure for deciding when to generate vote and consensus status +documents are described in section 1.4 on the voting timeline. + +Status documents contain a preamble, an authority section, a list of +router status entries, and one or more footer signature, in that order. + +Unlike other formats described above, a SP in these documents must be a +single space character (hex 20). + +Some items appear only in votes, and some items appear only in +consensuses. Unless specified, items occur in both. + +The preamble contains the following items. They SHOULD occur in the +order given here: + +"network-status-version" SP version NL + +[At start, exactly once.] + +A document format version. For this specification, the version is +"3". + +"vote-status" SP type NL + +[Exactly once.] + +The status MUST be "vote" or "consensus", depending on the type of +the document. + +"consensus-methods" SP IntegerList NL + +[At most once for votes; does not occur in consensuses.] + +A space-separated list of supported methods for generating +consensuses from votes. See section 3.8.1 for details. Absence of +the line means that only method "1" is supported. + +"consensus-method" SP Integer NL + +[At most once for consensuses; does not occur in votes.] +[No extra arguments] + +See section 3.8.1 for details. + +(Only included when the vote is generated with consensus-method 2 or +later.) + +"published" SP YYYY-MM-DD SP HH:MM:SS NL + +[Exactly once for votes; does not occur in consensuses.] + +The publication time for this status document (if a vote). + +"valid-after" SP YYYY-MM-DD SP HH:MM:SS NL + +[Exactly once.] + +The start of the Interval for this vote. Before this time, the +consensus document produced from this vote is not officially in +use. + +(Note that because of propagation delays, clients and relays +may see consensus documents that are up to `DistSeconds` +earlier than this time, and should not warn about them.) + +See section 1.4 for voting timeline information. + +"fresh-until" SP YYYY-MM-DD SP HH:MM:SS NL + +[Exactly once.] + +The time at which the next consensus should be produced; before this +time, there is no point in downloading another consensus, since there +won't be a new one. See section 1.4 for voting timeline information. + +"valid-until" SP YYYY-MM-DD SP HH:MM:SS NL + +[Exactly once.] + +The end of the Interval for this vote. After this time, all +clients should try to find a more recent consensus. See section 1.4 +for voting timeline information. + +In practice, clients continue to use the consensus for up to 24 hours +after it is no longer valid, if no more recent consensus can be +downloaded. + +"voting-delay" SP VoteSeconds SP DistSeconds NL + +[Exactly once.] + +VoteSeconds is the number of seconds that we will allow to collect +votes from all authorities; DistSeconds is the number of seconds +we'll allow to collect signatures from all authorities. See +section 1.4 for voting timeline information. + +"client-versions" SP VersionList NL + +[At most once.] + +A comma-separated list of recommended Tor versions for client +usage, in ascending order. The versions are given as defined by +version-spec.txt. If absent, no opinion is held about client +versions. + +"server-versions" SP VersionList NL + +[At most once.] + +A comma-separated list of recommended Tor versions for relay +usage, in ascending order. The versions are given as defined by +version-spec.txt. If absent, no opinion is held about server +versions. + +"package" SP PackageName SP Version SP URL SP DIGESTS NL + +[Any number of times.] + +For this element: + +```text + PACKAGENAME = NONSPACE + VERSION = NONSPACE + URL = NONSPACE + DIGESTS = DIGEST | DIGESTS SP DIGEST + DIGEST = DIGESTTYPE "=" DIGESTVAL + NONSPACE = one or more non-space printing characters + DIGESTVAL = DIGESTTYPE = one or more non-space printing characters + other than "=". +``` + +Indicates that a package called "package" of version VERSION may be +found at URL, and its digest as computed with DIGESTTYPE is equal to +DIGESTVAL. In consensuses, these lines are sorted lexically by +"PACKAGENAME VERSION" pairs, and DIGESTTYPES must appear in ascending +order. A consensus must not contain the same "PACKAGENAME VERSION" +more than once. If a vote contains the same "PACKAGENAME VERSION" +more than once, all but the last is ignored. + +Included in consensuses only for methods 19-33. Earlier methods +did not include this; method 34 removed it. + +"known-flags" SP FlagList NL + +[Exactly once.] + +A space-separated list of all of the flags that this document +might contain. A flag is "known" either because the authority +knows about them and might set them (if in a vote), or because +enough votes were counted for the consensus for an authoritative +opinion to have been formed about their status. + +"flag-thresholds" SP Thresholds NL + +[At most once for votes; does not occur in consensuses.] + +```text + A space-separated list of the internal performance thresholds + that the directory authority had at the moment it was forming + a vote. + + The metaformat is: + Thresholds = Threshold | Threshold SP Thresholds + Threshold = ThresholdKey '=' ThresholdVal + ThresholdKey = (KeywordChar | "_") + + ThresholdVal = [0-9]+("."[0-9]+)? "%"? + + Commonly used Thresholds at this point include: + + "stable-uptime" -- Uptime (in seconds) required for a relay + to be marked as stable. + + "stable-mtbf" -- MTBF (in seconds) required for a relay to be + marked as stable. + + "enough-mtbf" -- Whether we have measured enough MTBF to look + at stable-mtbf instead of stable-uptime. + + "fast-speed" -- Bandwidth (in bytes per second) required for + a relay to be marked as fast. + + "guard-wfu" -- WFU (in seconds) required for a relay to be + marked as guard. + + "guard-tk" -- Weighted Time Known (in seconds) required for a + relay to be marked as guard. + + "guard-bw-inc-exits" -- If exits can be guards, then all guards + must have a bandwidth this high. + + "guard-bw-exc-exits" -- If exits can't be guards, then all guards + must have a bandwidth this high. + + "ignoring-advertised-bws" -- 1 if we have enough measured bandwidths + that we'll ignore the advertised bandwidth + claims of routers without measured bandwidth. +``` + +"recommended-client-protocols" SP Entries NL +"recommended-relay-protocols" SP Entries NL +"required-client-protocols" SP Entries NL +"required-relay-protocols" SP Entries NL + +[At most once for each.] + +The "proto" element as specified in section 2.1.1. + +To vote on these entries, a protocol/version combination is included +only if it is listed by a majority of the voters. + +These lines should be voted on. A majority of votes is sufficient to +make a protocol un-supported. A supermajority of authorities (2/3) +are needed to make a protocol required. The required protocols +should not be torrc-configurable, but rather should be hardwired in +the Tor code. + +The tor-spec.txt section 9 details how a relay and a client should +behave when they encounter these lines in the consensus. + +"params" SP [Parameters] NL + +[At most once] + +Parameter ::= Keyword '=' Int32 +Int32 ::= A decimal integer between -2147483648 and 2147483647. +Parameters ::= Parameter | Parameters SP Parameter + +The parameters list, if present, contains a space-separated list of +case-sensitive key-value pairs, sorted in lexical order by their +keyword (as ASCII byte strings). Each parameter has its own meaning. + +(Only included when the vote is generated with consensus-method 7 or +later.) + +See param-spec.txt for a list of parameters and their meanings. + +"shared-rand-previous-value" SP NumReveals SP Value NL + +[At most once] + +NumReveals ::= An integer greater or equal to 0. +Value ::= Base64-encoded-data + +The shared_random_value that was generated during the second-to-last +shared randomness protocol run. For example, if this document was +created on the 5th of November, this field carries the shared random +value generated during the protocol run of the 3rd of November. + +See section [SRCALC] of srv-spec.txt for instructions on how to compute +this value, and see section [CONS] for why we include old shared random +values in votes and consensus. + +Value is the actual shared random value encoded in base64. It will +be exactly 256 bits long. NumReveals is the number of commits used +to generate this SRV. + +"shared-rand-current-value" SP NumReveals SP Value NL + +[At most once] + +NumReveals ::= An integer greater or equal to 0. +Value ::= Base64-encoded-data + +The shared_random_value that was generated during the latest shared +randomness protocol run. For example, if this document was created on +the 5th of November, this field carries the shared random value +generated during the protocol run of the 4th of November + +See section [SRCALC] of srv-spec.txt for instructions on how to compute +this value given the active commits. + +Value is the actual shared random value encoded in base64. It will +be exactly 256 bits long. NumReveals is the number of commits used to +generate this SRV. + +"bandwidth-file-headers" SP KeyValues NL + +[At most once for votes; does not occur in consensuses.] + +KeyValues ::= "" | KeyValue | KeyValues SP KeyValue +KeyValue ::= Keyword '=' Value +Value ::= ArgumentCharValue+ +ArgumentCharValue ::= any printing ASCII character except NL and SP. + +The headers from the bandwidth file used to generate this vote. +The bandwidth file headers are described in bandwidth-file-spec.txt. + +If an authority is not configured with a V3BandwidthsFile, this line +SHOULD NOT appear in its vote. + +If an authority is configured with a V3BandwidthsFile, but parsing +fails, this line SHOULD appear in its vote, but without any headers. + +First-appeared: Tor 0.3.5.1-alpha. + +"bandwidth-file-digest" 1*(SP algorithm "=" digest) NL + +[At most once for votes; does not occur in consensuses.] + +A digest of the bandwidth file used to generate this vote. +"algorithm" is the name of the hash algorithm producing "digest", +which can be "sha256" or another algorithm. "digest" is the +base64 encoding of the hash of the bandwidth file, with trailing =s +omitted. + +If an authority is not configured with a V3BandwidthsFile, this line +SHOULD NOT appear in its vote. + +If an authority is configured with a V3BandwidthsFile, but parsing +fails, this line SHOULD appear in its vote, with the digest(s) of the +unparseable file. + +First-appeared: Tor 0.4.0.4-alpha + +The authority section of a vote contains the following items, followed +in turn by the authority's current key certificate: + +```text + "dir-source" SP nickname SP identity SP address SP IP SP dirport SP + orport NL + + [Exactly once, at start] +``` + +Describes this authority. The nickname is a convenient identifier +for the authority. The identity is an uppercase hex fingerprint of +the authority's current (v3 authority) identity key. The address is +the server's hostname. The IP is the server's current IP address, +and dirport is its current directory port. The orport is the +port at that address where the authority listens for OR +connections. + +"contact" SP string NL + +[Exactly once] + +An arbitrary string describing how to contact the directory +server's administrator. Administrators should include at least an +email address and a PGP fingerprint. + +"legacy-dir-key" SP FINGERPRINT NL + +[At most once] + +Lists a fingerprint for an obsolete _identity_ key still used +by this authority to keep older clients working. This option +is used to keep key around for a little while in case the +authorities need to migrate many identity keys at once. +(Generally, this would only happen because of a security +vulnerability that affected multiple authorities, like the +Debian OpenSSL RNG bug of May 2008.) + +"shared-rand-participate" NL + +[At most once] + +Denotes that the directory authority supports and can participate in the +shared random protocol. + +"shared-rand-commit" SP Version SP AlgName SP Identity SP Commit [SP Reveal] NL + +[Any number of times] + +Version ::= An integer greater or equal to 0. +AlgName ::= 1*(ALPHA / DIGIT / "_" / "-") +Identity ::= 40 * HEXDIG +Commit ::= Base64-encoded-data +Reveal ::= Base64-encoded-data + +Denotes a directory authority commit for the shared randomness +protocol, containing the commitment value and potentially also the +reveal value. See sections [COMMITREVEAL] and [VALIDATEVALUES] of +srv-spec.txt on how to generate and validate these values. + +Version is the current shared randomness protocol version. AlgName is +the hash algorithm that is used (e.g. "sha3-256") and Identity is the +authority's SHA1 v3 identity fingerprint. Commit is the encoded +commitment value in base64. Reveal is optional and if it's set, it +contains the reveal value in base64. + +If a vote contains multiple commits from the same authority, the +receiver MUST only consider the first commit listed. + +"shared-rand-previous-value" SP NumReveals SP Value NL + +[At most once] + +See shared-rand-previous-value description above. + +"shared-rand-current-value" SP NumReveals SP Value NL + +[At most once] + +See shared-rand-current-value description above. + +The authority section of a consensus contains groups of the following items, +in the order given, with one group for each authority that contributed to +the consensus, with groups sorted by authority identity digest: + +```text + "dir-source" SP nickname SP identity SP address SP IP SP dirport SP + orport NL + + [Exactly once, at start] + + As in the authority section of a vote. + + "contact" SP string NL + + [Exactly once.] + + As in the authority section of a vote. + + "vote-digest" SP digest NL + + [Exactly once.] +``` + +A digest of the vote from the authority that contributed to this +consensus, as signed (that is, not including the signature). +(Hex, upper-case.) + +For each "legacy-dir-key" in the vote, there is an additional "dir-source" +line containing that legacy key's fingerprint, the authority's nickname +with "-legacy" appended, and all other fields as in the main "dir-source" +line for that authority. These "dir-source" lines do not have +corresponding "contact" or "vote-digest" entries. + +Each router status entry contains the following items. Router status +entries are sorted in ascending order by identity digest. + +```text + "r" SP nickname SP identity SP digest SP publication SP IP SP ORPort + SP DirPort NL + + [At start, exactly once.] +``` + +"Nickname" is the OR's nickname. "Identity" is a hash of its +identity key, encoded in base64, with trailing equals sign(s) +removed. "Digest" is a hash of its most recent descriptor as +signed (that is, not including the signature) by the RSA identity +key (see section 1.3.), encoded in base64. + +"Publication" was once the publication time of the router's most +recent descriptor, in the form YYYY-MM-DD HH:MM:SS, in UTC. Now +it is only used in votes, and may be set to a fixed value in +consensus documents. Implementations SHOULD ignore this value +in non-vote documents. + +"IP" is its current IP address; ORPort is its current OR port, +"DirPort" is its current directory port, or "0" for "none". + +"a" SP address ":" port NL + +[Any number] + +The first advertised IPv6 address for the OR, if it is reachable. + +Present only if the OR advertises at least one IPv6 address, and the +authority believes that the first advertised address is reachable. +Any other IPv4 or IPv6 addresses should be ignored. + +Address and port are as for "or-address" as specified in +section 2.1.1. + +(Only included when the vote or consensus is generated with +consensus-method 14 or later.) + +"s" SP Flags NL + +[Exactly once.] + +A series of space-separated status flags, in lexical order (as ASCII +byte strings). Currently documented flags are: + +```text + "Authority" if the router is a directory authority. + "BadExit" if the router is believed to be useless as an exit node + (because its ISP censors it, because it is behind a restrictive + proxy, or for some similar reason). + "Exit" if the router is more useful for building + general-purpose exit circuits than for relay circuits. The + path building algorithm uses this flag; see path-spec.txt. + "Fast" if the router is suitable for high-bandwidth circuits. + "Guard" if the router is suitable for use as an entry guard. + "HSDir" if the router is considered a v2 hidden service directory. + "MiddleOnly" if the router is considered unsuitable for + usage other than as a middle relay. Clients do not need + to handle this option, since when it is present, the authorities + will automatically vote against flags that would make the router + usable in other positions. (Since 0.4.7.2-alpha.) + "NoEdConsensus" if any Ed25519 key in the router's descriptor or + microdescriptor does not reflect authority consensus. + "Stable" if the router is suitable for long-lived circuits. + "StaleDesc" if the router should upload a new descriptor because + the old one is too old. + "Running" if the router is currently usable over all its published + ORPorts. (Authorities ignore IPv6 ORPorts unless configured to + check IPv6 reachability.) Relays without this flag are omitted + from the consensus, and current clients (since 0.2.9.4-alpha) + assume that every listed relay has this flag. + "Valid" if the router has been 'validated'. Clients before + 0.2.9.4-alpha would not use routers without this flag by + default. Currently, relays without this flag are omitted + from the consensus, and current (post-0.2.9.4-alpha) clients + assume that every listed relay has this flag. + "V2Dir" if the router implements the v2 directory protocol or + higher. + + "v" SP version NL + + [At most once.] +``` + +The version of the Tor protocol that this relay is running. If +the value begins with "Tor" SP, the rest of the string is a Tor +version number, and the protocol is "The Tor protocol as supported +by the given version of Tor." Otherwise, if the value begins with +some other string, Tor has upgraded to a more sophisticated +protocol versioning system, and the protocol is "a version of the +Tor protocol more recent than any we recognize." + +Directory authorities SHOULD omit version strings they receive from +descriptors if they would cause "v" lines to be over 128 characters +long. + +"pr" SP Entries NL + +[At most once.] + +The "proto" family element as specified in section 2.1.1. + +During voting, authorities copy these lines immediately below the "v" +lines. When a descriptor does not contain a "proto" entry, the +authorities should reconstruct it using the approach described below +in section D. They are included in the consensus using the same rules +as currently used for "v" lines, if a sufficiently late consensus +method is in use. + +"w" SP "Bandwidth=" INT [SP "Measured=" INT] [SP "Unmeasured=1"] NL + +[At most once.] + +An estimate of the bandwidth of this relay, in an arbitrary +unit (currently kilobytes per second). Used to weight router +selection. See section 3.4.2 for details on how the value of +Bandwidth is determined in a consensus. + +Additionally, the Measured= keyword is present in votes by +participating bandwidth measurement authorities to indicate +a measured bandwidth currently produced by measuring stream +capacities. It does not occur in consensuses. + +'Bandwidth=' and 'Measured=' values must be between 0 and +2^32 - 1 inclusive. + +The "Unmeasured=1" value is included in consensuses generated +with method 17 or later when the 'Bandwidth=' value is not +based on a threshold of 3 or more measurements for this relay. + +Other weighting keywords may be added later. +Clients MUST ignore keywords they do not recognize. + +"p" SP ("accept" / "reject") SP PortList NL + +[At most once.] + +PortList = PortOrRange +PortList = PortList "," PortOrRange +PortOrRange = INT "-" INT / INT + +A list of those ports that this router supports (if 'accept') +or does not support (if 'reject') for exit to "most +addresses". + +"m" SP methods 1*(SP algorithm "=" digest) NL + +[Any number, only in votes.] + +Microdescriptor hashes for all consensus methods that an authority +supports and that use the same microdescriptor format. "methods" +is a comma-separated list of the consensus methods that the +authority believes will produce "digest". "algorithm" is the name +of the hash algorithm producing "digest", which can be "sha256" or +something else, depending on the consensus "methods" supporting +this algorithm. "digest" is the base64 encoding of the hash of +the router's microdescriptor with trailing =s omitted. + +```text + "id" SP "ed25519" SP ed25519-identity NL + "id" SP "ed25519" SP "none" NL + [vote only, at most once] + + "stats" SP [KeyValues] NL + + [At most once. Vote only] +``` + +KeyValue ::= Keyword '=' Number +Number ::= [0-9]+("."[0-9]+)? +KeyValues ::= KeyValue | KeyValues SP KeyValue + +Line containing various statistics that an authority has computed for +this relay. Each stats is represented as a key + value. Reported keys +are: + +```text + "wfu" - Weighted Fractional Uptime + "tk" - Weighted Time Known + "mtbf" - Mean Time Between Failure (stability) + + (As of tor-0.4.6.1-alpha) +``` + +The footer section is delineated in all votes and consensuses supporting +consensus method 9 and above with the following: + +"directory-footer" NL +[No extra arguments] + +It contains two subsections, a bandwidths-weights line and a +directory-signature. (Prior to consensus method 9, footers only contained +directory-signatures without a 'directory-footer' line or +bandwidth-weights.) + +The bandwidths-weights line appears At Most Once for a consensus. It does +not appear in votes. + +"bandwidth-weights" [SP Weights] NL + +Weight ::= Keyword '=' Int32 +Int32 ::= A decimal integer between -2147483648 and 2147483647. +Weights ::= Weight | Weights SP Weight + +List of optional weights to apply to router bandwidths during path +selection. They are sorted in lexical order (as ASCII byte strings) and +values are divided by the consensus' "bwweightscale" param. Definition +of our known entries are... + +```text + Wgg - Weight for Guard-flagged nodes in the guard position + Wgm - Weight for non-flagged nodes in the guard Position + Wgd - Weight for Guard+Exit-flagged nodes in the guard Position + + Wmg - Weight for Guard-flagged nodes in the middle Position + Wmm - Weight for non-flagged nodes in the middle Position + Wme - Weight for Exit-flagged nodes in the middle Position + Wmd - Weight for Guard+Exit flagged nodes in the middle Position + + Weg - Weight for Guard flagged nodes in the exit Position + Wem - Weight for non-flagged nodes in the exit Position + Wee - Weight for Exit-flagged nodes in the exit Position + Wed - Weight for Guard+Exit-flagged nodes in the exit Position + + Wgb - Weight for BEGIN_DIR-supporting Guard-flagged nodes + Wmb - Weight for BEGIN_DIR-supporting non-flagged nodes + Web - Weight for BEGIN_DIR-supporting Exit-flagged nodes + Wdb - Weight for BEGIN_DIR-supporting Guard+Exit-flagged nodes + + Wbg - Weight for Guard flagged nodes for BEGIN_DIR requests + Wbm - Weight for non-flagged nodes for BEGIN_DIR requests + Wbe - Weight for Exit-flagged nodes for BEGIN_DIR requests + Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests + + These values are calculated as specified in section 3.8.3. +``` + +The signature contains the following item, which appears Exactly Once +for a vote, and At Least Once for a consensus. + +```text + "directory-signature" [SP Algorithm] SP identity SP signing-key-digest + NL Signature +``` + +This is a signature of the status document, with the initial item +"network-status-version", and the signature item +"directory-signature", using the signing key. (In this case, we take +the hash through the _space_ after directory-signature, not the +newline: this ensures that all authorities sign the same thing.) +"identity" is the hex-encoded digest of the authority identity key of +the signing authority, and "signing-key-digest" is the hex-encoded +digest of the current authority signing key of the signing authority. + +The Algorithm is one of "sha1" or "sha256" if it is present; +implementations MUST ignore directory-signature entries with an +unrecognized Algorithm. "sha1" is the default, if no Algorithm is +given. The algorithm describes how to compute the hash of the +document before signing it. + +"ns"-flavored consensus documents must contain only sha1 signatures. +Votes and microdescriptor documents may contain other signature +types. Note that only one signature from each authority should be +"counted" as meaning that the authority has signed the consensus. + +(Tor clients before 0.2.3.x did not understand the 'algorithm' +field.) + diff --git a/spec/ext-orport-spec.md b/spec/ext-orport-spec.md new file mode 100644 index 0000000..7a3a50b --- /dev/null +++ b/spec/ext-orport-spec.md @@ -0,0 +1,254 @@ +```text + Extended ORPort for pluggable transports + George Kadianakis, Nick Mathewson + +Table of Contents + + 1. Overview + 2. Establishing a connection and authenticating. + 2.1. Authentication type: SAFE_COOKIE + 2.1.2. Cookie-file format + 2.1.3. SAFE_COOKIE Protocol specification + 3. The extended ORPort protocol + 3.1. Protocol + 3.2. Command descriptions + 3.2.1. USERADDR + 3.2.2. TRANSPORT + 4. Security Considerations +``` + + +# Overview + +This document describes the "Extended ORPort" protocol, a wrapper +around Tor's ordinary ORPort protocol for use by bridges that +support pluggable transports. It provides a way for server-side PTs +and bridges to exchange additional information before beginning +the actual OR connection. + +See `tor-spec.txt` for information on the regular OR protocol, and +`pt-spec.txt` for information on pluggable transports. + +This protocol was originally proposed in proposal 196, and +extended with authentication in proposal 217. + + +# Establishing a connection and authenticating. + +When a client (that is to say, a server-side pluggable transport) +connects to an Extended ORPort, the server sends: + +```text + AuthTypes [variable] + EndAuthTypes [1 octet] + + Where, + + + AuthTypes are the authentication schemes that the server supports + for this session. They are multiple concatenated 1-octet values that + take values from 1 to 255. + + EndAuthTypes is the special value 0. +``` + +The client reads the list of supported authentication schemes, +chooses one, and sends it back: + +AuthType [1 octet] + +Where, + +```text + + AuthType is the authentication scheme that the client wants to use + for this session. A valid authentication type takes values from 1 to + 255. A value of 0 means that the client did not like the + authentication types offered by the server. +``` + +If the client sent an AuthType of value 0, or an AuthType that the +server does not support, the server MUST close the connection. + + +## Authentication type: SAFE_COOKIE + +We define one authentication type: SAFE_COOKIE. Its AuthType +value is 1. It is based on the client proving to the bridge that +it can access a given "cookie" file on disk. The purpose of +authentication is to defend against cross-protocol attacks. + +If the Extended ORPort is enabled, Tor should regenerate the cookie +file on startup and store it in +$DataDirectory/extended_orport_auth_cookie. + +The location of the cookie can be overridden by using the +configuration file parameter ExtORPortCookieAuthFile, which is +defined as: + +ExtORPortCookieAuthFile + +where is a filesystem path. + + +### Cookie-file format + +The format of the cookie-file is: + +```text + StaticHeader [32 octets] + Cookie [32 octets] + + Where, + + StaticHeader is the following string: + "! Extended ORPort Auth Cookie !\x0a" + + Cookie is the shared-secret. During the SAFE_COOKIE protocol, the + cookie is called CookieString. +``` + +Extended ORPort clients MUST make sure that the StaticHeader is +present in the cookie file, before proceeding with the +authentication protocol. + + +### SAFE_COOKIE Protocol specification + +A client that performs the SAFE_COOKIE handshake begins by sending: + +ClientNonce [32 octets] + +Where, ++ ClientNonce is 32 octets of random data. + +Then, the server replies with: + +```text + ServerHash [32 octets] + ServerNonce [32 octets] + + Where, + + ServerHash is computed as: + HMAC-SHA256(CookieString, + "ExtORPort authentication server-to-client hash" | ClientNonce | ServerNonce) + + ServerNonce is 32 random octets. +``` + +Upon receiving that data, the client computes ServerHash, and +validates it against the ServerHash provided by the server. + +If the server-provided ServerHash is invalid, the client MUST +terminate the connection. + +Otherwise the client replies with: + +ClientHash [32 octets] + +```text + Where, + + ClientHash is computed as: + HMAC-SHA256(CookieString, + "ExtORPort authentication client-to-server hash" | ClientNonce | ServerNonce) +``` + +Upon receiving that data, the server computes ClientHash, and +validates it against the ClientHash provided by the client. + +Finally, the server replies with: + +Status [1 octet] + +```text + Where, + + Status is 1 if the authentication was successful. If the + authentication failed, Status is 0. +``` + + +# The extended ORPort protocol + +Once a connection is established and authenticated, the parties +communicate with the protocol described here. + + +## Protocol + +The extended server port protocol is as follows: + +```text + COMMAND [2 bytes, big-endian] + BODYLEN [2 bytes, big-endian] + BODY [BODYLEN bytes] + + Commands sent from the transport proxy to the bridge are: + + [0x0000] DONE: There is no more information to give. The next + bytes sent by the transport will be those tunneled over it. + (body ignored) + + [0x0001] USERADDR: an address:port string that represents the + client's address. + + [0x0002] TRANSPORT: a string of the name of the pluggable + transport currently in effect on the connection. + + Replies sent from tor to the proxy are: + + [0x1000] OKAY: Send the user's traffic. (body ignored) + + [0x1001] DENY: Tor would prefer not to get more traffic from + this address for a while. (body ignored) + + [0x1002] CONTROL: (Not used) + + Parties MUST ignore command codes that they do not understand. +``` + +If the server receives a recognized command that does not parse, it +MUST close the connection to the client. + + +## Command descriptions + + +### USERADDR + +```text + An ASCII string holding the TCP/IP address of the client of the + pluggable transport proxy. A Tor bridge SHOULD use that address to + collect statistics about its clients. Recognized formats are: + 1.2.3.4:5678 + [1:2::3:4]:5678 +``` + +(Current Tor versions may accept other formats, but this is a bug: +transports MUST NOT send them.) + +The string MUST not be NUL-terminated. + + +### TRANSPORT + +An ASCII string holding the name of the pluggable transport used by +the client of the pluggable transport proxy. A Tor bridge that +supports multiple transports SHOULD use that information to collect +statistics about the popularity of individual pluggable transports. + +The string MUST not be NUL-terminated. + +Pluggable transport names are C-identifiers and Tor MUST check them +for correctness. + + +# Security Considerations + +Extended ORPort or TransportControlPort do _not_ provide link +confidentiality, authentication or integrity. Sensitive data, like +cryptographic material, should not be transferred through them. + +An attacker with superuser access is able to sniff network traffic, +and capture TransportControlPort identifiers and any data passed +through those ports. + +Tor SHOULD issue a warning if the bridge operator tries to bind +Extended ORPort to a non-localhost address. + +Pluggable transport proxies SHOULD issue a warning if they are +instructed to connect to a non-localhost Extended ORPort. + diff --git a/spec/gettor-spec.md b/spec/gettor-spec.md new file mode 100644 index 0000000..274d658 --- /dev/null +++ b/spec/gettor-spec.md @@ -0,0 +1,99 @@ +```text + GetTor specification + Jacob Appelbaum + +Table of Contents + + 0. Preface + 1. Overview + 2. Implementation + 2.1. Reference implementation + 3. SMTP transport + 3.1. SMTP transport security considerations + 3.2. SMTP transport privacy considerations + 4. Other transports + 5. Implementation suggestions +``` + + +# Preface + +This document describes GetTor and how to properly implementation GetTor. + + +# Overview + +GetTor was created to resolve direct and indirect censorship of Tor's +software. In many countries and networks Tor's main website is blocked and +would-be Tor users are unable to download even the source code to the Tor +program. Other software hosted by the Tor Project is similarly censored. The +filtering of the possible download sites is sometimes easy to bypass by using +our TLS enabled website. In other cases the website and all of the mirrors are +entirely blocked; this is a situation where a user seems to actually need Tor +to fetch Tor. We discovered that it is feasible to use alternate transport +methods such as SMTP between a non-trusted third party or with IRC and XDCC. + + +# Implementation + +Any compliant GetTor implementation will implement at least a single transport +to meet the needs of a certain class of users. It should be i18n and l10n +compliant for all user facing interactions; users should be able to manually +set their language and this should serve as their preference for localization +of any software delivered. The implementation must be free software and it +should be freely available by request from the implementation that they +interface with to download any of the other software available from that +GetTor instance. Security and privacy considerations should be described on a +per transport basis. + + +## Reference implementation + +We have implemented[0] a compliant GetTor that supports SMTP as a transport. + + +# SMTP transport + +The SMTP transport for GetTor should allow users to send any RFC822 compliant +message in any known human language; GetTor should respond in whatever +language is detected with supplementary translations in the same email. +GetTor shall offer a list of all available software in the body of the email - +it should offer the software as a list of packages and their subsequent +descriptions. + + +## SMTP transport security considerations + +Any GetTor instance that offers SMTP as a transport should optionally +implement the checking of DKIM signatures to ensure that email is not forged. +Optionally GetTor should take an OpenPGP key from the user and encrypt the +response with a blinded message. + + +## SMTP transport privacy considerations + +Any GetTor instance that offers SMTP as a transport must at least store the +requester's address for the time that it takes to process a response. This +should not be written to any permanent storage medium; GetTor should function +without any long term storage excepting a cache of files that it will send to +any user who requests it. + +GetTor may optionally collect anonymized usage statistics to better understand +how GetTor[1] is in use. This must not include any personally identifying +information about any of the requester beyond language selection. + + +# Other transports + +At this time no other transports have been specified. IRC XDCC is a likely +useful system as is XMPP/Jabber with the newest OTR file sharing transport. + + +# Implementation suggestions + +It is suggested that any compliant GetTor instance should be written in a so +called "safe" language such as Python. + +[0] https://gitweb.torproject.org/gettor.git +[1] https://metrics.torproject.org/packages.html + diff --git a/spec/glossary.md b/spec/glossary.md new file mode 100644 index 0000000..476e2cd --- /dev/null +++ b/spec/glossary.md @@ -0,0 +1,216 @@ +Glossary + +The Tor Project + +This document aims to specify terms, notations, and phrases related +to Tor, as used in the Tor specification documents and other documentation. + +This glossary is not a design document; it is only a reference. + +This glossary is a work-in-progress; double-check its definitions before +citing them authoritatively. ;) + +Table of Contents + +```text + 0. Preliminaries + 1.0. Commonly used Tor configuration terms + 2.0. Tor network components + 2.1. Relays, aka OR (onion router) + 2.1.1. Specific roles + 2.2. Client, aka OP (onion proxy) + 2.3. Authorities + 2.4. Hidden Service + 2.5. Circuit + 2.6. Edge connection + 2.7. Consensus + 2.8. Descriptor + 3.0. Tor network protocols + 3.1. Link handshake + 3.2. Circuit handshake + 3.3. Hidden Service Protocol + 3.4. Directory Protocol + 4.0. General network definitions +``` + + +# Preliminaries + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL +NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in +RFC 2119. + + +## Commonly used Tor configuration terms + +ORPort - Onion Router Port +DirPort - Directory Port + + +## Tor network components + + +## Relays, aka OR (onion router) + +[Style guide: prefer the term "Relay"] + + +### Specific roles + +Exit relay: The final hop in an exit circuit before traffic leaves +the Tor network to connect to external servers. + +Non-exit relay: Relays that send and receive traffic only to +other Tor relays and Tor clients. + +Entry relay: The first hop in a Tor circuit. Can be either a guard +relay or a bridge, depending on the client's configuration. + +Guard relay: A relay that a client uses as its entry for a longer +period of time. Guard relays are rotated more slowly to prevent +attacks that can come from being exposed to too many guards. + +Bridge: A relay intentionally not listed in the public Tor +consensus, with the purpose of circumventing entities (such as +governments or ISPs) seeking to block clients from using Tor. +Currently, bridges are used only as entry relays. + +Directory cache: A relay that downloads cached directory information +from the directory authorities and serves it to clients on demand. +Any relay will act as a directory cache, if its bandwidth is high enough. + +Rendezvous point: A relay connecting a client to a hidden service. +Each party builds a three-hop circuit, meeting at the +rendezvous point. + + +## Client, aka OP (onion proxy) + +[Style: the "OP" and "onion proxy" terms are deprecated.] + + +## Authorities: + +Directory Authority: Nine total in the Tor network, operated by +trusted individuals. Directory authorities define and serve the +consensus document, defining the "state of the network." This document +contains a "router status" section for every relay currently +in the network. Directory authorities also serve router descriptors, +extra info documents, microdescriptors, and the microdescriptor consensus. + +Bridge Authority: One total. Similar in responsibility to directory +authorities, but for bridges. + +Fallback directory mirror: One of a list of directory caches distributed +with the Tor software. (When a client first connects to the network, and +has no directory information, it asks a fallback directory. From then on, +the client can ask any directory cache that's listed in the directory +information it has.) + + +## Hidden Service: + +A hidden service is a server that will only accept incoming +connections via the hidden service protocol. Connection +initiators will not be able to learn the IP address of the hidden +service, allowing the hidden service to receive incoming connections, +serve content, etc, while preserving its location anonymity. + + +## Circuit: + +An established path through the network, where cryptographic keys +are negotiated using the ntor protocol or TAP (Tor Authentication +Protocol (deprecated)) with each hop. Circuits can differ in length +depending on their purpose. See also Leaky Pipe Topology. + +Origin Circuit - + +Exit Circuit: A circuit which connects clients to destinations +outside the Tor network. For example, if a client wanted to visit +duckduckgo.com, this connection would require an exit circuit. + +Internal Circuit: A circuit whose traffic never leaves the Tor +network. For example, a client could connect to a hidden service via +an internal circuit. + + +## Edge connection: + +```text +2.7. Consensus: The state of the Tor network, published every hour, + decided by a vote from the network's directory authorities. Clients + fetch the consensus from directory authorities, fallback + directories, or directory caches. + +2.8. Descriptor: Each descriptor represents information about one + relay in the Tor network. The descriptor includes the relay's IP + address, public keys, and other data. Relays send + descriptors to directory authorities, who vote and publish a + summary of them in the network consensus. +``` + + +## Tor network protocols + + +## Link handshake + +The link handshake establishes the TLS connection over which two +Tor participants will send Tor cells. This handshake also +authenticates the participants to each other, possibly using Tor +cells. + + +## Circuit handshake + +Circuit handshakes establish the hop-by-hop onion encryption +that clients use to tunnel their application traffic. The +client does a pairwise key establishment handshake with each +individual relay in the circuit. For every hop except the +first, these handshakes tunnel through existing hops in the +circuit. Each cell type in this protocol also has a newer +version (with a "2" suffix), e.g., CREATE2. + +CREATE cell: First part of a handshake, sent by the initiator. + +CREATED cell: Second part of a handshake, sent by the responder. + +EXTEND cell: (also known as a RELAY_EXTEND cell) First part of a +handshake, tunneled through an existing circuit. The last relay +in the circuit so far will decrypt this cell and send the +payload in a CREATED cell to the chosen next hop relay. + +EXTENDED cell: (also known as a RELAY_EXTENDED cell) Second part +of a handshake, tunneled through an existing circuit. The last +relay in the circuit so far receives the CREATED cell from the +new last hop relay and encrypts the payload in an EXTENDED cell +to tunnel back to the client. + +Onion skin: A CREATE/CREATE2 or EXTEND/EXTEND2 payload that +contains the first part of the TAP or ntor key establishment +handshake. + + +## Hidden Service Protocol + + +## Directory Protocol + + +## General network definitions + +Leaky Pipe Topology: The ability for the origin of a circuit to address +relay cells to be addressed to any hop in the path of a circuit. In Tor, +the destination hop is determined by using the 'recognized' field of relay +cells. + +Stream: A single application-level connection or request, multiplexed over +a Tor circuit. A 'Stream' can currently carry the contents of a TCP +connection, a DNS request, or a Tor directory request. + +Channel: A pairwise connection between two Tor relays, or between a +client and a relay. Circuits are multiplexed over Channels. All +channels are currently implemented as TLS connections. + diff --git a/spec/guard-spec-intro.md b/spec/guard-spec-intro.md new file mode 100644 index 0000000..bfc9e93 --- /dev/null +++ b/spec/guard-spec-intro.md @@ -0,0 +1,40 @@ +Tor Guard Specification + +```text + Isis Lovecruft + George Kadianakis + Ola Bini + Nick Mathewson + +Table of Contents + + 1. Introduction and motivation + 2. State instances + 3. Circuit Creation, Entry Guard Selection (1000 foot view) + 3.1 Path selection + 3.1.1 Managing entry guards + 3.1.2 Middle and exit node selection + 3.2 Circuit Building + 4. The algorithm. + 4.0. The guards listed in the current consensus. [Section:GUARDS] + 4.1. The Sampled Guard Set. [Section:SAMPLED] + 4.2. The Usable Sample [Section:FILTERED] + 4.3. The confirmed-guard list. [Section:CONFIRMED] + 4.4. The Primary guards [Section:PRIMARY] + 4.5. Retrying guards. [Section:RETRYING] + 4.6. Selecting guards for circuits. [Section:SELECTING] + 4.7. When a circuit fails. [Section:ON_FAIL] + 4.8. When a circuit succeeds [Section:ON_SUCCESS] + 4.9. Updating the list of waiting circuits [Section:UPDATE_WAITING] + 4.10. Whenever we get a new consensus. [Section:ON_CONSENSUS] + 4.11. Deciding whether to generate a new circuit. + 4.12. When we are missing descriptors. + A. Appendices + A.0. Acknowledgements + A.1. Parameters with suggested values. [Section:PARAM_VALS] + A.2. Random values [Section:RANDOM] + A.3. Why not a sliding scale of primaryness? [Section:CVP] + A.4. Controller changes + A.5. Persistent state format +``` + diff --git a/spec/guard-spec/algorithm.md b/spec/guard-spec/algorithm.md new file mode 100644 index 0000000..acdf343 --- /dev/null +++ b/spec/guard-spec/algorithm.md @@ -0,0 +1,594 @@ + +# The algorithm. + + +## The guards listed in the current consensus. [Section:GUARDS] + +By {set:GUARDS} we mean the set of all guards in the current +consensus that are usable for all circuits and directory +requests. (They must have the flags: Stable, Fast, V2Dir, Guard.) + +**Rationale** + +We require all guards to have the flags that we potentially need +from any guard, so that all guards are usable for all circuits. + + +## The Sampled Guard Set. [Section:SAMPLED] + +We maintain a set, {set:SAMPLED_GUARDS}, that persists across +invocations of Tor. It is a subset of the nodes ordered by a sample idx that +we have seen listed as a guard in the consensus at some point. +For each such guard, we record persistently: + +```text + - {pvar:ADDED_ON_DATE}: The date on which it was added to + sampled_guards. + + We set this value to a point in the past, using + RAND(now, {GUARD_LIFETIME}/10). See + Appendix [RANDOM] below. + + - {pvar:ADDED_BY_VERSION}: The version of Tor that added it to + sampled_guards. + + - {pvar:IS_LISTED}: Whether it was listed as a usable Guard in + the _most recent_ consensus we have seen. + + - {pvar:FIRST_UNLISTED_AT}: If IS_LISTED is false, the publication date + of the earliest consensus in which this guard was listed such that we + have not seen it listed in any later consensus. Otherwise "None." + We randomize this to a point in the past, based on + RAND(added_at_time, {REMOVE_UNLISTED_GUARDS_AFTER} / 5) +``` + +For each guard in {SAMPLED_GUARDS}, we also record this data, +non-persistently: + +```text + - {tvar:last_tried_connect}: A 'last tried to connect at' + time. Default 'never'. + + - {tvar:is_reachable}: an "is reachable" tristate, with + possible values { , , }. + Default '.' + + [Note: "yes" is not strictly necessary, but I'm + making it distinct from "maybe" anyway, to make our + logic clearer. A guard is "maybe" reachable if it's + worth trying. A guard is "yes" reachable if we tried + it and succeeded.] + + - {tvar:failing_since}: The first time when we failed to + connect to this guard. Defaults to "never". Reset to + "never" when we successfully connect to this guard. + + - {tvar:is_pending} A "pending" flag. This indicates that we + are trying to build an exploratory circuit through the + guard, and we don't know whether it will succeed. + + - {tvar:pending_since}: A timestamp. Set whenever we set + {tvar:is_pending} to true; cleared whenever we set + {tvar:is_pending} to false. NOTE +``` + +We require that {SAMPLED_GUARDS} contain at least +{MIN_FILTERED_SAMPLE} guards from the consensus (if possible), +but not more than {MAX_SAMPLE_THRESHOLD} of the number of guards +in the consensus, and not more than {MAX_SAMPLE_SIZE} in total. +(But if the maximum would be smaller than {MIN_FILTERED_SAMPLE}, we +set the maximum at {MIN_FILTERED_SAMPLE}.) + +To add a new guard to {SAMPLED_GUARDS}, pick an entry at random from +({GUARDS} - {SAMPLED_GUARDS}), according to the path selection rules. + +We remove an entry from {SAMPLED_GUARDS} if: + +```text + * We have a live consensus, and {IS_LISTED} is false, and + {FIRST_UNLISTED_AT} is over {REMOVE_UNLISTED_GUARDS_AFTER} + days in the past. + + OR + + * We have a live consensus, and {ADDED_ON_DATE} is over + {GUARD_LIFETIME} ago, *and* {CONFIRMED_ON_DATE} is either + "never", or over {GUARD_CONFIRMED_MIN_LIFETIME} ago. +``` + +Note that {SAMPLED_GUARDS} does not depend on our configuration. +It is possible that we can't actually connect to any of these +guards. + +**Rationale** + +The {SAMPLED_GUARDS} set is meant to limit the total number of +guards that a client will connect to in a given period. The +upper limit on its size prevents us from considering too many +guards. + +The first expiration mechanism is there so that our +{SAMPLED_GUARDS} list does not accumulate so many dead +guards that we cannot add new ones. + +The second expiration mechanism makes us rotate our guards slowly +over time. + +Ordering the {SAMPLED_GUARDS} set in the order in which we sampled those +guards and picking guards from that set according to this ordering improves +load-balancing. It is closer to offer the expected usage of the guard nodes +as per the path selection rules. + +The ordering also improves on another objective of this proposal: trying to +resist an adversary pushing clients over compromised guards, since the +adversary would need the clients to exhaust all their initial +{SAMPLED_GUARDS} set before having a chance to use a newly deployed +adversary node. + + +## The Usable Sample [Section:FILTERED] + +We maintain another set, {set:FILTERED_GUARDS}, that does not +persist. It is derived from: + +```text + - {SAMPLED_GUARDS} + - our current configuration, + - the path bias information. +``` + +A guard is a member of {set:FILTERED_GUARDS} if and only if all +of the following are true: + +```text + - It is a member of {SAMPLED_GUARDS}, with {IS_LISTED} set to + true. + - It is not disabled because of path bias issues. + - It is not disabled because of ReachableAddresses policy, + the ClientUseIPv4 setting, the ClientUseIPv6 setting, + the FascistFirewall setting, or some other + option that prevents using some addresses. + - It is not disabled because of ExcludeNodes. + - It is a bridge if UseBridges is true; or it is not a + bridge if UseBridges is false. + - Is included in EntryNodes if EntryNodes is set and + UseBridges is not. (But see 2.B above). +``` + +We have an additional subset, {set:USABLE_FILTERED_GUARDS}, which +is defined to be the subset of {FILTERED_GUARDS} where +{is_reachable} is or . + +We try to maintain a requirement that {USABLE_FILTERED_GUARDS} +contain at least {MIN_FILTERED_SAMPLE} elements: + +```text + Whenever we are going to sample from {USABLE_FILTERED_GUARDS}, + and it contains fewer than {MIN_FILTERED_SAMPLE} elements, we + add new elements to {SAMPLED_GUARDS} until one of the following + is true: + + * {USABLE_FILTERED_GUARDS} is large enough, + OR + * {SAMPLED_GUARDS} is at its maximum size. + + ** Rationale ** +``` + +These filters are applied _after_ sampling: if we applied them +before the sampling, then our sample would reflect the set of +filtering restrictions that we had in the past. + + +## The confirmed-guard list. [Section:CONFIRMED] + +[formerly USED_GUARDS] + +We maintain a persistent ordered list, {list:CONFIRMED_GUARDS}. +It contains guards that we have used before, in our preference +order of using them. It is a subset of {SAMPLED_GUARDS}. For +each guard in this list, we store persistently: + +- {pvar:IDENTITY} Its fingerprint. + +```text + - {pvar:CONFIRMED_ON_DATE} When we added this guard to + {CONFIRMED_GUARDS}. + + Randomized to a point in the past as RAND(now, {GUARD_LIFETIME}/10). +``` + +We append new members to {CONFIRMED_GUARDS} when we mark a circuit +built through a guard as "for user traffic." + +Whenever we remove a member from {SAMPLED_GUARDS}, we also remove +it from {CONFIRMED_GUARDS}. + +```text + [Note: You can also regard the {CONFIRMED_GUARDS} list as a + total ordering defined over a subset of {SAMPLED_GUARDS}.] +``` + +Definition: we call Guard A "higher priority" than another Guard B +if, when A and B are both reachable, we would rather use A. We +define priority as follows: + +```text + * Every guard in {CONFIRMED_GUARDS} has a higher priority + than every guard not in {CONFIRMED_GUARDS}. + + * Among guards in {CONFIRMED_GUARDS}, the one appearing earlier + on the {CONFIRMED_GUARDS} list has a higher priority. + + * Among guards that do not appear in {CONFIRMED_GUARDS}, + {is_pending}==true guards have higher priority. + + * Among those, the guard with earlier {last_tried_connect} time + has higher priority. + + * Finally, among guards that do not appear in + {CONFIRMED_GUARDS} with {is_pending==false}, all have equal + priority. + + ** Rationale ** +``` + +We add elements to this ordering when we have actually used them +for building a usable circuit. We could mark them at some other +time (such as when we attempt to connect to them, or when we +actually connect to them), but this approach keeps us from +committing to a guard before we actually use it for sensitive +traffic. + + +## The Primary guards [Section:PRIMARY] + +We keep a run-time non-persistent ordered list of +{list:PRIMARY_GUARDS}. It is a subset of {FILTERED_GUARDS}. It +contains {N_PRIMARY_GUARDS} elements. + +To compute primary guards, take the ordered intersection of +{CONFIRMED_GUARDS} and {FILTERED_GUARDS}, and take the first +{N_PRIMARY_GUARDS} elements. If there are fewer than +{N_PRIMARY_GUARDS} elements, append additional elements to +PRIMARY_GUARDS chosen from ({FILTERED_GUARDS} - {CONFIRMED_GUARDS}), +ordered in "sample order" (that is, by {ADDED_ON_DATE}). + +Once an element has been added to {PRIMARY_GUARDS}, we do not remove it +until it is replaced by some element from {CONFIRMED_GUARDS}. +That is: if a non-primary guard becomes confirmed and not every primary +guard is confirmed, then the list of primary guards list is regenerated, +first from the confirmed guards (as before), and then from any +non-confirmed primary guards. + +Note that {PRIMARY_GUARDS} do not have to be in +{USABLE_FILTERED_GUARDS}: they might be unreachable. + +** Rationale ** + +These guards are treated differently from other guards. If one of +them is usable, then we use it right away. For other guards +{FILTERED_GUARDS}, if it's usable, then before using it we might +first double-check whether perhaps one of the primary guards is +usable after all. + + +## Retrying guards. [Section:RETRYING] + +(We run this process as frequently as needed. It can be done once +a second, or just-in-time.) + +If a primary sampled guard's {is_reachable} status is , then +we decide whether to update its {is_reachable} status to +based on its {last_tried_connect} time, its {failing_since} time, +and the {PRIMARY_GUARDS_RETRY_SCHED} schedule. + +If a non-primary sampled guard's {is_reachable} status is , then +we decide whether to update its {is_reachable} status to +based on its {last_tried_connect} time, its {failing_since} time, +and the {GUARDS_RETRY_SCHED} schedule. + +** Rationale ** + +An observation that a guard has been 'unreachable' only lasts for +a given amount of time, since we can't infer that it's unreachable +now from the fact that it was unreachable a few minutes ago. + + +## Selecting guards for circuits. [Section:SELECTING] + +Every origin circuit is now in one of these states: + +```text + , + , + , or + . +``` + +You may only attach streams to circuits. +(Additionally, you may only send RENDEZVOUS cells, ESTABLISH_INTRO +cells, and INTRODUCE cells on circuits.) + +The per-circuit state machine is: + +```text + New circuits are or + . + + A circuit may become , or may + fail. + + A circuit may become + ; may become ; or may + fail. + + A circuit will become , or will + be closed, or will fail. + + A circuit remains until it fails or is + closed. + + Each of these transitions is described below. + + We keep, as global transient state: + + * {tvar:last_time_on_internet} -- the last time at which we + successfully used a circuit or connected to a guard. At + startup we set this to "infinitely far in the past." + + When we want to build a circuit, and we need to pick a guard: + + * If any entry in PRIMARY_GUARDS has {is_reachable} status of + or , return one of the first + {NUM_USABLE_PRIMARY_GUARDS} or + {NUM_USABLE_PRIMARY_DIRECTORY_GUARDS} such guards, chosen + uniformly at random. The circuit is . + + [Note: We do not use {is_pending} on primary guards, since we + are willing to try to build multiple circuits through them + before we know for sure whether they work, and since we will + not use any non-primary guards until we are sure that the + primary guards are all down. (XX is this good?)] + + * Otherwise, if the ordered intersection of {CONFIRMED_GUARDS} + and {USABLE_FILTERED_GUARDS} is nonempty, return the first + entry in that intersection that has {is_pending} set to + false. Set its value of {is_pending} to true, + and set its {pending_since} to the current time. + The circuit + is now . (If all entries have + {is_pending} true, pick the first one.) + + * Otherwise, if there is no such entry, select a member from + {USABLE_FILTERED_GUARDS} in sample order. Set its {is_pending} field to + true, and set its {pending_since} to the current time. + The circuit is . + + * Otherwise, if USABLE_FILTERED_GUARDS is empty, we have exhausted + all the sampled guards. In this case we proceed by marking all guards + as reachable so that we can keep on trying circuits. +``` + +Whenever we select a guard for a new circuit attempt, we update the +{last_tried_connect} time for the guard to 'now.' + +In some cases (for example, when we need a certain directory feature, +or when we need to avoid using a certain exit as a guard), we need to +restrict the guards that we use for a single circuit. When this happens, we +remember the restrictions that applied when choosing the guard for +that circuit, since we will need them later (see [UPDATE_WAITING].). + +** Rationale ** + +We're getting to the core of the algorithm here. Our main goals are to +make sure that + +```text + 1. If it's possible to use a primary guard, we do. + 2. We probably use the first primary guard. +``` + +So we only try non-primary guards if we're pretty sure that all +the primary guards are down, and we only try a given primary guard +if the earlier primary guards seem down. + +When we _do_ try non-primary guards, however, we only build one +circuit through each, to give it a chance to succeed or fail. If +ever such a circuit succeeds, we don't use it until we're pretty +sure that it's the best guard we're getting. (see below). + +[XXX timeout.] + + +## When a circuit fails. [Section:ON_FAIL] + +When a circuit fails in a way that makes us conclude that a guard +is not reachable, we take the following steps: + +```text + * Set the guard's {is_reachable} status to . If it had + {is_pending} set to true, we make it non-pending and clear + {pending_since}. + + * Close the circuit, of course. (This removes it from + consideration by the algorithm in [UPDATE_WAITING].) + + * Update the list of waiting circuits. (See [UPDATE_WAITING] + below.) +``` + +[Note: the existing Tor logic will cause us to create more +circuits in response to some of these steps; and also see +[ON_CONSENSUS].] + +** Rationale ** + +See [SELECTING] above for rationale. + + +## When a circuit succeeds [Section:ON_SUCCESS] + +When a circuit succeeds in a way that makes us conclude that a +guard _was_ reachable, we take these steps: + +```text + * We set its {is_reachable} status to . + * We set its {failing_since} to "never". + * If the guard was {is_pending}, we clear the {is_pending} flag + and set {pending_since} to false. + * If the guard was not a member of {CONFIRMED_GUARDS}, we add + it to the end of {CONFIRMED_GUARDS}. + + * If this circuit was , this circuit is + now . You may attach streams to this circuit, + and use it for hidden services. + + * If this circuit was , it is now + . You may not yet attach streams to it. + Then check whether the {last_time_on_internet} is more than + {INTERNET_LIKELY_DOWN_INTERVAL} seconds ago: + + * If it is, then mark all {PRIMARY_GUARDS} as "maybe" + reachable. + + * If it is not, update the list of waiting circuits. (See + [UPDATE_WAITING] below) +``` + +[Note: the existing Tor logic will cause us to create more +circuits in response to some of these steps; and see +[ON_CONSENSUS].] + +** Rationale ** + +See [SELECTING] above for rationale. + + +## Updating the list of waiting circuits [Section:UPDATE_WAITING] + +We run this procedure whenever it's possible that a + circuit might be ready to be called +. + +```text + * If any circuit C1 is , AND: + * All primary guards have reachable status of . + * There is no circuit C2 that "blocks" C1. + Then, upgrade C1 to . + + Definition: In the algorithm above, C2 "blocks" C1 if: + * C2 obeys all the restrictions that C1 had to obey, AND + * C2 has higher priority than C1, AND + * Either C2 is , or C2 is , + or C2 has been for no more than + {NONPRIMARY_GUARD_CONNECT_TIMEOUT} seconds. + + We run this procedure periodically: + + * If any circuit stays in + for more than {NONPRIMARY_GUARD_IDLE_TIMEOUT} seconds, + time it out. + + **Rationale** +``` + +If we open a connection to a guard, we might want to use it +immediately (if we're sure that it's the best we can do), or we +might want to wait a little while to see if some other circuit +which we like better will finish. + +When we mark a circuit , we don't close the +lower-priority circuits immediately: we might decide to use +them after all if the circuit goes down before +{NONPRIMARY_GUARD_IDLE_TIMEOUT} seconds. + + +### Without a list of waiting circuits [Section:NO_CIRCLIST] + +As an alternative to the section [SECTION:UPDATE_WAITING] above, +this section presents a new way to maintain guard status +independently of tracking individual circuit status. This +formulation gives a result equivalent or similar to the approach +above, but simplifies the necessary communications between the +guard and circuit subsystems. + +As before, when all primary guards are Unreachable, we need to +try non-primary guards. We select the first such guard (in +preference order) that is neither Unreachable nor Pending. +Whenever we give out such a guard, if the guard's status is +Unknown, then we call that guard "Pending" with its {is_pending} +flag, until the attempt to use it succeeds or fails. We remember +when the guard became Pending with the {pending_since variable}. + +After completing a circuit, the implementation must check whether +its guard is usable. A guard's usability status may be "usable", +"unusable", or "unknown". A guard is usable according to +these rules: + +1. Primary guards are always usable. + +2. Non-primary guards are usable _for a given circuit_ if every +guard earlier in the preference list is either unsuitable for +that circuit (e.g. because of family restrictions), or marked as +Unreachable, or has been pending for at least +`{NONPRIMARY_GUARD_CONNECT_TIMEOUT}`. + +Non-primary guards are not usable _for a given circuit_ if some +guard earlier in the preference list is suitable for the circuit +_and_ Reachable. + +Non-primary guards are unusable if they have not become +usable after `{NONPRIMARY_GUARD_IDLE_TIMEOUT}` seconds. + +3. If a circuit's guard is not usable or unusable immediately, the +circuit is not discarded; instead, it is kept (but not used) until the +guard becomes usable or unusable. + + +## Whenever we get a new consensus. [Section:ON_CONSENSUS] + +We update {GUARDS}. + +For every guard in {SAMPLED_GUARDS}, we update {IS_LISTED} and +{FIRST_UNLISTED_AT}. + +[**] We remove entries from {SAMPLED_GUARDS} if appropriate, +according to the sampled-guards expiration rules. If they were +in {CONFIRMED_GUARDS}, we also remove them from +{CONFIRMED_GUARDS}. + +We recompute {FILTERED_GUARDS}, and everything that derives from +it, including {USABLE_FILTERED_GUARDS}, and {PRIMARY_GUARDS}. + +(Whenever one of the configuration options that affects the +filter is updated, we repeat the process above, starting at the +[**] line.) + +```text +4.11. Deciding whether to generate a new circuit. + [Section:NEW_CIRCUIT_NEEDED] +``` + +We generate a new circuit when we don't have +enough circuits either built or in-progress to handle a given +stream, or an expected stream. + +For the purpose of this rule, we say that +circuits are neither built nor in-progress; that +circuits are built; and that the other states are in-progress. + +```text +4.12. When we are missing descriptors. + [Section:MISSING_DESCRIPTORS] +``` + +We need either a router descriptor or a microdescriptor in order +to build a circuit through a guard. If we do not have such a +descriptor for a guard, we can still use the guard for one-hop +directory fetches, but not for longer circuits. + +(Also, when we are missing descriptors for our first +{NUM_USABLE_PRIMARY_GUARDS} primary guards, we don't build +circuits at all until we have fetched them.) + diff --git a/spec/guard-spec/appendices.md b/spec/guard-spec/appendices.md new file mode 100644 index 0000000..ea743f7 --- /dev/null +++ b/spec/guard-spec/appendices.md @@ -0,0 +1,211 @@ + +# Appendices + + +## Acknowledgements + +This research was supported in part by NSF grants CNS-1111539, +CNS-1314637, CNS-1526306, CNS-1619454, and CNS-1640548. + + +## Parameters with suggested values. [Section:PARAM_VALS] + +(All suggested values chosen arbitrarily) + +{param:MAX_SAMPLE_THRESHOLD} -- 20% + +{param:MAX_SAMPLE_SIZE} -- 60 + +{param:GUARD_LIFETIME} -- 120 days + +```text + {param:REMOVE_UNLISTED_GUARDS_AFTER} -- 20 days + [previously ENTRY_GUARD_REMOVE_AFTER] + + {param:MIN_FILTERED_SAMPLE} -- 20 + + {param:N_PRIMARY_GUARDS} -- 3 + + {param:PRIMARY_GUARDS_RETRY_SCHED} + + We recommend the following schedule, which is the one + used in Arti: + + -- Use the "decorrelated-jitter" algorithm from "dir-spec.txt" + section 5.5 where `base_delay` is 30 seconds and `cap` + is 6 hours. + + This legacy schedule is the one used in C tor: + + -- every 10 minutes for the first six hours, + -- every 90 minutes for the next 90 hours, + -- every 4 hours for the next 3 days, + -- every 9 hours thereafter. + + {param:GUARDS_RETRY_SCHED} -- + + We recommend the following schedule, which is the one + used in Arti: + + -- Use the "decorrelated-jitter" algorithm from "dir-spec.txt" + section 5.5 where `base_delay` is 10 minutes and `cap` + is 36 hours. + + This legacy schedule is the one used in C tor: + + -- every hour for the first six hours, + -- every 4 hours for the 90 hours, + -- every 18 hours for the next 3 days, + -- every 36 hours thereafter. + + {param:INTERNET_LIKELY_DOWN_INTERVAL} -- 10 minutes + + {param:NONPRIMARY_GUARD_CONNECT_TIMEOUT} -- 15 seconds + + {param:NONPRIMARY_GUARD_IDLE_TIMEOUT} -- 10 minutes + + {param:MEANINGFUL_RESTRICTION_FRAC} -- .2 + + {param:EXTREME_RESTRICTION_FRAC} -- .01 + + {param:GUARD_CONFIRMED_MIN_LIFETIME} -- 60 days + + {param:NUM_USABLE_PRIMARY_GUARDS} -- 1 + + {param:NUM_USABLE_PRIMARY_DIRECTORY_GUARDS} -- 3 +``` + + +## Random values [Section:RANDOM] + +Frequently, we want to randomize the expiration time of something +so that it's not easy for an observer to match it to its start +time. We do this by randomizing its start date a little, so that +we only need to remember a fixed expiration interval. + +By RAND(now, INTERVAL) we mean a time between now and INTERVAL in +the past, chosen uniformly at random. + + +## Why not a sliding scale of primaryness? [Section:CVP] + +At one meeting, I floated the idea of having "primaryness" be a +continuous variable rather than a boolean. + +I'm no longer sure this is a great idea, but I'll try to outline +how it might work. + +To begin with: being "primary" gives it a few different traits: + +1) We retry primary guards more frequently. [Section:RETRYING] + +```text + 2) We don't even _try_ building circuits through + lower-priority guards until we're pretty sure that the + higher-priority primary guards are down. (With non-primary + guards, on the other hand, we launch exploratory circuits + which we plan not to use if higher-priority guards + succeed.) [Section:SELECTING] + + 3) We retry them all one more time if a circuit succeeds after + the net has been down for a while. [Section:ON_SUCCESS] + + We could make each of the above traits continuous: + + 1) We could make the interval at which a guard is retried + depend continuously on its position in CONFIRMED_GUARDS. + + 2) We could change the number of guards we test in parallel + based on their position in CONFIRMED_GUARDS. + + 3) We could change the rule for how long the higher-priority + guards need to have been down before we call a + circuit based on a + possible network-down condition. For example, we could + retry the first guard if we tried it more than 10 seconds + ago, the second if we tried it more than 20 seconds ago, + etc. +``` + +I am pretty sure, however, that if these are worth doing, they +need more analysis! Here's why: + +```text + * They all have the potential to leak more information about a + guard's exact position on the list. Is that safe? Is there + any way to exploit that? I don't think we know. + + * They all seem like changes which it would be relatively + simple to make to the code after we implement the simpler + version of the algorithm described above. +``` + + +## Controller changes + +We will add to control-spec.txt a new possible circuit state, GUARD_WAIT, +that can be given as part of circuit events and GETINFO responses about +circuits. A circuit is in the GUARD_WAIT state when it is fully built, +but we will not use it because a circuit with a better guard might +become built too. + + +## Persistent state format + +The persistent state format doesn't need to be part of this +specification, since different implementations can do it +differently. Nonetheless, here's the one Tor uses: + +The "state" file contains one Guard entry for each sampled guard +in each instance of the guard state (see section 2). The value +of this Guard entry is a set of space-separated K=V entries, +where K contains any nonspace character except =, and V contains +any nonspace characters. + +Implementations must retain any unrecognized K=V entries for a +sampled guard when they regenerate the state file. + +The order of K=V entries is not allowed to matter. + +Recognized fields (values of K) are: + +```text + "in" -- the name of the guard state instance that this + sampled guard is in. If a sampled guard is in two guard + states instances, it appears twice, with a different "in" + field each time. Required. + + "rsa_id" -- the RSA id digest for this guard, encoded in + hex. Required. + + "bridge_addr" -- If the guard is a bridge, its configured address and + port (this can be the ORPort or a pluggable transport port). Optional. + + "nickname" -- the guard's nickname, if any. Optional. + + "sampled_on" -- the date when the guard was sampled. Required. + + "sampled_by" -- the Tor version that sampled this guard. + Optional. + + "unlisted_since" -- the date since which the guard has been + unlisted. Optional. + + "listed" -- 0 if the guard is not listed; 1 if it is. Required. + + "confirmed_on" -- date when the guard was + confirmed. Optional. + + "confirmed_idx" -- position of the guard in the confirmed + list. Optional. + + "pb_use_attempts", "pb_use_successes", "pb_circ_attempts", + "pb_circ_successes", "pb_successful_circuits_closed", + "pb_collapsed_circuits", "pb_unusable_circuits", + "pb_timeouts" -- state for the circuit path bias algorithm, + given in decimal fractions. Optional. +``` + +All dates here are given as a (spaceless) ISO8601 combined date +and time in UTC (e.g., 2016-11-29T19:39:31). + diff --git a/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md b/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md new file mode 100644 index 0000000..13cc73e --- /dev/null +++ b/spec/guard-spec/circuit-creation-entry-guard-selection-1000-foot.md @@ -0,0 +1,72 @@ + +# Circuit Creation, Entry Guard Selection (1000 foot view) + +A circuit in Tor is a path through the network connecting a client to +its destination. At a high-level, a three-hop exit circuit will look +like this: + +Client <-> Entry Guard <-> Middle Node <-> Exit Node <-> Destination + +Entry guards are the only nodes which a client will connect to +directly. Exit relays are the nodes by which traffic exits the +Tor network in order to connect to an external destination. + +3.1 Path selection + +For any multi-hop circuit, at least one entry guard and middle node(s) are +required. An exit node is required if traffic will exit the Tor +network. Depending on its configuration, a relay listed in a +consensus could be used for any of these roles. However, this +specification defines how entry guards specifically should be selected and +managed, as opposed to middle or exit nodes. + +3.1.1 Managing entry guards + +At a high level, a relay listed in a consensus will move through the +following states in the process from initial selection to eventual +usage as an entry guard: + +```text + relays listed in consensus + | + sampled + | | + confirmed filtered + | | | + primary usable_filtered +``` + +Relays listed in the latest consensus can be sampled for guard usage +if they have the "Guard" flag. Sampling is random but weighted by +a measured bandwidth multiplied by bandwidth-weights (Wgg if guard only, +Wgd if guard+exit flagged). + +Once a path is built and a circuit established using this guard, it +is marked as confirmed. Until this point, guards are first sampled +and then filtered based on information such as our current +configuration (see SAMPLED and FILTERED sections) and later marked as +usable_filtered if the guard is not primary but can be reached. + +It is always preferable to use a primary guard when building a new +circuit in order to reduce guard churn; only on failure to connect to +existing primary guards will new guards be used. + +3.1.2 Middle and exit node selection + +Middle nodes are selected at random from relays listed in the latest +consensus, weighted by bandwidth and bandwidth-weights. Exit nodes are +chosen similarly but restricted to relays with a sufficiently permissive +exit policy. + +3.2 Circuit Building + +Once a path is chosen, Tor will use this path to build a new circuit. + +If the circuit is built successfully, Tor will either use it +immediately, or Tor will wait for a circuit with a more preferred +guard if there's a good chance that it will be able to make one. + +If the circuit fails in a way that makes us conclude that a guard +is not reachable, the guard is marked as unreachable, the circuit is +closed, and waiting circuits are updated. + diff --git a/spec/guard-spec/introduction-motivation.md b/spec/guard-spec/introduction-motivation.md new file mode 100644 index 0000000..f552434 --- /dev/null +++ b/spec/guard-spec/introduction-motivation.md @@ -0,0 +1,45 @@ + +# Introduction and motivation + +Tor uses entry guards to prevent an attacker who controls some +fraction of the network from observing a fraction of every user's +traffic. If users chose their entries and exits uniformly at +random from the list of servers every time they build a circuit, +then an adversary who had (k/N) of the network would deanonymize +F=(k/N)^2 of all circuits... and after a given user had built C +circuits, the attacker would see them at least once with +probability 1-(1-F)^C. With large C, the attacker would get a +sample of every user's traffic with probability 1. + +To prevent this from happening, Tor clients choose a small number +of guard nodes (e.g. 3). These guard nodes are the only +nodes that the client will connect to directly. If they are not +compromised, the user's paths are not compromised. + +This specification outlines Tor's guard housekeeping algorithm, +which tries to meet the following goals: + +```text + - Heuristics and algorithms for determining how and which guards + are chosen should be kept as simple and easy to understand as + possible. + + - Clients in censored regions or who are behind a fascist + firewall who connect to the Tor network should not experience + any significant disadvantage in terms of reachability or + usability. + + - Tor should make a best attempt at discovering the most + appropriate behavior, with as little user input and + configuration as possible. + + - Tor clients should discover usable guards without too much + delay. + + - Tor clients should resist (to the extent possible) attacks + that try to force them onto compromised guards. + + - Should maintain the load-balancing offered by the path selection + algorithm +``` + diff --git a/spec/guard-spec/state-instances.md b/spec/guard-spec/state-instances.md new file mode 100644 index 0000000..1f292b2 --- /dev/null +++ b/spec/guard-spec/state-instances.md @@ -0,0 +1,49 @@ + +# State instances + +In the algorithm below, we describe a set of persistent and +non-persistent state variables. These variables should be +treated as an object, of which multiple instances can exist. + +In particular, we specify the use of three particular instances: + +A. UseBridges + +```text + If UseBridges is set, then we replace the {GUARDS} set in + [Sec:GUARDS] below with the list of configured + bridges. We maintain a separate persistent instance of + {SAMPLED_GUARDS} and {CONFIRMED_GUARDS} and other derived + values for the UseBridges case. + + In this case, we impose no upper limit on the sample size. + + B. EntryNodes / ExcludeNodes / Reachable*Addresses / + FascistFirewall / ClientUseIPv4=0 + + If one of the above options is set, and UseBridges is not, + then we compare the fraction of usable guards in the consensus + to the total number of guards in the consensus. + + If this fraction is less than {MEANINGFUL_RESTRICTION_FRAC}, + we use a separate instance of the state. + + (While Tor is running, we do not change back and forth between + the separate instance of the state and the default instance + unless the fraction of usable guards is 5% higher than, or 5% + lower than, {MEANINGFUL_RESTRICTION_FRAC}. This prevents us + from flapping back and forth between instances if we happen to + hit {MEANINGFUL_RESTRICTION_FRAC} exactly. + + If this fraction is less than {EXTREME_RESTRICTION_FRAC}, we use a + separate instance of the state, and warn the user. + + [TODO: should we have a different instance for each set of heavily + restricted options?] + + C. Default + + If neither of the above variant-state instances is used, + we use a default instance. +``` + diff --git a/spec/guard-spec/still-non-addressed-issues-sectiontodo.md b/spec/guard-spec/still-non-addressed-issues-sectiontodo.md new file mode 100644 index 0000000..e565194 --- /dev/null +++ b/spec/guard-spec/still-non-addressed-issues-sectiontodo.md @@ -0,0 +1,29 @@ + +# Still non-addressed issues [Section:TODO] + +Simulate to answer: Will this work in a dystopic world? + +Simulate actual behavior. + +For all lifetimes: instead of storing the "this began at" time, +store the "remove this at" time, slightly randomized. + +Clarify that when you get a circuit, you might need to +relaunch circuits through that same guard immediately, if they +are circuits that have to be independent. + +Fix all items marked XX or TODO. + +"Directory guards" -- do they matter? + +```text + Suggestion: require that all guards support downloads via BEGINDIR. + We don't need to worry about directory guards for relays, since we + aren't trying to prevent relay enumeration. + + IP version preferences via ClientPreferIPv6ORPort + + Suggestion: Treat it as a preference when adding to + {CONFIRMED_GUARDS}, but not otherwise. +``` + diff --git a/spec/padding-spec-intro.md b/spec/padding-spec-intro.md new file mode 100644 index 0000000..ce9e985 --- /dev/null +++ b/spec/padding-spec-intro.md @@ -0,0 +1,40 @@ +Tor Padding Specification + +Mike Perry, George Kadianakis + +Note: This is an attempt to specify Tor as currently implemented. Future +versions of Tor will implement improved algorithms. + +This document tries to cover how Tor chooses to use cover traffic to obscure +various traffic patterns from external and internal observers. Other +implementations MAY take other approaches, but implementors should be aware of +the anonymity and load-balancing implications of their choices. + +```text + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL + NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + RFC 2119. + +Table of Contents + + 1. Overview + 2. Connection-level padding + 2.1. Background + 2.2. Implementation + 2.3. Padding Cell Timeout Distribution Statistics + 2.4. Maximum overhead bounds + 2.5. Reducing or Disabling Padding via Negotiation + 2.6. Consensus Parameters Governing Behavior + 3. Circuit-level padding + 3.1. Circuit Padding Negotiation + 3.2. Circuit Padding Machine Message Management + 3.3. Obfuscating client-side onion service circuit setup + 3.3.1. Common general circuit construction sequences + 3.3.2. Client-side onion service introduction circuit obfuscation + 3.3.3. Client-side rendezvous circuit hiding + 3.3.4. Circuit setup machine overhead + 3.4. Circuit padding consensus parameters + A. Acknowledgments +``` + diff --git a/spec/padding-spec/acknowledgments.md b/spec/padding-spec/acknowledgments.md new file mode 100644 index 0000000..7213127 --- /dev/null +++ b/spec/padding-spec/acknowledgments.md @@ -0,0 +1,28 @@ + +# Acknowledgments + +This research was supported in part by NSF grants CNS-1111539, +CNS-1314637, CNS-1526306, CNS-1619454, and CNS-1640548. + +```text +1. https://en.wikipedia.org/wiki/NetFlow +2. http://infodoc.alcatel-lucent.com/html/0_add-h-f/93-0073-10-01/7750_SR_OS_Router_Configuration_Guide/Cflowd-CLI.html +3. http://www.cisco.com/en/US/docs/ios/12_3t/netflow/command/reference/nfl_a1gt_ps5207_TSD_Products_Command_Reference_Chapter.html#wp1185203 +4. http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/70974-netflow-catalyst6500.html#opconf +5. https://www.juniper.net/techpubs/software/erx/junose60/swconfig-routing-vol1/html/ip-jflow-stats-config4.html#560916 +6. http://www.jnpr.net/techpubs/en_US/junos15.1/topics/reference/configuration-statement/flow-active-timeout-edit-forwarding-options-po.html +7. http://www.jnpr.net/techpubs/en_US/junos15.1/topics/reference/configuration-statement/flow-active-timeout-edit-forwarding-options-po.html +8. http://www.h3c.com/portal/Technical_Support___Documents/Technical_Documents/Switches/H3C_S9500_Series_Switches/Command/Command/H3C_S9500_CM-Release1648%5Bv1.24%5D-System_Volume/200901/624854_1285_0.htm#_Toc217704193 +9. http://docs-legacy.fortinet.com/fgt/handbook/cli52_html/FortiOS%205.2%20CLI/config_system.23.046.html +10. http://wiki.mikrotik.com/wiki/Manual:IP/Traffic_Flow +11. https://metrics.torproject.org/dirbytes.html +12. http://freehaven.net/anonbib/cache/murdoch-pet2007.pdf +13. https://gitweb.torproject.org/torspec.git/tree/proposals/188-bridge-guards.txt +14. http://www.ntop.org/wp-content/uploads/2013/03/nProbe_UserGuide.pdf +15. http://arxiv.org/pdf/1512.00524 +16. https://www.cs.kau.se/pulls/hot/thebasketcase-ape/ +17. https://github.com/torproject/tor/tree/master/doc/HACKING/CircuitPaddingDevelopment.md +18. https://www.usenix.org/node/190967 + https://blog.torproject.org/technical-summary-usenix-fingerprinting-paper +``` + diff --git a/spec/padding-spec/circuit-level-padding.md b/spec/padding-spec/circuit-level-padding.md new file mode 100644 index 0000000..5f821ca --- /dev/null +++ b/spec/padding-spec/circuit-level-padding.md @@ -0,0 +1,281 @@ + +# Circuit-level padding + +The circuit padding system in Tor is an extension of the WTF-PAD +event-driven state machine design[15]. At a high level, this design places +one or more padding state machines at the client, and one or more padding +state machines at a relay, on each circuit. + +State transition and histogram generation has been generalized to be fully +programmable, and probability distribution support was added to support more +compact representations like APE[16]. Additionally, packet count limits, +rate limiting, and circuit application conditions have been added. + +At present, Tor uses this system to deploy two pairs of circuit padding +machines, to obscure differences between the setup phase of client-side +onion service circuits, up to the first 10 cells. + +This specification covers only the resulting behavior of these padding +machines, and thus does not cover the state machine implementation details or +operation. For full details on using the circuit padding system to develop +future padding defenses, see the research developer documentation[17]. + + +## Circuit Padding Negotiation + +Circuit padding machines are advertised as "Padding" subprotocol versions +(see tor-spec.txt Section 9). The onion service circuit padding machines are +advertised as "Padding=2". + +Because circuit padding machines only become active at certain points in +circuit lifetime, and because more than one padding machine may be active at +any given point in circuit lifetime, there is also a padding negotiation +cell and a negotiated response. These are relay commands 41 and 42, with +relay headers as per section 6.1 of tor-spec.txt. + +The fields of the relay cell Data payload of a negotiate request are +as follows: + +```text + const CIRCPAD_COMMAND_STOP = 1; + const CIRCPAD_COMMAND_START = 2; + + const CIRCPAD_RESPONSE_OK = 1; + const CIRCPAD_RESPONSE_ERR = 2; + + const CIRCPAD_MACHINE_CIRC_SETUP = 1; + + struct circpad_negotiate { + u8 version IN [0]; + u8 command IN [CIRCPAD_COMMAND_START, CIRCPAD_COMMAND_STOP]; + + u8 machine_type IN [CIRCPAD_MACHINE_CIRC_SETUP]; + + u8 unused; // Formerly echo_request + + u32 machine_ctr; + }; +``` + +When a client wants to start a circuit padding machine, it first checks that +the desired destination hop advertises the appropriate subprotocol version for +that machine. It then sends a circpad_negotiate cell to that hop with +command=CIRCPAD_COMMAND_START, and machine_type=CIRCPAD_MACHINE_CIRC_SETUP (for +the circ setup machine, the destination hop is the second hop in the +circuit). The machine_ctr is the count of which machine instance this is on +the circuit. It is used to disambiguate shutdown requests. + +When a relay receives a circpad_negotiate cell, it checks that it supports +the requested machine, and sends a circpad_negotiated cell, which is formatted +in the data payload of a relay cell with command number 42 (see tor-spec.txt +section 6.1), as follows: + +```text + struct circpad_negotiated { + u8 version IN [0]; + u8 command IN [CIRCPAD_COMMAND_START, CIRCPAD_COMMAND_STOP]; + u8 response IN [CIRCPAD_RESPONSE_OK, CIRCPAD_RESPONSE_ERR]; + + u8 machine_type IN [CIRCPAD_MACHINE_CIRC_SETUP]; + + u32 machine_ctr; + }; +``` + +If the machine is supported, the response field will contain +CIRCPAD_RESPONSE_OK. If it is not, it will contain CIRCPAD_RESPONSE_ERR. + +Either side may send a CIRCPAD_COMMAND_STOP to shut down the padding machines +(clients MUST only send circpad_negotiate, and relays MUST only send +circpad_negotiated for this purpose). + +If the machine_ctr does not match the current machine instance count +on the circuit, the command is ignored. + + +## Circuit Padding Machine Message Management + +Clients MAY send padding cells towards the relay before receiving the +circpad_negotiated response, to allow for outbound cover traffic before +negotiation completes. + +Clients MAY send another circpad_negotiate cell before receiving the +circpad_negotiated response, to allow for rapid machine changes. + +Relays MUST NOT send padding cells or circpad_negotiated cells, unless a +padding machine is active. Any padding-related cells that arrive at the client +from unexpected relay sources are protocol violations, and clients MAY +immediately tear down such circuits to avoid side channel risk. + + +## Obfuscating client-side onion service circuit setup + +The circuit padding currently deployed in Tor attempts to hide client-side +onion service circuit setup. Service-side setup is not covered, because doing +so would involve significantly more overhead, and/or require interaction with +the application layer. + +The approach taken aims to make client-side introduction and rendezvous +circuits match the cell direction sequence and cell count of 3 hop general +circuits used for normal web traffic, for the first 10 cells only. The +lifespan of introduction circuits is also made to match the lifespan +of general circuits. + +Note that inter-arrival timing is not obfuscated by this defense. + + +### Common general circuit construction sequences + +Most general Tor circuits used to surf the web or download directory +information start with the following 6-cell relay cell sequence (cells +surrounded in [brackets] are outgoing, the others are incoming): + +[EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 -> [BEGIN] -> CONNECTED + +When this is done, the client has established a 3-hop circuit and also opened +a stream to the other end. Usually after this comes a series of DATA cell that +either fetches pages, establishes an SSL connection or fetches directory +information: + +[DATA] -> [DATA] -> DATA -> DATA...(inbound cells continue) + +The above stream of 10 relay cells defines the grand majority of general +circuits that come out of Tor browser during our testing, and it's what we use +to make introduction and rendezvous circuits blend in. + +Please note that in this section we only investigate relay cells and not +connection-level cells like CREATE/CREATED or AUTHENTICATE/etc. that are used +during the link-layer handshake. The rationale is that connection-level cells +depend on the type of guard used and are not an effective fingerprint for a +network/guard-level adversary. + + +### Client-side onion service introduction circuit obfuscation + +Two circuit padding machines work to hide client-side introduction circuits: +one machine at the origin, and one machine at the second hop of the circuit. +Each machine sends padding towards the other. The padding from the origin-side +machine terminates at the second hop and does not get forwarded to the actual +introduction point. + +From Section 3.3.1 above, most general circuits have the following initial +relay cell sequence (outgoing cells marked in [brackets]): + +```text + [EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 -> [BEGIN] -> CONNECTED + -> [DATA] -> [DATA] -> DATA -> DATA...(inbound data cells continue) + + Whereas normal introduction circuits usually look like: + + [EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 + -> [INTRO1] -> INTRODUCE_ACK +``` + +This means that up to the sixth cell (first line of each sequence above), +both general and intro circuits have identical cell sequences. After that +we want to mimic the second line sequence of + +-> [DATA] -> [DATA] -> DATA -> DATA...(inbound data cells continue) + +We achieve this by starting padding INTRODUCE1 has been sent. With padding +negotiation cells, in the common case of the second line looks like: + +-> [INTRO1] -> [PADDING_NEGOTIATE] -> PADDING_NEGOTIATED -> INTRO_ACK + +Then, the middle node will send between INTRO_MACHINE_MINIMUM_PADDING (7) and +INTRO_MACHINE_MAXIMUM_PADDING (10) cells, to match the "...(inbound data cells +continue)" portion of the trace (aka the rest of an HTTPS response body). + +We also set a special flag which keeps the circuit open even after the +introduction is performed. With this feature the circuit will stay alive for +the same duration as normal web circuits before they expire (usually 10 +minutes). + + +### Client-side rendezvous circuit hiding + +Following a similar argument as for intro circuits, we are aiming for padded +rendezvous circuits to blend in with the initial cell sequence of general +circuits which usually look like this: + +```text + [EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 -> [BEGIN] -> CONNECTED + -> [DATA] -> [DATA] -> DATA -> DATA...(incoming cells continue) + + Whereas normal rendezvous circuits usually look like: + + [EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 -> [EST_REND] -> REND_EST + -> REND2 -> [BEGIN] +``` + +This means that up to the sixth cell (the first line), both general and +rend circuits have identical cell sequences. + +After that we want to mimic a [DATA] -> [DATA] -> DATA -> DATA sequence. + +With padding negotiation right after the REND_ESTABLISHED, the sequence +becomes: + +```text + [EXTEND2] -> EXTENDED2 -> [EXTEND2] -> EXTENDED2 -> [EST_REND] -> REND_EST + -> [PADDING_NEGOTIATE] -> [DROP] -> PADDING_NEGOTIATED -> DROP... + + After which normal application DATA cells continue on the circuit. +``` + +Hence this way we make rendezvous circuits look like general circuits up +till the end of the circuit setup. + +After that our machine gets deactivated, and we let the actual rendezvous +circuit shape the traffic flow. Since rendezvous circuits usually imitate +general circuits (their purpose is to surf the web), we can expect that they +will look alike. + + +### Circuit setup machine overhead + +For the intro circuit case, we see that the origin-side machine just sends a +single [PADDING_NEGOTIATE] cell, whereas the origin-side machine sends a +PADDING_NEGOTIATED cell and between 7 to 10 DROP cells. This means that the +average overhead of this machine is 11 padding cells per introduction circuit. + +For the rend circuit case, this machine is quite light. Both sides send 2 +padding cells, for a total of 4 padding cells. + + +## Circuit padding consensus parameters + +The circuit padding system has a handful of consensus parameters that can +either disable circuit padding entirely, or rate limit the total overhead +at relays and clients. + +```text + * circpad_padding_disabled + - If set to 1, no circuit padding machines will negotiate, and all + current padding machines will cease padding immediately. + - Default: 0 + + * circpad_padding_reduced + - If set to 1, only circuit padding machines marked as "reduced"/"low + overhead" will be used. (Currently no such machines are marked + as "reduced overhead"). + - Default: 0 + + * circpad_global_allowed_cells + - This is the number of padding cells that must be sent before + the 'circpad_global_max_padding_percent' parameter is applied. + - Default: 0 + + * circpad_global_max_padding_percent + - This is the maximum ratio of padding cells to total cells, specified + as a percent. If the global ratio of padding cells to total cells + across all circuits exceeds this percent value, no more padding is sent + until the ratio becomes lower. 0 means no limit. + - Default: 0 + + * circpad_max_circ_queued_cells + - This is the maximum number of cells that can be in the circuitmux queue + before padding stops being sent on that circuit. + - Default: CIRCWINDOW_START_MAX (1000) +``` + diff --git a/spec/padding-spec/connection-level-padding.md b/spec/padding-spec/connection-level-padding.md new file mode 100644 index 0000000..3bb7f12 --- /dev/null +++ b/spec/padding-spec/connection-level-padding.md @@ -0,0 +1,283 @@ + +# Connection-level padding + + +## Background + +Tor clients and relays make use of CELL_PADDING to reduce the resolution of +connection-level metadata retention by ISPs and surveillance infrastructure. + +Such metadata retention is implemented by Internet routers in the form of +Netflow, jFlow, Netstream, or IPFIX records. These records are emitted by +gateway routers in a raw form and then exported (often over plaintext) to a +"collector" that either records them verbatim, or reduces their granularity +further[1]. + +Netflow records and the associated data collection and retention tools are +very configurable, and have many modes of operation, especially when +configured to handle high throughput. However, at ISP scale, per-flow records +are very likely to be employed, since they are the default, and also provide +very high resolution in terms of endpoint activity, second only to full packet +and/or header capture. + +Per-flow records record the endpoint connection 5-tuple, as well as the +total number of bytes sent and received by that 5-tuple during a particular +time period. They can store additional fields as well, but it is primarily +timing and bytecount information that concern us. + +When configured to provide per-flow data, routers emit these raw flow +records periodically for all active connections passing through them +based on two parameters: the "active flow timeout" and the "inactive +flow timeout". + +The "active flow timeout" causes the router to emit a new record +periodically for every active TCP session that continuously sends data. The +default active flow timeout for most routers is 30 minutes, meaning that a +new record is created for every TCP session at least every 30 minutes, no +matter what. This value can be configured from 1 minute to 60 minutes on +major routers. + +The "inactive flow timeout" is used by routers to create a new record if a +TCP session is inactive for some number of seconds. It allows routers to +avoid the need to track a large number of idle connections in memory, and +instead emit a separate record only when there is activity. This value +ranges from 10 seconds to 600 seconds on common routers. It appears as +though no routers support a value lower than 10 seconds. + +For reference, here are default values and ranges (in parenthesis when +known) for common routers, along with citations to their manuals. + +Some routers speak other collection protocols than Netflow, and in the +case of Juniper, use different timeouts for these protocols. Where this +is known to happen, it has been noted. + +```text + Inactive Timeout Active Timeout + Cisco IOS[3] 15s (10-600s) 30min (1-60min) + Cisco Catalyst[4] 5min 32min + Juniper (jFlow)[5] 15s (10-600s) 30min (1-60min) + Juniper (Netflow)[6,7] 60s (10-600s) 30min (1-30min) + H3C (Netstream)[8] 60s (60-600s) 30min (1-60min) + Fortinet[9] 15s 30min + MicroTik[10] 15s 30min + nProbe[14] 30s 120s + Alcatel-Lucent[2] 15s (10-600s) 30min (1-600min) +``` + +The combination of the active and inactive netflow record timeouts allow us +to devise a low-cost padding defense that causes what would otherwise be +split records to "collapse" at the router even before they are exported to +the collector for storage. So long as a connection transmits data before the +"inactive flow timeout" expires, then the router will continue to count the +total bytes on that flow before finally emitting a record at the "active +flow timeout". + +This means that for a minimal amount of padding that prevents the "inactive +flow timeout" from expiring, it is possible to reduce the resolution of raw +per-flow netflow data to the total amount of bytes send and received in a 30 +minute window. This is a vast reduction in resolution for HTTP, IRC, XMPP, +SSH, and other intermittent interactive traffic, especially when all +user traffic in that time period is multiplexed over a single connection +(as it is with Tor). + +Though flow measurement in principle can be bidirectional (counting cells +sent in both directions between a pair of IPs) or unidirectional (counting +only cells sent from one IP to another), we assume for safety that all +measurement is unidirectional, and so traffic must be sent by both parties +in order to prevent record splitting. + + +## Implementation + +Tor clients currently maintain one TLS connection to their Guard node to +carry actual application traffic, and make up to 3 additional connections to +other nodes to retrieve directory information. + +We pad only the client's connection to the Guard node, and not any other +connection. We treat Bridge node connections to the Tor network as client +connections, and pad them, but otherwise not pad between normal relays. + +Both clients and Guards will maintain a timer for all application (ie: +non-directory) TLS connections. Every time a padding packet sent by an +endpoint, that endpoint will sample a timeout value from +the max(X,X) distribution described in Section 2.3. The default +range is from 1.5 seconds to 9.5 seconds time range, subject to consensus +parameters as specified in Section 2.6. + +(The timing is randomized to avoid making it obvious which cells are +padding.) + +If another cell is sent for any reason before this timer expires, the timer +is reset to a new random value. + +If the connection remains inactive until the timer expires, a +single CELL_PADDING cell will be sent on that connection (which will +also start a new timer). + +In this way, the connection will only be padded in a given direction in +the event that it is idle in that direction, and will always transmit a +packet before the minimum 10 second inactive timeout. + +(In practice, an implementation may not be able to determine when, +exactly, a cell is sent on a given channel. For example, even though the +cell has been given to the kernel via a call to `send(2)`, the kernel may +still be buffering that cell. In cases such as these, implementations +should use a reasonable proxy for the time at which a cell is sent: for +example, when the cell is queued. If this strategy is used, +implementations should try to observe the innermost (closest to the wire) +queue that they practically can, and if this queue is already nonempty, +padding should not be scheduled until after the queue does become empty.) + + +## Padding Cell Timeout Distribution Statistics + +To limit the amount of padding sent, instead of sampling each endpoint +timeout uniformly, we instead sample it from max(X,X), where X is +uniformly distributed. + +If X is a random variable uniform from 0..R-1 (where R=high-low), then the +random variable Y = max(X,X) has Prob(Y == i) = (2.0*i + 1)/(R*R). + +Then, when both sides apply timeouts sampled from Y, the resulting +bidirectional padding packet rate is now a third random variable: +Z = min(Y,Y). + +The distribution of Z is slightly bell-shaped, but mostly flat around the +mean. It also turns out that Exp[Z] ~= Exp[X]. Here's a table of average +values for each random variable: + +```text + R Exp[X] Exp[Z] Exp[min(X,X)] Exp[Y=max(X,X)] + 2000 999.5 1066 666.2 1332.8 + 3000 1499.5 1599.5 999.5 1999.5 + 5000 2499.5 2666 1666.2 3332.8 + 6000 2999.5 3199.5 1999.5 3999.5 + 7000 3499.5 3732.8 2332.8 4666.2 + 8000 3999.5 4266.2 2666.2 5332.8 + 10000 4999.5 5328 3332.8 6666.2 + 15000 7499.5 7995 4999.5 9999.5 + 20000 9900.5 10661 6666.2 13332.8 +``` + + +## Maximum overhead bounds + +With the default parameters and the above distribution, we expect a +padded connection to send one padding cell every 5.5 seconds. This +averages to 103 bytes per second full duplex (~52 bytes/sec in each +direction), assuming a 512 byte cell and 55 bytes of TLS+TCP+IP headers. +For a client connection that remains otherwise idle for its expected +~50 minute lifespan (governed by the circuit available timeout plus a +small additional connection timeout), this is about 154.5KB of overhead +in each direction (309KB total). + +With 2.5M completely idle clients connected simultaneously, 52 bytes per +second amounts to 130MB/second in each direction network-wide, which is +roughly the current amount of Tor directory traffic[11]. Of course, our +2.5M daily users will neither be connected simultaneously, nor entirely +idle, so we expect the actual overhead to be much lower than this. + + +## Reducing or Disabling Padding via Negotiation + +To allow mobile clients to either disable or reduce their padding overhead, +the CELL_PADDING_NEGOTIATE cell (tor-spec.txt section 7.2) may be sent from +clients to relays. This cell is used to instruct relays to cease sending +padding. + +If the client has opted to use reduced padding, it continues to send +padding cells sampled from the range [9000,14000] milliseconds (subject to +consensus parameter alteration as per Section 2.6), still using the +Y=max(X,X) distribution. Since the padding is now unidirectional, the +expected frequency of padding cells is now governed by the Y distribution +above as opposed to Z. For a range of 5000ms, we can see that we expect to +send a padding packet every 9000+3332.8 = 12332.8ms. We also half the +circuit available timeout from ~50min down to ~25min, which causes the +client's OR connections to be closed shortly there after when it is idle, +thus reducing overhead. + +These two changes cause the padding overhead to go from 309KB per one-time-use +Tor connection down to 69KB per one-time-use Tor connection. For continual +usage, the maximum overhead goes from 103 bytes/sec down to 46 bytes/sec. + +If a client opts to completely disable padding, it sends a +CELL_PADDING_NEGOTIATE to instruct the relay not to pad, and then does not +send any further padding itself. + +Currently, clients negotiate padding only when a channel is created, +immediately after sending their NETINFO cell. Recipients SHOULD, however, +accept padding negotiation messages at any time. + +If a client which previously negotiated reduced, or disabled, padding, and +wishes to re-enable default padding (ie padding according to the consensus +parameters), it SHOULD send CELL_PADDING_NEGOTIATE START with zero in the +ito_low_ms and ito_high_ms fields. (It therefore SHOULD NOT copy the values +from its own established consensus into the CELL_PADDING_NEGOTIATE cell.) +This avoids the client needing to send updated padding negotiations if the +consensus parameters should change. The recipient's clamping of the timing +parameters will cause the recipient to use its notion of the consensus +parameters. + +Clients and bridges MUST reject padding negotiation messages from relays, +and close the channel if they receive one. + + +## Consensus Parameters Governing Behavior + +Connection-level padding is controlled by the following consensus parameters: + +```text + * nf_ito_low + - The low end of the range to send padding when inactive, in ms. + - Default: 1500 + + * nf_ito_high + - The high end of the range to send padding, in ms. + - Default: 9500 + - If nf_ito_low == nf_ito_high == 0, padding will be disabled. + + * nf_ito_low_reduced + - For reduced padding clients: the low end of the range to send padding + when inactive, in ms. + - Default: 9000 + + * nf_ito_high_reduced + - For reduced padding clients: the high end of the range to send padding, + in ms. + - Default: 14000 + + * nf_conntimeout_clients + - The number of seconds to keep never-used circuits opened and + available for clients to use. Note that the actual client timeout is + randomized uniformly from this value to twice this value. + - The number of seconds to keep idle (not currently used) canonical + channels are open and available. (We do this to ensure a sufficient + time duration of padding, which is the ultimate goal.) + - This value is also used to determine how long, after a port has been + used, we should attempt to keep building predicted circuits for that + port. (See path-spec.txt section 2.1.1.) This behavior was + originally added to work around implementation limitations, but it + serves as a reasonable default regardless of implementation. + - For all use cases, reduced padding clients use half the consensus + value. + - Implementations MAY mark circuits held open past the reduced padding + quantity (half the consensus value) as "not to be used for streams", + to prevent their use from becoming a distinguisher. + - Default: 1800 + + * nf_pad_before_usage + - If set to 1, OR connections are padded before the client uses them + for any application traffic. If 0, OR connections are not padded + until application data begins. + - Default: 1 + + * nf_pad_relays + - If set to 1, we also pad inactive relay-to-relay connections + - Default: 0 + + * nf_conntimeout_relays + - The number of seconds that idle relay-to-relay connections are kept + open. + - Default: 3600 +``` + diff --git a/spec/padding-spec/overview.md b/spec/padding-spec/overview.md new file mode 100644 index 0000000..76c8865 --- /dev/null +++ b/spec/padding-spec/overview.md @@ -0,0 +1,28 @@ + +# Overview + +Tor supports two classes of cover traffic: connection-level padding, and +circuit-level padding. + +Connection-level padding uses the CELL_PADDING cell command for cover +traffic, where as circuit-level padding uses the RELAY_COMMAND_DROP relay +command. CELL_PADDING is single-hop only and can be differentiated from +normal traffic by Tor relays ("internal" observers), but not by entities +monitoring Tor OR connections ("external" observers). + +RELAY_COMMAND_DROP is multi-hop, and is not visible to intermediate Tor +relays, because the relay command field is covered by circuit layer +encryption. Moreover, Tor's 'recognized' field allows RELAY_COMMAND_DROP +padding to be sent to any intermediate node in a circuit (as per Section +6.1 of tor-spec.txt). + +Tor uses both connection level and circuit level padding. Connection +level padding is described in section 2. Circuit level padding is +described in section 3. + +The circuit-level padding system is completely orthogonal to the +connection-level padding. The connection-level padding system regards +circuit-level padding as normal data traffic, and hence the connection-level +padding system will not add any additional overhead while the circuit-level +padding system is actively padding. + diff --git a/spec/param-spec.md b/spec/param-spec.md new file mode 100644 index 0000000..3277609 --- /dev/null +++ b/spec/param-spec.md @@ -0,0 +1,535 @@ +Tor network parameters + +This file lists the recognized parameters that can appear on the "params" +line of a directory consensus. + +Table of Contents + +1. Network protocol parameters +2. Performance-tuning parameters +3. Voting-related parameters +4. Circuit-build-timeout parameters +5. Directory-related parameters +6. Pathbias parameters +7. Relay behavior +8. V3 onion service parameters +9. Denial-of-service parameters +10. Padding-related parameters +11. Guard-related parameters +X. Obsolete parameters + + +# Network protocol parameters + +"circwindow" -- the default package window that circuits should be +established with. It started out at 1000 cells, but some research +indicates that a lower value would mean fewer cells in transit in the +network at any given time. +Min: 100, Max: 1000, Default: 1000 +First-appeared: Tor 0.2.1.20 + +"UseOptimisticData" -- If set to zero, clients by default shouldn't try +to send optimistic data to servers until they have received a +RELAY_CONNECTED cell. +Min: 0, Max: 1, Default: 1 +First-appeared: 0.2.3.3-alpha +Default was 0 before: 0.2.9.1-alpha +Removed in 0.4.5.1-alpha; now always on. + +"usecreatefast" -- Used to control whether clients use the CREATE_FAST +handshake on the first hop of their circuits. +Min: 0, Max: 1. Default: 1. +First-appeared: 0.2.4.23, 0.2.5.2-alpha +Removed in 0.4.5.1-alpha; now always off. + +"min_paths_for_circs_pct" -- A percentage threshold that determines +whether clients believe they have enough directory information to +build circuits. This value applies to the total fraction of +bandwidth-weighted paths that the client could build; see +path-spec.txt for more information. +Min: 25, Max: 95, Default: 60 +First-appeared: 0.2.4 + +"ExtendByEd25519ID" -- If true, clients should include Ed25519 +identities for relays when generating EXTEND2 cells. +Min: 0. Max: 1. Default: 0. +First-appeared: 0.3.0 + +"sendme_emit_min_version" -- Minimum SENDME version that can be sent. +Min: 0. Max: 255. Default 0. +First appeared: 0.4.1.1-alpha. + +"sendme_accept_min_version" -- Minimum SENDME version that is accepted. +Min: 0. Max: 255. Default 0. +First appeared: 0.4.1.1-alpha. + +"allow-network-reentry" -- If true, the Exit relays allow connections that +are exiting the network to re-enter. If false, any exit connections going +to a relay ORPort or an authority ORPort and DirPort is denied and the +stream is terminated. +Min: 0. Max: 1. Default: 0 +First appeared: 0.4.5.1-alpha. + + +# Performance-tuning parameters + +"CircuitPriorityHalflifeMsec" -- the halflife parameter used when +weighting which circuit will send the next cell. Obeyed by Tor +0.2.2.10-alpha and later. (Versions of Tor between 0.2.2.7-alpha and +0.2.2.10-alpha recognized a "CircPriorityHalflifeMsec" parameter, but +mishandled it badly.) +Min: 1, Max: 2147483647 (INT32_MAX), Default: 30000. +First-appeared: Tor 0.2.2.11-alpha + +```text + "perconnbwrate" and "perconnbwburst" -- if set, each relay sets up a + separate token bucket for every client OR connection, and rate limits + that connection independently. Typically left unset, except when used for + performance experiments around trac entry 1750. Only honored by relays + running Tor 0.2.2.16-alpha and later. (Note that relays running + 0.2.2.7-alpha through 0.2.2.14-alpha looked for bwconnrate and + bwconnburst, but then did the wrong thing with them; see bug 1830 for + details.) + Min: 1, Max: 2147483647 (INT32_MAX), Default: (user setting of + BandwidthRate/BandwidthBurst). + First-appeared: 0.2.2.7-alpha + Removed-in: 0.2.2.16-alpha +``` + +"NumNTorsPerTAP" -- When balancing ntor and TAP cells at relays, +how many ntor handshakes should we perform for each TAP handshake? +Min: 1. Max: 100000. Default: 10. +First-appeared: 0.2.4.17-rc + +"circ_max_cell_queue_size" -- This parameter determines the maximum +number of cells allowed per circuit queue. +Min: 1000. Max: 2147483647 (INT32_MAX). Default: 50000. +First-appeared: 0.3.3.6-rc. + +"KISTSchedRunInterval" -- How frequently should the "KIST" scheduler +run in order to decide which data to write to the network? Value in +units of milliseconds. +Min: 2. Max: 100. Default: 2 +First appeared: 0.3.2 + +"KISTSchedRunIntervalClient" -- How frequently should the "KIST" scheduler +run in order to decide which data to write to the network, on clients? Value +in units of milliseconds. The client value needs to be much lower than +the relay value. +Min: 2. Max: 100. Default: 2. +First appeared: 0.4.8.2 + + +# Voting-related parameters + +"bwweightscale" -- Value that bandwidth-weights are divided by. If not +present then this defaults to 10000. +Min: 1 +First-appeared: 0.2.2.10-alpha + +"maxunmeasuredbw" -- Used by authorities during voting with method 17 or +later. The maximum value to give for any Bandwidth= entry for a router +that isn't based on at least three measurements. + +(Note: starting in version 0.4.6.1-alpha +there was a bug where Tor authorities would instead look at +a parameter called "maxunmeasurdbw", without the "e". +This bug was fixed in 0.4.9.1-alpha and in 0.4.8.8. +Until all relays are running a fixed version, then either this parameter +must not be set, or it must be set to the same value for both +spellings.) + +First-appeared: 0.2.4.11-alpha + +"FastFlagMinThreshold", "FastFlagMaxThreshold" -- lowest and highest +allowable values for the cutoff for routers that should get the Fast +flag. This is used during voting to prevent the threshold for getting +the Fast flag from being too low or too high. +FastFlagMinThreshold: Min: 4. Max: INT32_MAX: Default: 4. +FastFlagMaxThreshold: Min: -. Max: INT32_MAX: Default: INT32_MAX +First-appeared: 0.2.3.11-alpha + +"AuthDirNumSRVAgreements" -- Minimum number of agreeing directory +authority votes required for a fresh shared random value to be written in +the consensus (this rule only applies on the first commit round of the +shared randomness protocol). +Min: 1. Max: INT32_MAX. Default: 2/3 of the total number of +dirauth. + + +# Circuit-build-timeout parameters + +"cbtdisabled", "cbtnummodes", "cbtrecentcount", "cbtmaxtimeouts", +"cbtmincircs", "cbtquantile", "cbtclosequantile", "cbttestfreq", +"cbtmintimeout", "cbtlearntimeout", "cbtmaxopencircs", and +"cbtinitialtimeout" -- see "2.4.5. Consensus parameters governing +behavior" in path-spec.txt for a series of circuit build time related +consensus parameters. + + +# Directory-related parameters + +"max-consensus-age-to-cache-for-diff" -- Determines how much +consensus history (in hours) relays should try to cache in order to +serve diffs. (min 0, max 8192, default 72) + +"try-diff-for-consensus-newer-than" -- This parameter determines how +old a consensus can be (in hours) before a client should no longer +try to find a diff for it. (min 0, max 8192, default 72) + + +# Pathbias parameters + +"pb_mincircs", "pb_noticepct", "pb_warnpct", "pb_extremepct", +"pb_dropguards", "pb_scalecircs", "pb_scalefactor", +"pb_multfactor", "pb_minuse", "pb_noticeusepct", +"pb_extremeusepct", "pb_scaleuse" -- DOCDOC + + +# Relay behavior + +"refuseunknownexits" -- if set to one, exit relays look at the previous +hop of circuits that ask to open an exit stream, and refuse to exit if +they don't recognize it as a relay. The goal is to make it harder for +people to use them as one-hop proxies. See trac entry 1751 for details. +Min: 0, Max: 1 +First-appeared: 0.2.2.17-alpha + +"onion-key-rotation-days" -- (min 1, max 90, default 28) + +"onion-key-grace-period-days" -- (min 1, max +onion-key-rotation-days, default 7) + +Every relay should list each onion key it generates for +onion-key-rotation-days days after generating it, and then +replace it. Relays should continue to accept their most recent +previous onion key for an additional onion-key-grace-period-days +days after it is replaced. (Introduced in 0.3.1.1-alpha; +prior versions of tor hardcoded both of these values to 7 days.) + +"AllowNonearlyExtend" -- If true, permit EXTEND cells that are not inside +RELAY_EARLY cells. +Min: 0. Max: 1. Default: 0. +First-appeared: 0.2.3.11-alpha + +"overload_dns_timeout_scale_percent" -- This value is a percentage of how +many DNS timeout over N seconds we accept before reporting the overload +general state. It is scaled by a factor of 1000 in order to be able to +represent decimal point. As an example, a value of 1000 means 1%. +Min: 0. Max: 100000. Default: 1000. +First-appeared: 0.4.6.8 +Deprecated: 0.4.7.3-alpha-dev + +"overload_dns_timeout_period_secs" -- This value is the period in seconds +of the DNS timeout measurements (the N in the +"overload_dns_timeout_scale_percent" parameter). For this amount of +seconds, we will gather DNS statistics and at the end, we'll do an +assessment on the overload general signal with regards to DNS timeouts. +Min: 0. Max: 2147483647. Default: 600 +First-appeared: 0.4.6.8 +Deprecated: 0.4.7.3-alpha-dev + +"overload_onionskin_ntor_scale_percent" -- This value is a percentage of +how many onionskin ntor drop over N seconds we accept before reporting the +overload general state. It is scaled by a factor of 1000 in order to be +able to represent decimal point. As an example, a value of 1000 means 1%. +Min: 0. Max: 100000. Default: 1000. +First-appeared: 0.4.7.5-alpha + +"overload_onionskin_ntor_period_secs" -- This value is the period in +seconds of the onionskin ntor overload measurements (the N in the +"overload_onionskin_ntor_scale_percent" parameter). For this amount of +seconds, we will gather onionskin ntor statistics and at the end, we'll do +an assessment on the overload general signal. +Min: 0. Max: 2147483647. Default: 21600 (6 hours) +First-appeared: 0.4.7.5-alpha + +"assume-reachable" -- If true, relays should publish descriptors +even when they cannot make a connection to their IPv4 ORPort. +Min: 0. Max: 1. Default: 0. +First appeared: 0.4.5.1-alpha. + +"assume-reachable-ipv6" -- If true, relays should publish +descriptors even when they cannot make a connection to their IPv6 +ORPort. +Min: 0. Max: 1. Default: 0. +First appeared: 0.4.5.1-alpha. + +"exit_dns_timeout" -- The time in milliseconds an Exit sets libevent to +wait before it considers the DNS timed out. The corresponding libevent +option is "timeout:". +Min: 1. Max: 120000. Default: 1000 (1sec) +First appeared: 0.4.7.5-alpha. + +"exit_dns_num_attempts" -- How many attempts _after the first_ should an +Exit should try a timing-out DNS query before calling it hopeless? (Each of +these attempts will wait for "exit_dns_timeout" independently). The +corresponding libevent option is "attempts:". +Min: 0. Max: 255. Default: 2 +First appeared: 0.4.7.5-alpha. + + +# V3 onion service parameters + +"hs_intro_min_introduce2", "hs_intro_max_introduce2" -- +Minimum/maximum amount of INTRODUCE2 cells allowed per circuits +before rotation (actual amount picked at random between these two +values). +Min: 0. Max: INT32_MAX. Defaults: 16384, 32768. + +"hs_intro_min_lifetime", "hs_intro_max_lifetime" -- Minimum/maximum +lifetime in seconds that a service should keep an intro point for +(actual lifetime picked at random between these two values). +Min: 0. Max: INT32_MAX. Defaults: 18 hours, 24 hours. + +"hs_intro_num_extra" -- Number of extra intro points a service is +allowed to open. This concept comes from proposal #155. +Min: 0. Max: 128. Default: 2. + +"hsdir_interval" -- The length of a time period, _in minutes_. See +rend-spec-v3.txt section [TIME-PERIODS]. +Min: 30. Max: 14400. Default: 1440. + +"hsdir_n_replicas" -- Number of HS descriptor replicas. +Min: 1. Max: 16. Default: 2. + +"hsdir_spread_fetch" -- Total number of HSDirs per replica a tor +client should select to try to fetch a descriptor. +Min: 1. Max: 128. Default: 3. + +"hsdir_spread_store" -- Total number of HSDirs per replica a service +will upload its descriptor to. +Min: 1. Max: 128. Default: 4 + +"HSV3MaxDescriptorSize" -- Maximum descriptor size (in bytes). +Min: 1. Max: INT32_MAX. Default: 50000 + +"hs_service_max_rdv_failures" -- This parameter determines the +maximum number of rendezvous attempt an HS service can make per +introduction. +Min 1. Max 10. Default 2. +First-appeared: 0.3.3.0-alpha. + +"HiddenServiceEnableIntroDoSDefense" -- This parameter makes tor +start using this defense if the introduction point supports it +(for protover HSIntro=5). +Min: 0. Max: 1. Default: 0. +First appeared: 0.4.2.1-alpha. + +"HiddenServiceEnableIntroDoSBurstPerSec" -- Maximum burst to be used +for token bucket for the introduction point rate-limiting. +Min: 0. Max: INT32_MAX. Default: 200 +First appeared: 0.4.2.1-alpha. + +"HiddenServiceEnableIntroDoSRatePerSec" -- Refill rate to be used +for token bucket for the introduction point rate-limiting. +Min: 0. Max: INT32_MAX. Default: 25 +First appeared: 0.4.2.1-alpha. + + +# Denial-of-service parameters + +Denial of Service mitigation parameters. Introduced in 0.3.3.2-alpha: + +"DoSCircuitCreationEnabled" -- Enable the circuit creation DoS +mitigation. + +"DoSCircuitCreationMinConnections" -- Minimum threshold of +concurrent connections before a client address can be flagged as +executing a circuit creation DoS + +"DoSCircuitCreationRate" -- Allowed circuit creation rate per second +per client IP address once the minimum concurrent connection +threshold is reached. + +"DoSCircuitCreationBurst" -- The allowed circuit creation burst per +client IP address once the minimum concurrent connection threshold +is reached. + +```text + "DoSCircuitCreationDefenseType" -- Defense type applied to a + detected client address for the circuit creation mitigation. + 1: No defense. + 2: Refuse circuit creation for the length of + "DoSCircuitCreationDefenseTimePeriod". +``` + +"DoSCircuitCreationDefenseTimePeriod" -- The base time period that +the DoS defense is activated for. + +"DoSConnectionEnabled" -- Enable the connection DoS mitigation. + +"DoSConnectionMaxConcurrentCount" -- The maximum threshold of +concurrent connection from a client IP address. + +```text + "DoSConnectionDefenseType" -- Defense type applied to a detected + client address for the connection mitigation. Possible values are: + 1: No defense. + 2: Immediately close new connections. +``` + +"DoSRefuseSingleHopClientRendezvous" -- Refuse establishment of +rendezvous points for single hop clients. + + +# Padding-related parameters + +"circpad_max_circ_queued_cells" -- The circuitpadding module will +stop sending more padding cells if more than this many cells are in +the circuit queue a given circuit. +Min: 0. Max: 50000. Default 1000. +First appeared: 0.4.0.3-alpha. + +"circpad_global_allowed_cells" -- DOCDOC + +"circpad_global_max_padding_pct" -- DOCDOC + +"circpad_padding_disabled" -- DOCDOC + +"circpad_padding_reduced" -- DOCDOC + +"nf_conntimeout_clients" -- DOCDOC + +"nf_conntimeout_relays" -- DOCDOC + +"nf_ito_high_reduced" -- DOCDOC + +"nf_ito_low" -- DOCDOC + +"nf_ito_low_reduced" -- DOCDOC + +"nf_pad_before_usage" -- DOCDOC + +"nf_pad_relays" -- DOCDOC + +"nf_pad_single_onion" -- DOCDOC + + +# Guard-related parameters + +(See guard-spec.txt for more information on the vocabulary used here.) + +"UseGuardFraction" -- If true, clients use `GuardFraction` +information from the consensus in order to decide how to weight +guards when picking them. +Min: 0. Max: 1. Default: 0. +First appeared: 0.2.6 + +"guard-lifetime-days" -- Controls guard lifetime. If an unconfirmed +guard has been sampled more than this many days ago, it should be +removed from the guard sample. +Min: 1. Max: 3650. Default: 120. +First appeared: 0.3.0 + +"guard-confirmed-min-lifetime-days" -- Controls confirmed guard +lifetime: if a guard was confirmed more than this many days ago, it +should be removed from the guard sample. +Min: 1. Max: 3650. Default: 60. +First appeared: 0.3.0 + +"guard-internet-likely-down-interval" -- If Tor has been unable to +build a circuit for this long (in seconds), assume that the internet +connection is down, and treat guard failures as unproven. +Min: 1. Max: INT32_MAX. Default: 600. +First appeared: 0.3.0 + +"guard-max-sample-size" -- Largest number of guards that clients +should try to collect in their sample. +Min: 1. Max: INT32_MAX. Default: 60. +First appeared: 0.3.0 + +"guard-max-sample-threshold-percent" -- Largest bandwidth-weighted +fraction of guards that clients should try to collect in their +sample. +Min: 1. Max: 100. Default: 20. +First appeared: 0.3.0 + +"guard-meaningful-restriction-percent" -- If the client has +configured tor to exclude so many guards that the available guard +bandwidth is less than this percentage of the total, treat the guard +sample as "restricted", and keep it in a separate sample. +Min: 1. Max: 100. Default: 20. +First appeared: 0.3.0 + +"guard-extreme-restriction-percent" -- Warn the user if they have +configured tor to exclude so many guards that the available guard +bandwidth is less than this percentage of the total. +Min: 1. Max: 100. Default: 1. +First appeared: 0.3.0. MAX was INT32_MAX, which would have no meaningful +effect. MAX lowered to 100 in 0.4.7. + +"guard-min-filtered-sample-size" -- If fewer than this number of +guards is available in the sample after filtering out unusable +guards, the client should try to add more guards to the sample (if +allowed). +Min: 1. Max: INT32_MAX. Default: 20. +First appeared: 0.3.0 + +"guard-n-primary-guards" -- The number of confirmed guards that the +client should treat as "primary guards". +Min: 1. Max: INT32_MAX. Default: 3. +First appeared: 0.3.0 + +```text + "guard-n-primary-guards-to-use", "guard-n-primary-dir-guards-to-use" + -- number of primary guards and primary directory guards that the + client should be willing to use in parallel. Other primary guards + won't get used unless the earlier ones are down. + "guard-n-primary-guards-to-use": + Min 1, Max INT32_MAX: Default: 1. + "guard-n-primary-dir-guards-to-use" + Min 1, Max INT32_MAX: Default: 3. + First appeared: 0.3.0 +``` + +"guard-nonprimary-guard-connect-timeout" -- When trying to confirm +nonprimary guards, if a guard doesn't answer for more than this long +in seconds, treat lower-priority guards as usable. +Min: 1. Max: INT32_MAX. Default: 15 +First appeared: 0.3.0 + +"guard-nonprimary-guard-idle-timeout" -- When trying to confirm +nonprimary guards, if a guard doesn't answer for more than this long +in seconds, treat it as down. +Min: 1. Max: INT32_MAX. Default: 600 +First appeared: 0.3.0 + +"guard-remove-unlisted-guards-after-days" -- If a guard has been +unlisted in the consensus for at least this many days, remove it +from the sample. +Min: 1. Max: 3650. Default: 20. +First appeared: 0.3.0 + + +# Obsolete parameters + +"NumDirectoryGuards", "NumEntryGuards" -- Number of guard nodes +clients should use by default. If NumDirectoryGuards is 0, we +default to NumEntryGuards. +NumDirectoryGuards: Min: 0. Max: 10. Default: 0 +NumEntryGuards: Min: 1. Max: 10. Default: 3 +First-appeared: 0.2.4.23, 0.2.5.6-alpha +Removed in: 0.3.0 + +"GuardLifetime" -- Duration for which clients should choose guard +nodes, in seconds. +Min: 30 days. Max: 1826 days. Default: 60 days. +First-appeared: 0.2.4.12-alpha +Removed in: 0.3.0. + +"UseNTorHandshake" -- If true, then versions of Tor that support +NTor will prefer to use it by default. +Min: 0, Max: 1. Default: 1. +First-appeared: 0.2.4.8-alpha +Removed in: 0.2.9. + +"Support022HiddenServices" -- Used to implement a mass switch-over +from sending timestamps to hidden services by default to sending no +timestamps at all. If this option is absent, or is set to 1, +clients with the default configuration send timestamps; otherwise, +they do not. +Min: 0, Max: 1. Default: 1. +First-appeared: 0.2.4.18-rc +Removed in: 0.2.6 + diff --git a/spec/path-spec-intro.md b/spec/path-spec-intro.md new file mode 100644 index 0000000..218aa9b --- /dev/null +++ b/spec/path-spec-intro.md @@ -0,0 +1,66 @@ +Tor Path Specification + +```text + Roger Dingledine + Nick Mathewson +``` + +Note: This is an attempt to specify Tor as currently implemented. Future +versions of Tor will implement improved algorithms. + +This document tries to cover how Tor chooses to build circuits and assign +streams to circuits. Other implementations MAY take other approaches, but +implementors should be aware of the anonymity and load-balancing implications +of their choices. + +THIS SPEC ISN'T DONE YET. + +```text + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL + NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + RFC 2119. + +Tables of Contents + + 1. General operation + 1.1. Terminology + 1.2. A relay's bandwidth + 2. Building circuits + 2.1. When we build + 2.1.0. We don't build circuits until we have enough directory info + 2.1.1. Clients build circuits preemptively + 2.1.2. Clients build circuits on demand + 2.1.3. Relays build circuits for testing reachability and bandwidth + 2.1.4. Hidden-service circuits + 2.1.5. Rate limiting of failed circuits + 2.1.6. When to tear down circuits + 2.2. Path selection and constraints + 2.2.1. Choosing an exit + 2.2.2. User configuration + 2.3. Cannibalizing circuits + 2.4. Learning when to give up ("timeout") on circuit construction + 2.4.1 Distribution choice and parameter estimation + 2.4.2. How much data to record + 2.4.3. How to record timeouts + 2.4.4. Detecting Changing Network Conditions + 2.4.5. Consensus parameters governing behavior + 2.4.6. Consensus parameters governing behavior + 2.5. Handling failure + 3. Attaching streams to circuits + 4. Hidden-service related circuits + 5. Guard nodes + 5.1. How consensus bandwidth weights factor into entry guard selection + 6. Server descriptor purposes + 7. Detecting route manipulation by Guard nodes (Path Bias) + 7.1. Measuring path construction success rates + 7.2. Measuring path usage success rates + 7.3. Scaling success counts + 7.4. Parametrization + 7.5. Known barriers to enforcement + X. Old notes + X.1. Do we actually do this? + X.2. A thing we could do to deal with reachability. + X.3. Some stuff that worries me about entry guards. 2006 Jun, Nickm. +``` + diff --git a/spec/path-spec/attaching-streams-to-circuits.md b/spec/path-spec/attaching-streams-to-circuits.md new file mode 100644 index 0000000..a603b98 --- /dev/null +++ b/spec/path-spec/attaching-streams-to-circuits.md @@ -0,0 +1,19 @@ + +# Attaching streams to circuits + +When a circuit that might support a request is built, Tor tries to attach +the request's stream to the circuit and sends a BEGIN, BEGIN_DIR, +or RESOLVE relay +cell as appropriate. If the request completes unsuccessfully, Tor +considers the reason given in the CLOSE relay cell. [XXX yes, and?] + +After a request has remained unattached for SocksTimeout (2 minutes +by default), Tor abandons the attempt and signals an error to the +client as appropriate (e.g., by closing the SOCKS connection). + +XXX Timeouts and when Tor auto-retries. + +* What stream-end-reasons are appropriate for retrying. + +If no reply to BEGIN/RESOLVE, then the stream will timeout and fail. + diff --git a/spec/path-spec/building-circuits.md b/spec/path-spec/building-circuits.md new file mode 100644 index 0000000..2382a88 --- /dev/null +++ b/spec/path-spec/building-circuits.md @@ -0,0 +1,3 @@ + +# Building circuits + diff --git a/spec/path-spec/cannibalizing-circuits.md b/spec/path-spec/cannibalizing-circuits.md new file mode 100644 index 0000000..8d07489 --- /dev/null +++ b/spec/path-spec/cannibalizing-circuits.md @@ -0,0 +1,15 @@ + +## Cannibalizing circuits + +If we need a circuit and have a clean one already established, in +some cases we can adapt the clean circuit for our new +purpose. Specifically, + +For hidden service interactions, we can "cannibalize" a clean internal +circuit if one is available, so we don't need to build those circuits +from scratch on demand. + +We can also cannibalize clean circuits when the client asks to exit +at a given node -- either via the ".exit" notation or because the +destination is running at the same location as an exit node. + diff --git a/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md b/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md new file mode 100644 index 0000000..1191efc --- /dev/null +++ b/spec/path-spec/detecting-route-manipulation-by-guard-nodes-path.md @@ -0,0 +1,193 @@ + +# Detecting route manipulation by Guard nodes (Path Bias) + +The Path Bias defense is designed to defend against a type of route +capture where malicious Guard nodes deliberately fail or choke circuits +that extend to non-colluding Exit nodes to maximize their network +utilization in favor of carrying only compromised traffic. + +In the extreme, the attack allows an adversary that carries c/n +of the network capacity to deanonymize c/n of the network +connections, breaking the O((c/n)^2) property of Tor's original +threat model. It also allows targeted attacks aimed at monitoring +the activity of specific users, bridges, or Guard nodes. + +There are two points where path selection can be manipulated: +during construction, and during usage. Circuit construction +can be manipulated by inducing circuit failures during circuit +extend steps, which causes the Tor client to transparently retry +the circuit construction with a new path. Circuit usage can be +manipulated by abusing the stream retry features of Tor (for +example by withholding stream attempt responses from the client +until the stream timeout has expired), at which point the tor client +will also transparently retry the stream on a new path. + +The defense as deployed therefore makes two independent sets of +measurements of successful path use: one during circuit construction, +and one during circuit usage. + +The intended behavior is for clients to ultimately disable the use +of Guards responsible for excessive circuit failure of either type +(see section 7.4); however known issues with the Tor network currently +restrict the defense to being informational only at this stage (see +section 7.5). + + +## Measuring path construction success rates + +Clients maintain two counts for each of their guards: a count of the +number of times a circuit was extended to at least two hops through that +guard, and a count of the number of circuits that successfully complete +through that guard. The ratio of these two numbers is used to determine +a circuit success rate for that Guard. + +Circuit build timeouts are counted as construction failures if the +circuit fails to complete before the 95% "right-censored" timeout +interval, not the 80% timeout condition (see section 2.4). + +If a circuit closes prematurely after construction but before being +requested to close by the client, this is counted as a failure. + + +## Measuring path usage success rates + +Clients maintain two usage counts for each of their guards: a count +of the number of usage attempts, and a count of the number of +successful usages. + +A usage attempt means any attempt to attach a stream to a circuit. + +Usage success status is temporarily recorded by state flags on circuits. +Guard usage success counts are not incremented until circuit close. A +circuit is marked as successfully used if we receive a properly +recognized RELAY cell on that circuit that was expected for the current +circuit purpose. + +If subsequent stream attachments fail or time out, the successfully used +state of the circuit is cleared, causing it once again to be regarded +as a usage attempt only. + +Upon close by the client, all circuits that are still marked as usage +attempts are probed using a RELAY_BEGIN cell constructed with a +destination of the form 0.a.b.c:25, where a.b.c is a 24 bit random +nonce. If we get a RELAY_COMMAND_END in response matching our nonce, +the circuit is counted as successfully used. + +If any unrecognized RELAY cells arrive after the probe has been sent, +the circuit is counted as a usage failure. + +If the stream failure reason codes DESTROY, TORPROTOCOL, or INTERNAL +are received in response to any stream attempt, such circuits are not +probed and are declared usage failures. + +Prematurely closed circuits are not probed, and are counted as usage +failures. + + +## Scaling success counts + +To provide a moving average of recent Guard activity while +still preserving the ability to verify correctness, we periodically +"scale" the success counts by multiplying them by a scale factor +between 0 and 1.0. + +Scaling is performed when either usage or construction attempt counts +exceed a parametrized value. + +To avoid error due to scaling during circuit construction and use, +currently open circuits are subtracted from the usage counts before +scaling, and added back after scaling. + + +## Parametrization + +The following consensus parameters tune various aspects of the +defense. + +```text + pb_mincircs + Default: 150 + Min: 5 + Effect: This is the minimum number of circuits that must complete + at least 2 hops before we begin evaluating construction rates. + + pb_noticepct + Default: 70 + Min: 0 + Max: 100 + Effect: If the circuit success rate falls below this percentage, + we emit a notice log message. + + pb_warnpct + Default: 50 + Min: 0 + Max: 100 + Effect: If the circuit success rate falls below this percentage, + we emit a warn log message. + + pb_extremepct + Default: 30 + Min: 0 + Max: 100 + Effect: If the circuit success rate falls below this percentage, + we emit a more alarmist warning log message. If + pb_dropguard is set to 1, we also disable the use of the + guard. + + pb_dropguards + Default: 0 + Min: 0 + Max: 1 + Effect: If the circuit success rate falls below pb_extremepct, + when pb_dropguard is set to 1, we disable use of that + guard. + + pb_scalecircs + Default: 300 + Min: 10 + Effect: After this many circuits have completed at least two hops, + Tor performs the scaling described in Section 7.3. + + pb_multfactor and pb_scalefactor + Default: 1/2 + Min: 0.0 + Max: 1.0 + Effect: The double-precision result obtained from + pb_multfactor/pb_scalefactor is multiplied by our current + counts to scale them. + + pb_minuse + Default: 20 + Min: 3 + Effect: This is the minimum number of circuits that we must attempt to + use before we begin evaluating construction rates. + + pb_noticeusepct + Default: 80 + Min: 3 + Effect: If the circuit usage success rate falls below this percentage, + we emit a notice log message. + + pb_extremeusepct + Default: 60 + Min: 3 + Effect: If the circuit usage success rate falls below this percentage, + we emit a warning log message. We also disable the use of the + guard if pb_dropguards is set. + + pb_scaleuse + Default: 100 + Min: 10 + Effect: After we have attempted to use this many circuits, + Tor performs the scaling described in Section 7.3. +``` + + +## Known barriers to enforcement + +Due to intermittent CPU overload at relays, the normal rate of +successful circuit completion is highly variable. The Guard-dropping +version of the defense is unlikely to be deployed until the ntor +circuit handshake is enabled, or the nature of CPU overload induced +failure is better understood. + diff --git a/spec/path-spec/general-operation.md b/spec/path-spec/general-operation.md new file mode 100644 index 0000000..7a1c281 --- /dev/null +++ b/spec/path-spec/general-operation.md @@ -0,0 +1,94 @@ + +# General operation + +Tor begins building circuits as soon as it has enough directory +information to do so (see section 5 of dir-spec.txt). Some circuits are +built preemptively because we expect to need them later (for user +traffic), and some are built because of immediate need (for user traffic +that no current circuit can handle, for testing the network or our +reachability, and so on). + +```text + [Newer versions of Tor (0.2.6.2-alpha and later): + If the consensus contains Exits (the typical case), Tor will build both + exit and internal circuits. When bootstrap completes, Tor will be ready + to handle an application requesting an exit circuit to services like the + World Wide Web. +``` + +If the consensus does not contain Exits, Tor will only build internal +circuits. In this case, earlier statuses will have included "internal" +as indicated above. When bootstrap completes, Tor will be ready to handle +an application requesting an internal circuit to hidden services at +".onion" addresses. + +If a future consensus contains Exits, exit circuits may become available.] + +When a client application creates a new stream (by opening a SOCKS +connection or launching a resolve request), we attach it to an appropriate +open circuit if one exists, or wait if an appropriate circuit is +in-progress. We launch a new circuit only +if no current circuit can handle the request. We rotate circuits over +time to avoid some profiling attacks. + +To build a circuit, we choose all the nodes we want to use, and then +construct the circuit. Sometimes, when we want a circuit that ends at a +given hop, and we have an appropriate unused circuit, we "cannibalize" the +existing circuit and extend it to the new terminus. + +These processes are described in more detail below. + +This document describes Tor's automatic path selection logic only; path +selection can be overridden by a controller (with the EXTENDCIRCUIT and +ATTACHSTREAM commands). Paths constructed through these means may +violate some constraints given below. + + +## Terminology + +A "path" is an ordered sequence of nodes, not yet built as a circuit. + +A "clean" circuit is one that has not yet been used for any traffic. + +A "fast" or "stable" or "valid" node is one that has the 'Fast' or +'Stable' or 'Valid' flag +set respectively, based on our current directory information. A "fast" +or "stable" circuit is one consisting only of "fast" or "stable" nodes. + +In an "exit" circuit, the final node is chosen based on waiting stream +requests if any, and in any case it avoids nodes with exit policy of +"reject *:*". An "internal" circuit, on the other hand, is one where +the final node is chosen just like a middle node (ignoring its exit +policy). + +A "request" is a client-side stream or DNS resolve that needs to be +served by a circuit. + +A "pending" circuit is one that we have started to build, but which has +not yet completed. + +A circuit or path "supports" a request if it is okay to use the +circuit/path to fulfill the request, according to the rules given below. +A circuit or path "might support" a request if some aspect of the request +is unknown (usually its target IP), but we believe the path probably +supports the request according to the rules given below. + + +## A relay's bandwidth + +Old versions of Tor did not report bandwidths in network status +documents, so clients had to learn them from the routers' advertised +relay descriptors. + +For versions of Tor prior to 0.2.1.17-rc, everywhere below where we +refer to a relay's "bandwidth", we mean its clipped advertised +bandwidth, computed by taking the smaller of the 'rate' and +'observed' arguments to the "bandwidth" element in the relay's +descriptor. If a router's advertised bandwidth is greater than +MAX_BELIEVABLE_BANDWIDTH (currently 10 MB/s), we clipped to that +value. + +For more recent versions of Tor, we take the bandwidth value declared +in the consensus, and fall back to the clipped advertised bandwidth +only if the consensus does not have bandwidths listed. + diff --git a/spec/path-spec/guard-nodes.md b/spec/path-spec/guard-nodes.md new file mode 100644 index 0000000..6b7bc8b --- /dev/null +++ b/spec/path-spec/guard-nodes.md @@ -0,0 +1,43 @@ + +# Guard nodes + +We use Guard nodes (also called "helper nodes" in the research +literature) to prevent certain profiling attacks. For an overview of +our Guard selection algorithm -- which has grown rather complex -- see +guard-spec.txt. + + +## How consensus bandwidth weights factor into entry guard selection + +When weighting a list of routers for choosing an entry guard, the following +consensus parameters (from the "bandwidth-weights" line) apply: + +```text + Wgg - Weight for Guard-flagged nodes in the guard position + Wgm - Weight for non-flagged nodes in the guard Position + Wgd - Weight for Guard+Exit-flagged nodes in the guard Position + Wgb - Weight for BEGIN_DIR-supporting Guard-flagged nodes + Wmb - Weight for BEGIN_DIR-supporting non-flagged nodes + Web - Weight for BEGIN_DIR-supporting Exit-flagged nodes + Wdb - Weight for BEGIN_DIR-supporting Guard+Exit-flagged nodes +``` + +Please see "bandwidth-weights" in §3.4.1 of dir-spec.txt for more in depth +descriptions of these parameters. + +If a router has been marked as both an entry guard and an exit, then we +prefer to use it more, with our preference for doing so (roughly) linearly +increasing w.r.t. the router's non-guard bandwidth and bandwidth weight +(calculated without taking the guard flag into account). From proposal +#236: + +| +| Let Wpf denote the weight from the 'bandwidth-weights' line a +| client would apply to N for position p if it had the guard +| flag, Wpn the weight if it did not have the guard flag, and B the +| measured bandwidth of N in the consensus. Then instead of choosing +| N for position p proportionally to Wpf*B or Wpn*B, clients should +| choose N proportionally to F*Wpf*B + (1-F)*Wpn*B. + +where F is the weight as calculated using the above parameters. + diff --git a/spec/path-spec/handling-failure.md b/spec/path-spec/handling-failure.md new file mode 100644 index 0000000..d7bb80f --- /dev/null +++ b/spec/path-spec/handling-failure.md @@ -0,0 +1,17 @@ + +## Handling failure + +If an attempt to extend a circuit fails (either because the first create +failed or a subsequent extend failed) then the circuit is torn down and is +no longer pending. (XXXX really?) Requests that might have been +supported by the pending circuit thus become unsupported, and a new +circuit needs to be constructed. + +If a stream "begin" attempt fails with an EXITPOLICY error, we +decide that the exit node's exit policy is not correctly advertised, +so we treat the exit node as if it were a non-exit until we retrieve +a fresh descriptor for it. + +Excessive amounts of either type of failure can indicate an +attack on anonymity. See section 7 for how excessive failure is handled. + diff --git a/spec/path-spec/hidden-service-related-circuits.md b/spec/path-spec/hidden-service-related-circuits.md new file mode 100644 index 0000000..e0e78e3 --- /dev/null +++ b/spec/path-spec/hidden-service-related-circuits.md @@ -0,0 +1,5 @@ + +# Hidden-service related circuits + +XXX Tracking expected hidden service use (client-side and hidserv-side) + diff --git a/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md b/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md new file mode 100644 index 0000000..75f8641 --- /dev/null +++ b/spec/path-spec/learning-when-to-give-up-timeout-on-circuit-construction.md @@ -0,0 +1,293 @@ + +## Learning when to give up ("timeout") on circuit construction + +Since version 0.2.2.8-alpha, Tor clients attempt to learn when to give +up on circuits based on network conditions. + + +### Distribution choice + +Based on studies of build times, we found that the distribution of +circuit build times appears to be a Frechet distribution (and a multi-modal +Frechet distribution, if more than one guard or bridge is used). However, +estimators and quantile functions of the Frechet distribution are difficult +to work with and slow to converge. So instead, since we are only interested +in the accuracy of the tail, clients approximate the tail of the multi-modal +distribution with a single Pareto curve. + + +### How much data to record + +From our observations, the minimum number of circuit build times for a +reasonable fit appears to be on the order of 100. However, to keep a +good fit over the long term, clients store 1000 most recent circuit build +times in a circular array. + +These build times only include the times required to build three-hop +circuits, and the times required to build the first three hops of circuits +with more than three hops. Circuits of fewer than three hops are not +recorded, and hops past the third are not recorded. + +The Tor client should build test circuits at a rate of one every 'cbttestfreq' +(10 seconds) until 'cbtmincircs' (100 circuits) are built, with a maximum of +'cbtmaxopencircs' (default: 10) circuits open at once. This allows a fresh +Tor to have a CircuitBuildTimeout estimated within 30 minutes after install +or network change (see section 2.4.5 below). + +Timeouts are stored on disk in a histogram of 10ms bin width, the same +width used to calculate the Xm value above. The timeouts recorded in the +histogram must be shuffled after being read from disk, to preserve a +proper expiration of old values after restart. + +Thus, some build time resolution is lost during restart. Implementations may +choose a different persistence mechanism than this histogram, but be aware +that build time binning is still needed for parameter estimation. + + +### Parameter estimation + +Once 'cbtmincircs' build times are recorded, Tor clients update the +distribution parameters and recompute the timeout every circuit completion +(though see section 2.4.5 for when to pause and reset timeout due to +too many circuits timing out). + +Tor clients calculate the parameters for a Pareto distribution fitting the +data using the maximum likelihood estimator. For derivation, see: +https://en.wikipedia.org/wiki/Pareto_distribution#Estimation_of_parameters + +Because build times are not a true Pareto distribution, we alter how Xm is +computed. In a max likelihood estimator, the mode of the distribution is +used directly as Xm. + +Instead of using the mode of discrete build times directly, Tor clients +compute the Xm parameter using the weighted average of the midpoints +of the 'cbtnummodes' (10) most frequently occurring 10ms histogram bins. +Ties are broken in favor of earlier bins (that is, in favor of bins +corresponding to shorter build times). + +(The use of 10 modes was found to minimize error from the selected +cbtquantile, with 10ms bins for quantiles 60-80, compared to many other +heuristics). + +To avoid ln(1.0+epsilon) precision issues, use log laws to rewrite the +estimator for 'alpha' as the sum of logs followed by subtraction, rather +than multiplication and division: + +alpha = n/(Sum_n{ln(MAX(Xm, x_i))} - n*ln(Xm)) + +In this, n is the total number of build times that have completed, x_i is +the ith recorded build time, and Xm is the modes of x_i as above. + +All times below Xm are counted as having the Xm value via the MAX(), +because in Pareto estimators, Xm is supposed to be the lowest value. +However, since clients use mode averaging to estimate Xm, there can be +values below our Xm. Effectively, the Pareto estimator then treats that +everything smaller than Xm happened at Xm. One can also see that if +clients did not do this, alpha could underflow to become negative, which +results in an exponential curve, not a Pareto probability distribution. + +The timeout itself is calculated by using the Pareto Quantile function (the +inverted CDF) to give us the value on the CDF such that 80% of the mass +of the distribution is below the timeout value (parameter 'cbtquantile'). + +The Pareto Quantile Function (inverse CDF) is: + +F(q) = Xm/((1.0-q)^(1.0/alpha)) + +Thus, clients obtain the circuit build timeout for 3-hop circuits by +computing: + +timeout_ms = F(0.8) # 'cbtquantile' == 0.8 + +With this, we expect that the Tor client will accept the fastest 80% of the +total number of paths on the network. + +Clients obtain the circuit close time to completely abandon circuits as: + +close_ms = F(0.99) # 'cbtclosequantile' == 0.99 + +To avoid waiting an unreasonably long period of time for circuits that +simply have relays that are down, Tor clients cap timeout_ms at the max +build time actually observed so far, and cap close_ms at twice this max, +but at least 60 seconds: + +```text + timeout_ms = MIN(timeout_ms, max_observed_timeout) + close_ms = MAX(MIN(close_ms, 2*max_observed_timeout), 'cbtinitialtimeout') +``` + + +### Calculating timeouts thresholds for circuits of different lengths + +The timeout_ms and close_ms estimates above are good only for 3-hop +circuits, since only 3-hop circuits are recorded in the list of build +times. + +To calculate the appropriate timeouts and close timeouts for circuits of +other lengths, the client multiples the timeout_ms and close_ms values +by a scaling factor determined by the number of communication hops +needed to build their circuits: + +timeout_ms[hops=n] = timeout_ms * Actions(N) / Actions(3) + +close_ms[hops=n] = close_ms * Actions(N) / Actions(3) + +where Actions(N) = N * (N + 1) / 2. + +To calculate timeouts for operations other than circuit building, +the client should add X to Actions(N) for every round-trip communication +required with the Xth hop. + + +### How to record timeouts + +Pareto estimators begin to lose their accuracy if the tail is omitted. +Hence, Tor clients actually calculate two timeouts: a usage timeout, and a +close timeout. + +Circuits that pass the usage timeout are marked as measurement circuits, +and are allowed to continue to build until the close timeout corresponding +to the point 'cbtclosequantile' (default 99) on the Pareto curve, or 60 +seconds, whichever is greater. + +The actual completion times for these measurement circuits should be +recorded. + +Implementations should completely abandon a circuit and ignore the circuit +if the total build time exceeds the close threshold. Such closed circuits +should be ignored, as this typically means one of the relays in the path is +offline. + + +### Detecting Changing Network Conditions + +Tor clients attempt to detect both network connectivity loss and drastic +changes in the timeout characteristics. + +To detect changing network conditions, clients keep a history of +the timeout or non-timeout status of the past 'cbtrecentcount' circuits +(20 circuits) that successfully completed at least one hop. If more than +90% of these circuits timeout, the client discards all buildtimes history, +resets the timeout to 'cbtinitialtimeout' (60 seconds), and then begins +recomputing the timeout. + +If the timeout was already at least `cbtinitialtimeout`, +the client doubles the timeout. + +The records here (of how many circuits succeeded or failed among the most +recent 'cbrrecentcount') are not stored as persistent state. On reload, +we start with a new, empty state. + + +### Consensus parameters governing behavior + +Clients that implement circuit build timeout learning should obey the +following consensus parameters that govern behavior, in order to allow +us to handle bugs or other emergent behaviors due to client circuit +construction. If these parameters are not present in the consensus, +the listed default values should be used instead. + +```text + cbtdisabled + Default: 0 + Min: 0 + Max: 1 + Effect: If 1, all CircuitBuildTime learning code should be + disabled and history should be discarded. For use in + emergency situations only. + + cbtnummodes + Default: 10 + Min: 1 + Max: 20 + Effect: This value governs how many modes to use in the weighted + average calculation of Pareto parameter Xm. Selecting Xm as the + average of multiple modes improves accuracy of the Pareto tail + for quantile cutoffs from 60-80% (see cbtquantile). + + cbtrecentcount + Default: 20 + Min: 3 + Max: 1000 + Effect: This is the number of circuit build outcomes (success vs + timeout) to keep track of for the following option. + + cbtmaxtimeouts + Default: 18 + Min: 3 + Max: 10000 + Effect: When this many timeouts happen in the last 'cbtrecentcount' + circuit attempts, the client should discard all of its + history and begin learning a fresh timeout value. + + Note that if this parameter's value is greater than the value + of 'cbtrecentcount', then the history will never be + discarded because of this feature. + + cbtmincircs + Default: 100 + Min: 1 + Max: 10000 + Effect: This is the minimum number of circuits to build before + computing a timeout. + + Note that if this parameter's value is higher than 1000 (the + number of time observations that a client keeps in its + circular buffer), circuit build timeout calculation is + effectively disabled, and the default timeouts are used + indefinitely. + + cbtquantile + Default: 80 + Min: 10 + Max: 99 + Effect: This is the position on the quantile curve to use to set the + timeout value. It is a percent (10-99). + + cbtclosequantile + Default: 99 + Min: Value of cbtquantile parameter + Max: 99 + Effect: This is the position on the quantile curve to use to set the + timeout value to use to actually close circuits. It is a + percent (0-99). + + cbttestfreq + Default: 10 + Min: 1 + Max: 2147483647 (INT32_MAX) + Effect: Describes how often in seconds to build a test circuit to + gather timeout values. Only applies if less than 'cbtmincircs' + have been recorded. + + cbtmintimeout + Default: 10 + Min: 10 + Max: 2147483647 (INT32_MAX) + Effect: This is the minimum allowed timeout value in milliseconds. + + cbtinitialtimeout + Default: 60000 + Min: Value of cbtmintimeout + Max: 2147483647 (INT32_MAX) + Effect: This is the timeout value to use before we have enough data + to compute a timeout, in milliseconds. If we do not have + enough data to compute a timeout estimate (see cbtmincircs), + then we use this interval both for the close timeout and the + abandon timeout. + + cbtlearntimeout + Default: 180 + Min: 10 + Max: 60000 + Effect: This is how long idle circuits will be kept open while cbt is + learning a new timeout value. + + cbtmaxopencircs + Default: 10 + Min: 0 + Max: 14 + Effect: This is the maximum number of circuits that can be open at + at the same time during the circuit build time learning phase. +``` + diff --git a/spec/path-spec/old-notes.md b/spec/path-spec/old-notes.md new file mode 100644 index 0000000..10623c1 --- /dev/null +++ b/spec/path-spec/old-notes.md @@ -0,0 +1,60 @@ + +# Old notes + + +## Do we actually do this? + +```text +How to deal with network down. + - While all helpers are down/unreachable and there are no established + or on-the-way testing circuits, launch a testing circuit. (Do this + periodically in the same way we try to establish normal circuits + when things are working normally.) + (Testing circuits are a special type of circuit, that streams won't + attach to by accident.) + - When a testing circuit succeeds, mark all helpers up and hold + the testing circuit open. + - If a connection to a helper succeeds, close all testing circuits. + Else mark that helper down and try another. + - If the last helper is marked down and we already have a testing + circuit established, then add the first hop of that testing circuit + to the end of our helper node list, close that testing circuit, + and go back to square one. (Actually, rather than closing the + testing circuit, can we get away with converting it to a normal + circuit and beginning to use it immediately?) +``` + +[Do we actually do any of the above? If so, let's spec it. If not, let's +remove it. -NM] + + +## A thing we could do to deal with reachability. + +And as a bonus, it leads to an answer to Nick's attack ("If I pick +my helper nodes all on 18.0.0.0:*, then I move, you'll know where I +bootstrapped") -- the answer is to pick your original three helper nodes +without regard for reachability. Then the above algorithm will add some +more that are reachable for you, and if you move somewhere, it's more +likely (though not certain) that some of the originals will become useful. +Is that smart or just complex? + + +## Some stuff that worries me about entry guards. 2006 Jun, Nickm. + +It is unlikely for two users to have the same set of entry guards. +Observing a user is sufficient to learn its entry guards. So, as we move +around, entry guards make us linkable. If we want to change guards when +our location (IP? subnet?) changes, we have two bad options. We could + +```text + - Drop the old guards. But if we go back to our old location, + we'll not use our old guards. For a laptop that sometimes gets used + from work and sometimes from home, this is pretty fatal. + - Remember the old guards as associated with the old location, and use + them again if we ever go back to the old location. This would be + nasty, since it would force us to record where we've been. +``` + +[Do we do any of this now? If not, this should move into 099-misc or +098-todo. -NM] + diff --git a/spec/path-spec/path-selection-constraints.md b/spec/path-spec/path-selection-constraints.md new file mode 100644 index 0000000..f590b36 --- /dev/null +++ b/spec/path-spec/path-selection-constraints.md @@ -0,0 +1,130 @@ + +## Path selection and constraints + +We choose the path for each new circuit before we build it. We choose the +exit node first, followed by the other nodes in the circuit, front to +back. (In other words, for a 3-hop circuit, we first pick hop 3, +then hop 1, then hop 2.) All paths we generate obey the following +constraints: + +```text + - We do not choose the same router twice for the same path. + - We do not choose any router in the same family as another in the same + path. (Two routers are in the same family if each one lists the other + in the "family" entries of its descriptor.) + - We do not choose more than one router in a given /16 subnet + (unless EnforceDistinctSubnets is 0). + - We don't choose any non-running or non-valid router unless we have + been configured to do so. By default, we are configured to allow + non-valid routers in "middle" and "rendezvous" positions. + - If we're using Guard nodes, the first node must be a Guard (see 5 + below) + - XXXX Choosing the length +``` + +For "fast" circuits, we only choose nodes with the Fast flag. For +non-"fast" circuits, all nodes are eligible. + +For all circuits, we weight node selection according to router bandwidth. + +We also weight the bandwidth of Exit and Guard flagged nodes depending on +the fraction of total bandwidth that they make up and depending upon the +position they are being selected for. + +These weights are published in the consensus, and are computed as described +in Section "Computing Bandwidth Weights" of dir-spec.txt. They are: + +```text + Wgg - Weight for Guard-flagged nodes in the guard position + Wgm - Weight for non-flagged nodes in the guard Position + Wgd - Weight for Guard+Exit-flagged nodes in the guard Position + + Wmg - Weight for Guard-flagged nodes in the middle Position + Wmm - Weight for non-flagged nodes in the middle Position + Wme - Weight for Exit-flagged nodes in the middle Position + Wmd - Weight for Guard+Exit flagged nodes in the middle Position + + Weg - Weight for Guard flagged nodes in the exit Position + Wem - Weight for non-flagged nodes in the exit Position + Wee - Weight for Exit-flagged nodes in the exit Position + Wed - Weight for Guard+Exit-flagged nodes in the exit Position + + Wgb - Weight for BEGIN_DIR-supporting Guard-flagged nodes + Wmb - Weight for BEGIN_DIR-supporting non-flagged nodes + Web - Weight for BEGIN_DIR-supporting Exit-flagged nodes + Wdb - Weight for BEGIN_DIR-supporting Guard+Exit-flagged nodes + + Wbg - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests + Wbm - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests + Wbe - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests + Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests +``` + +If any of those weights is malformed or not present in a consensus, +clients proceed with the regular path selection algorithm setting +the weights to the default value of 10000. + +Additionally, we may be building circuits with one or more requests in +mind. Each kind of request puts certain constraints on paths: + +```text + - All service-side introduction circuits and all rendezvous paths + should be Stable. + - All connection requests for connections that we think will need to + stay open a long time require Stable circuits. Currently, Tor decides + this by examining the request's target port, and comparing it to a + list of "long-lived" ports. (Default: 21, 22, 706, 1863, 5050, + 5190, 5222, 5223, 6667, 6697, 8300.) + - DNS resolves require an exit node whose exit policy is not equivalent + to "reject *:*". + - Reverse DNS resolves require a version of Tor with advertised eventdns + support (available in Tor 0.1.2.1-alpha-dev and later). + - All connection requests require an exit node whose exit policy + supports their target address and port (if known), or which "might + support it" (if the address isn't known). See 2.2.1. + - Rules for Fast? XXXXX +``` + + +### Choosing an exit + +If we know what IP address we want to connect to or resolve, we can +trivially tell whether a given router will support it by simulating +its declared exit policy. + +Because we often connect to addresses of the form hostname:port, we do not +always know the target IP address when we select an exit node. In these +cases, we need to pick an exit node that "might support" connections to a +given address port with an unknown address. An exit node "might support" +such a connection if any clause that accepts any connections to that port +precedes all clauses (if any) that reject all connections to that port. + +Unless requested to do so by the user, we never choose an exit node +flagged as "BadExit" by more than half of the authorities who advertise +themselves as listing bad exits. + + +### User configuration + +Users can alter the default behavior for path selection with configuration +options. + +```text + - If "ExitNodes" is provided, then every request requires an exit node on + the ExitNodes list. (If a request is supported by no nodes on that list, + and StrictExitNodes is false, then Tor treats that request as if + ExitNodes were not provided.) + + - "EntryNodes" and "StrictEntryNodes" behave analogously. + + - If a user tries to connect to or resolve a hostname of the form + ..exit, the request is rewritten to a request for + , and the request is only supported by the exit whose nickname + or fingerprint is . + + - When set, "HSLayer2Nodes" and "HSLayer3Nodes" relax Tor's path + restrictions to allow nodes in the same /16 and node family to reappear + in the path. They also allow the guard node to be chosen as the RP, IP, + and HSDIR, and as the hop before those positions. +``` + diff --git a/spec/path-spec/server-descriptor-purposes.md b/spec/path-spec/server-descriptor-purposes.md new file mode 100644 index 0000000..6e3a71e --- /dev/null +++ b/spec/path-spec/server-descriptor-purposes.md @@ -0,0 +1,19 @@ + +# Server descriptor purposes + +There are currently three "purposes" supported for server descriptors: +general, controller, and bridge. Most descriptors are of type general +-- these are the ones listed in the consensus, and the ones fetched +and used in normal cases. + +Controller-purpose descriptors are those delivered by the controller +and labelled as such: they will be kept around (and expire like +normal descriptors), and they can be used by the controller in its +CIRCUITEXTEND commands. Otherwise they are ignored by Tor when it +chooses paths. + +Bridge-purpose descriptors are for routers that are used as bridges. See +doc/design-paper/blocking.pdf for more design explanation, or proposal +125 for specific details. Currently bridge descriptors are used in place +of normal entry guards, for Tor clients that have UseBridges enabled. + diff --git a/spec/path-spec/when-we-build.md b/spec/path-spec/when-we-build.md new file mode 100644 index 0000000..4320c58 --- /dev/null +++ b/spec/path-spec/when-we-build.md @@ -0,0 +1,161 @@ + +## When we build + + +### We don't build circuits until we have enough directory info + +There's a class of possible attacks where our directory servers +only give us information about the relays that they would like us +to use. To prevent this attack, we don't build multi-hop +circuits for real traffic (like those in 2.1.1, 2.1.2, 2.1.4 +below) until we have enough directory information to be +reasonably confident this attack isn't being done to us. + +Here, "enough" directory information is defined as: + +```text + * Having a consensus that's been valid at some point in the + last REASONABLY_LIVE_TIME interval (24 hours). + + * Having enough descriptors that we could build at least some + fraction F of all bandwidth-weighted paths, without taking + ExitNodes/EntryNodes/etc into account. + + (F is set by the PathsNeededToBuildCircuits option, + defaulting to the 'min_paths_for_circs_pct' consensus + parameter, with a final default value of 60%.) + + * Having enough descriptors that we could build at least some + fraction F of all bandwidth-weighted paths, _while_ taking + ExitNodes/EntryNodes/etc into account. + + (F is as above.) + + * Having a descriptor for every one of the first + NUM_USABLE_PRIMARY_GUARDS guards among our primary guards. (see + guard-spec.txt) +``` + +We define the "fraction of bandwidth-weighted paths" as the product of +these three fractions. + +```text + * The fraction of descriptors that we have for nodes with the Guard + flag, weighted by their bandwidth for the guard position. + * The fraction of descriptors that we have for all nodes, + weighted by their bandwidth for the middle position. + * The fraction of descriptors that we have for nodes with the Exit + flag, weighted by their bandwidth for the exit position. +``` + +If the consensus has zero weighted bandwidth for a given kind of +relay (Guard, Middle, or Exit), Tor instead uses the fraction of relays +for which it has the descriptor (not weighted by bandwidth at all). + +If the consensus lists zero exit-flagged relays, Tor instead uses the +fraction of middle relays. + + +### Clients build circuits preemptively + +When running as a client, Tor tries to maintain at least a certain +number of clean circuits, so that new streams can be handled +quickly. To increase the likelihood of success, Tor tries to +predict what circuits will be useful by choosing from among nodes +that support the ports we have used in the recent past (by default +one hour). Specifically, on startup Tor tries to maintain one clean +fast exit circuit that allows connections to port 80, and at least +two fast clean stable internal circuits in case we get a resolve +request or hidden service request (at least three if we _run_ a +hidden service). + +After that, Tor will adapt the circuits that it preemptively builds +based on the requests it sees from the user: it tries to have two fast +clean exit circuits available for every port seen within the past hour +(each circuit can be adequate for many predicted ports -- it doesn't +need two separate circuits for each port), and it tries to have the +above internal circuits available if we've seen resolves or hidden +service activity within the past hour. If there are 12 or more clean +circuits open, it doesn't open more even if it has more predictions. + +Only stable circuits can "cover" a port that is listed in the +LongLivedPorts config option. Similarly, hidden service requests +to ports listed in LongLivedPorts make us create stable internal +circuits. + +Note that if there are no requests from the user for an hour, Tor +will predict no use and build no preemptive circuits. + +The Tor client SHOULD NOT store its list of predicted requests to a +persistent medium. + + +### Clients build circuits on demand + +Additionally, when a client request exists that no circuit (built or +pending) might support, we create a new circuit to support the request. +For exit connections, we pick an exit node that will handle the +most pending requests (choosing arbitrarily among ties), launch a +circuit to end there, and repeat until every unattached request +might be supported by a pending or built circuit. For internal +circuits, we pick an arbitrary acceptable path, repeating as needed. + +Clients consider a circuit to become "dirty" as soon as a stream is +attached to it, or some other request is performed over the circuit. +If a circuit has been "dirty" for at least MaxCircuitDirtiness seconds, +new circuits may not be attached to it. + +In some cases we can reuse an already established circuit if it's +clean; see Section 2.3 (cannibalizing circuits) for details. + + +### Relays build circuits for testing reachability and bandwidth + +Tor relays test reachability of their ORPort once they have +successfully built a circuit (on startup and whenever their IP address +changes). They build an ordinary fast internal circuit with themselves +as the last hop. As soon as any testing circuit succeeds, the Tor +relay decides it's reachable and is willing to publish a descriptor. + +We launch multiple testing circuits (one at a time), until we +have NUM_PARALLEL_TESTING_CIRC (4) such circuits open. Then we +do a "bandwidth test" by sending a certain number of relay drop +cells down each circuit: BandwidthRate * 10 / CELL_NETWORK_SIZE +total cells divided across the four circuits, but never more than +CIRCWINDOW_START (1000) cells total. This exercises both outgoing and +incoming bandwidth, and helps to jumpstart the observed bandwidth +(see dir-spec.txt). + +Tor relays also test reachability of their DirPort once they have +established a circuit, but they use an ordinary exit circuit for +this purpose. + + +### Hidden-service circuits + +See section 4 below. + + +### Rate limiting of failed circuits + +If we fail to build a circuit N times in a X second period (see Section +2.3 for how this works), we stop building circuits until the X seconds +have elapsed. +XXXX + + +### When to tear down circuits + +Clients should tear down circuits (in general) only when those circuits +have no streams on them. Additionally, clients should tear-down +stream-less circuits only under one of the following conditions: + +```text + - The circuit has never had a stream attached, and it was created too + long in the past (based on CircuitsAvailableTimeout or + cbtlearntimeout, depending on timeout estimate status). + + - The circuit is dirty (has had a stream attached), and it has been + dirty for at least MaxCircuitDirtiness. +``` + diff --git a/spec/pt-spec-intro.md b/spec/pt-spec-intro.md new file mode 100644 index 0000000..016effa --- /dev/null +++ b/spec/pt-spec-intro.md @@ -0,0 +1,38 @@ +Pluggable Transport Specification (Version 1) + +Abstract + +Pluggable Transports (PTs) are a generic mechanism for the rapid +development and deployment of censorship circumvention, +based around the idea of modular sub-processes that transform +traffic to defeat censors. + +This document specifies the sub-process startup, shutdown, +and inter-process communication mechanisms required to utilize +PTs. + +Table of Contents + +```text + 1. Introduction + 1.1. Requirements Notation + 2. Architecture Overview + 3. Specification + 3.1. Pluggable Transport Naming + 3.2. Pluggable Transport Configuration Environment Variables + 3.2.1. Common Environment Variables + 3.2.2. Pluggable Transport Client Environment Variables + 3.2.3. Pluggable Transport Server Environment Variables + 3.3. Pluggable Transport To Parent Process Communication + 3.3.1. Common Messages + 3.3.2. Pluggable Transport Client Messages + 3.3.3. Pluggable Transport Server Messages + 3.4. Pluggable Transport Shutdown + 3.5. Pluggable Transport Client Per-Connection Arguments + 4. Anonymity Considerations + 5 References + 6. Acknowledgments + Appendix A. Example Client Pluggable Transport Session + Appendix B. Example Server Pluggable Transport Session +``` + diff --git a/spec/pt-spec/acknowledgments.md b/spec/pt-spec/acknowledgments.md new file mode 100644 index 0000000..8c12c83 --- /dev/null +++ b/spec/pt-spec/acknowledgments.md @@ -0,0 +1,6 @@ + +# Acknowledgments + +This specification draws heavily from prior versions done by Jacob +Appelbaum, Nick Mathewson, and George Kadianakis. + diff --git a/spec/pt-spec/anonymity-considerations.md b/spec/pt-spec/anonymity-considerations.md new file mode 100644 index 0000000..80b3826 --- /dev/null +++ b/spec/pt-spec/anonymity-considerations.md @@ -0,0 +1,23 @@ + +# Anonymity Considerations + +When designing and implementing a Pluggable Transport, care +should be taken to preserve the privacy of clients and to avoid +leaking personally identifying information. + +Examples of client related considerations are: + +- Not logging client IP addresses to disk. + +- Not leaking DNS addresses except when necessary. + +```text + - Ensuring that "TOR_PT_PROXY"'s "fail closed" behavior is + implemented correctly. +``` + +Additionally, certain obfuscation mechanisms rely on information +such as the server IP address/port being confidential, so clients +also need to take care to preserve server side information +confidential when applicable. + diff --git a/spec/pt-spec/architecture-overview.md b/spec/pt-spec/architecture-overview.md new file mode 100644 index 0000000..44718b2 --- /dev/null +++ b/spec/pt-spec/architecture-overview.md @@ -0,0 +1,37 @@ + +# Architecture Overview + +```text + +------------+ +---------------------------+ + | Client App +-- Local Loopback --+ PT Client (SOCKS Proxy) +--+ + +------------+ +---------------------------+ | + | + Public Internet (Obfuscated/Transformed traffic) ==> | + | + +------------+ +---------------------------+ | + | Server App +-- Local Loopback --+ PT Server (Reverse Proxy) +--+ + +------------+ +---------------------------+ +``` + +On the client's host, the PT Client software exposes a SOCKS proxy +[RFC1928] to the client application, and obfuscates or otherwise +transforms traffic before forwarding it to the server's host. + +On the server's host, the PT Server software exposes a reverse proxy +that accepts connections from PT Clients, and handles reversing the +obfuscation/transformation applied to traffic, before forwarding it +to the actual server software. An optional lightweight protocol +exists to facilitate communicating connection meta-data that would +otherwise be lost such as the source IP address and port +[EXTORPORT]. + +All PT instances are configured by the respective parent process via +a set of standardized environment variables (3.2) that are set at +launch time, and report status information back to the parent via +writing output in a standardized format to stdout (3.3). + +Each invocation of a PT MUST be either a client OR a server. + +All PT client forward proxies MUST support either SOCKS 4 or SOCKS 5, +and SHOULD prefer SOCKS 5 over SOCKS 4. + diff --git a/spec/pt-spec/example-client-pluggable-transport-session.md b/spec/pt-spec/example-client-pluggable-transport-session.md new file mode 100644 index 0000000..017bcaf --- /dev/null +++ b/spec/pt-spec/example-client-pluggable-transport-session.md @@ -0,0 +1,19 @@ + +# Appendix A: Example Client Pluggable Transport Session + +Environment variables: + +TOR_PT_MANAGED_TRANSPORT_VER=1 +TOR_PT_STATE_LOCATION=/var/lib/tor/pt_state/ +TOR_PT_EXIT_ON_STDIN_CLOSE=1 +TOR_PT_PROXY=socks5://127.0.0.1:8001 +TOR_PT_CLIENT_TRANSPORTS=obfs3,obfs4 + +Messages the PT Proxy writes to stdin: + +VERSION 1 +PROXY DONE +CMETHOD obfs3 socks5 127.0.0.1:32525 +CMETHOD obfs4 socks5 127.0.0.1:37347 +CMETHODS DONE + diff --git a/spec/pt-spec/example-server-pluggable-transport-session.md b/spec/pt-spec/example-server-pluggable-transport-session.md new file mode 100644 index 0000000..15a9ab1 --- /dev/null +++ b/spec/pt-spec/example-server-pluggable-transport-session.md @@ -0,0 +1,18 @@ + +# Appendix B: Example Server Pluggable Transport Session + +Environment variables: + +TOR_PT_MANAGED_TRANSPORT_VER=1 +TOR_PT_STATE_LOCATION=/var/lib/tor/pt_state +TOR_PT_EXIT_ON_STDIN_CLOSE=1 +TOR_PT_SERVER_TRANSPORTS=obfs3,obfs4 +TOR_PT_SERVER_BINDADDR=obfs3-198.51.100.1:1984 + +Messages the PT Proxy writes to stdin: + +VERSION 1 +SMETHOD obfs3 198.51.100.1:1984 +SMETHOD obfs4 198.51.100.1:43734 ARGS:cert=HszPy3vWfjsESCEOo9ZBkRv6zQ/1mGHzc8arF0y2SpwFr3WhsMu8rK0zyaoyERfbz3ddFw,iat-mode=0 +SMETHODS DONE + diff --git a/spec/pt-spec/introduction.md b/spec/pt-spec/introduction.md new file mode 100644 index 0000000..fae42bc --- /dev/null +++ b/spec/pt-spec/introduction.md @@ -0,0 +1,26 @@ + +# Introduction + +This specification describes a way to decouple protocol-level +obfuscation from an application's client/server code, in a manner +that promotes rapid development of obfuscation/circumvention +tools and promotes reuse beyond the scope of the Tor Project's +efforts in that area. + +This is accomplished by utilizing helper sub-processes that +implement the necessary forward/reverse proxy servers that handle +the censorship circumvention, with a well defined and +standardized configuration and management interface. + +Any application code that implements the interfaces as specified +in this document will be able to use all spec compliant Pluggable +Transports. + + +## Requirements Notation + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL +NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in +[RFC2119]. + diff --git a/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md b/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md new file mode 100644 index 0000000..cce6748 --- /dev/null +++ b/spec/pt-spec/pluggable-transport-client-per-connection-arguments.md @@ -0,0 +1,38 @@ + +## Pluggable Transport Client Per-Connection Arguments + +Certain PT transport protocols require that the client provides +per-connection arguments when making outgoing connections. On +the server side, this is handled by the "ARGS" optional argument +as part of the "SMETHOD" message. + +On the client side, arguments are passed via the authentication +fields that are part of the SOCKS protocol. + +First the "=" formatted arguments MUST be escaped, +such that all backslash, equal sign, and semicolon characters +are escaped with a backslash. + +Second, all of the escaped are concatenated together. + +Example: + +shared-secret=rahasia;secrets-file=/tmp/blob + +Lastly the arguments are transmitted when making the outgoing +connection using the authentication mechanism specific to the +SOCKS protocol version. + +```text + - In the case of SOCKS 4, the concatenated argument list is + transmitted in the "USERID" field of the "CONNECT" request. + + - In the case of SOCKS 5, the parent process must negotiate + "Username/Password" authentication [RFC1929], and transmit + the arguments encoded in the "UNAME" and "PASSWD" fields. +``` + +If the encoded argument list is less than 255 bytes in +length, the "PLEN" field must be set to "1" and the "PASSWD" +field must contain a single NUL character. + diff --git a/spec/pt-spec/pluggable-transport-configuration-environment.md b/spec/pt-spec/pluggable-transport-configuration-environment.md new file mode 100644 index 0000000..61b6b01 --- /dev/null +++ b/spec/pt-spec/pluggable-transport-configuration-environment.md @@ -0,0 +1,255 @@ + +## Pluggable Transport Configuration Environment Variables + +All Pluggable Transport proxy instances are configured by their +parent process at launch time via a set of well defined +environment variables. + +The "TOR_PT_" prefix is used for namespacing reasons and does not +indicate any relations to Tor, except for the origins of this +specification. + + +### Common Environment Variables + +When launching either a client or server Pluggable Transport proxy, +the following common environment variables MUST be set. + +"TOR_PT_MANAGED_TRANSPORT_VER" + +Specifies the versions of the Pluggable Transport specification +the parent process supports, delimited by commas. All PTs MUST +accept any well-formed list, as long as a compatible version is +present. + +Valid versions MUST consist entirely of non-whitespace, +non-comma printable ASCII characters. + +The version of the Pluggable Transport specification as of this +document is "1". + +Example: + +TOR_PT_MANAGED_TRANSPORT_VER=1,1a,2b,this_is_a_valid_ver + +"TOR_PT_STATE_LOCATION" + +Specifies an absolute path to a directory where the PT is +allowed to store state that will be persisted across +invocations. The directory is not required to exist when +the PT is launched, however PT implementations SHOULD be +able to create it as required. + +PTs MUST only store files in the path provided, and MUST NOT +create or modify files elsewhere on the system. + +Example: + +TOR_PT_STATE_LOCATION=/var/lib/tor/pt_state/ + +"TOR_PT_EXIT_ON_STDIN_CLOSE" + +Specifies that the parent process will close the PT proxy's +standard input (stdin) stream to indicate that the PT proxy +should gracefully exit. + +PTs MUST NOT treat a closed stdin as a signal to terminate +unless this environment variable is set to "1". + +PTs SHOULD treat stdin being closed as a signal to gracefully +terminate if this environment variable is set to "1". + +Example: + +TOR_PT_EXIT_ON_STDIN_CLOSE=1 + +"TOR_PT_OUTBOUND_BIND_ADDRESS_V4" + +Specifies an IPv4 IP address that the PT proxy SHOULD use as source address for +outgoing IPv4 IP packets. This feature allows people with multiple network +interfaces to specify explicitly which interface they prefer the PT proxy to +use. + +If this value is unset or empty, the PT proxy MUST use the default source +address for outgoing connections. + +This setting MUST be ignored for connections to +loopback addresses (127.0.0.0/8). + +Example: + +TOR_PT_OUTBOUND_BIND_ADDRESS_V4=203.0.113.4 + +"TOR_PT_OUTBOUND_BIND_ADDRESS_V6" + +Specifies an IPv6 IP address that the PT proxy SHOULD use as source address for +outgoing IPv6 IP packets. This feature allows people with multiple network +interfaces to specify explicitly which interface they prefer the PT proxy to +use. + +If this value is unset or empty, the PT proxy MUST use the default source +address for outgoing connections. + +This setting MUST be ignored for connections to the loopback address ([::1]). + +IPv6 addresses MUST always be wrapped in square brackets. + +Example:: + +TOR_PT_OUTBOUND_BIND_ADDRESS_V6=[2001:db8::4] + + +### Pluggable Transport Client Environment Variables + +Client-side Pluggable Transport forward proxies are configured +via the following environment variables. + +"TOR_PT_CLIENT_TRANSPORTS" + +Specifies the PT protocols the client proxy should initialize, +as a comma separated list of PT names. + +PTs SHOULD ignore PT names that it does not recognize. + +Parent processes MUST set this environment variable when +launching a client-side PT proxy instance. + +Example: + +TOR_PT_CLIENT_TRANSPORTS=obfs2,obfs3,obfs4 + +"TOR_PT_PROXY" + +Specifies an upstream proxy that the PT MUST use when making +outgoing network connections. It is a URI [RFC3986] of the +format: + +://[[:][@]:. + +The "TOR_PT_PROXY" environment variable is OPTIONAL and +MUST be omitted if there is no need to connect via an +upstream proxy. + +Examples: + +```text + TOR_PT_PROXY=socks5://tor:test1234@198.51.100.1:8000 + TOR_PT_PROXY=socks4a://198.51.100.2:8001 + TOR_PT_PROXY=http://198.51.100.3:443 +``` + + +### Pluggable Transport Server Environment Variables + +Server-side Pluggable Transport reverse proxies are configured +via the following environment variables. + +"TOR_PT_SERVER_TRANSPORTS" + +Specifies the PT protocols the server proxy should initialize, +as a comma separated list of PT names. + +PTs SHOULD ignore PT names that it does not recognize. + +Parent processes MUST set this environment variable when +launching a server-side PT reverse proxy instance. + +Example: + +TOR_PT_SERVER_TRANSPORTS=obfs3,scramblesuit + +"TOR_PT_SERVER_TRANSPORT_OPTIONS" + +Specifies per-PT protocol configuration directives, as a +semicolon-separated list of : pairs, where +is a PT name and is a k=v string value with options +that are to be passed to the transport. + +Colons, semicolons, and backslashes MUST be +escaped with a backslash. + +If there are no arguments that need to be passed to any of +PT transport protocols, "TOR_PT_SERVER_TRANSPORT_OPTIONS" +MAY be omitted. + +Example: + +TOR_PT_SERVER_TRANSPORT_OPTIONS=scramblesuit:key=banana;automata:rule=110;automata:depth=3 + +```text + Will pass to 'scramblesuit' the parameter 'key=banana' and to + 'automata' the arguments 'rule=110' and 'depth=3'. + + "TOR_PT_SERVER_BINDADDR" +``` + +A comma separated list of - pairs, where is +a PT name and is the
: on which it +should listen for incoming client connections. + +The keys holding transport names MUST be in the same order as +they appear in "TOR_PT_SERVER_TRANSPORTS". + +The
MAY be a locally scoped address as long as port +forwarding is done externally. + +The
: combination MUST be an IP address +supported by `bind()`, and MUST NOT be a host name. + +Applications MUST NOT set more than one
: pair +per PT name. + +If there is no specific
: combination to be +configured for any transports, "TOR_PT_SERVER_BINDADDR" MAY +be omitted. + +Example: + +TOR_PT_SERVER_BINDADDR=obfs3-198.51.100.1:1984,scramblesuit-127.0.0.1:4891 + +"TOR_PT_ORPORT" + +Specifies the destination that the PT reverse proxy should forward +traffic to after transforming it as appropriate, as an +
:. + +Connections to the destination specified via "TOR_PT_ORPORT" +MUST only contain application payload. If the parent process +requires the actual source IP address of client connections +(or other metadata), it should set "TOR_PT_EXTENDED_SERVER_PORT" +instead. + +Example: + +TOR_PT_ORPORT=127.0.0.1:9001 + +"TOR_PT_EXTENDED_SERVER_PORT" + +Specifies the destination that the PT reverse proxy should +forward traffic to, via the Extended ORPort protocol [EXTORPORT] +as an
:. + +The Extended ORPort protocol allows the PT reverse proxy to +communicate per-connection metadata such as the PT name and +client IP address/port to the parent process. + +If the parent process does not support the ExtORPort protocol, +it MUST set "TOR_PT_EXTENDED_SERVER_PORT" to an empty string. + +Example: + +TOR_PT_EXTENDED_SERVER_PORT=127.0.0.1:4200 + +"TOR_PT_AUTH_COOKIE_FILE" + +Specifies an absolute filesystem path to the Extended ORPort +authentication cookie, required to communicate with the +Extended ORPort specified via "TOR_PT_EXTENDED_SERVER_PORT". + +If the parent process is not using the ExtORPort protocol for +incoming traffic, "TOR_PT_AUTH_COOKIE_FILE" MUST be omitted. + +Example: + +TOR_PT_AUTH_COOKIE_FILE=/var/lib/tor/extended_orport_auth_cookie + diff --git a/spec/pt-spec/pluggable-transport-naming.md b/spec/pt-spec/pluggable-transport-naming.md new file mode 100644 index 0000000..50ebcb3 --- /dev/null +++ b/spec/pt-spec/pluggable-transport-naming.md @@ -0,0 +1,13 @@ + +## Pluggable Transport Naming + +Pluggable Transport names serve as unique identifiers, and every +PT MUST have a unique name. + +PT names MUST be valid C identifiers. PT names MUST begin with +a letter or underscore, and the remaining characters MUST be +ASCII letters, numbers or underscores. No length limit is +imposted. + +PT names MUST satisfy the regular expression "[a-zA-Z_][a-zA-Z0-9_]*". + diff --git a/spec/pt-spec/pluggable-transport-shutdown.md b/spec/pt-spec/pluggable-transport-shutdown.md new file mode 100644 index 0000000..b2c0fcc --- /dev/null +++ b/spec/pt-spec/pluggable-transport-shutdown.md @@ -0,0 +1,32 @@ + +## Pluggable Transport Shutdown + +The recommended way for Pluggable Transport using applications and +Pluggable Transports to handle graceful shutdown is as follows. + +```text + - (Parent) Set "TOR_PT_EXIT_ON_STDIN_CLOSE" (3.2.1) when + launching the PT proxy, to indicate that stdin will be used + for graceful shutdown notification. + + - (Parent) When the time comes to terminate the PT proxy: + + 1. Close the PT proxy's stdin. + 2. Wait for a "reasonable" amount of time for the PT to exit. + 3. Attempt to use OS specific mechanisms to cause graceful + PT shutdown (eg: 'SIGTERM') + 4. Use OS specific mechanisms to force terminate the PT + (eg: 'SIGKILL', 'ProccessTerminate()'). + + - PT proxies SHOULD monitor stdin, and exit gracefully when + it is closed, if the parent supports that behavior. + + - PT proxies SHOULD handle OS specific mechanisms to gracefully + terminate (eg: Install a signal handler on 'SIGTERM' that + causes cleanup and a graceful shutdown if able). + + - PT proxies SHOULD attempt to detect when the parent has + terminated (eg: via detecting that its parent process ID has + changed on U*IX systems), and gracefully terminate. +``` + diff --git a/spec/pt-spec/pluggable-transport-to-parent-process-communication.md b/spec/pt-spec/pluggable-transport-to-parent-process-communication.md new file mode 100644 index 0000000..7f7de76 --- /dev/null +++ b/spec/pt-spec/pluggable-transport-to-parent-process-communication.md @@ -0,0 +1,294 @@ + +## Pluggable Transport To Parent Process Communication + +All Pluggable Transport Proxies communicate to the parent process +via writing NL-terminated lines to stdout. The line metaformat is: + + ::= + ::= | + ::= + ::= * + ::= | + ::= + ::= + ::= + +The parent process MUST ignore lines received from PT proxies with +unknown keywords. + + +### Common Messages + +When a PT proxy first starts up, it must determine which version +of the Pluggable Transports Specification to use to configure +itself. + +It does this via the "TOR_PT_MANAGED_TRANSPORT_VER" (3.2.1) +environment variable which contains all of the versions supported +by the application. + +Upon determining the version to use, or lack thereof, the PT +proxy responds with one of two messages. + +VERSION-ERROR + +The "VERSION-ERROR" message is used to signal that there was +no compatible Pluggable Transport Specification version +present in the "TOR_PT_MANAGED_TRANSPORT_VER" list. + +The SHOULD be set to "no-version" for +historical reasons but MAY be set to a useful error message +instead. + +PT proxies MUST terminate after outputting a "VERSION-ERROR" +message. + +Example: + +VERSION-ERROR no-version + +VERSION + +The "VERSION" message is used to signal the Pluggable Transport +Specification version (as in "TOR_PT_MANAGED_TRANSPORT_VER") +that the PT proxy will use to configure its transports and +communicate with the parent process. + +The version for the environment values and reply messages +specified by this document is "1". + +PT proxies MUST either report an error and terminate, or output +a "VERSION" message before moving on to client/server proxy +initialization and configuration. + +Example: + +VERSION 1 + +After version negotiation has been completed the PT proxy must +then validate that all of the required environment variables are +provided, and that all of the configuration values supplied are +well formed. + +At any point, if there is an error encountered related to +configuration supplied via the environment variables, it MAY +respond with an error message and terminate. + +ENV-ERROR + +The "ENV-ERROR" message is used to signal the PT proxy's +failure to parse the configuration environment variables (3.2). + +The SHOULD consist of a useful error message +that can be used to diagnose and correct the root cause of +the failure. + +PT proxies MUST terminate after outputting a "ENV-ERROR" +message. + +Example: + +ENV-ERROR No TOR_PT_AUTH_COOKIE_FILE when TOR_PT_EXTENDED_SERVER_PORT set + + +### Pluggable Transport Client Messages + +After negotiating the Pluggable Transport Specification version, +PT client proxies MUST first validate "TOR_PT_PROXY" (3.2.2) if +it is set, before initializing any transports. + +Assuming that an upstream proxy is provided, PT client proxies +MUST respond with a message indicating that the proxy is valid, +supported, and will be used OR a failure message. + +PROXY DONE + +The "PROXY DONE" message is used to signal the PT proxy's +acceptance of the upstream proxy specified by "TOR_PT_PROXY". + +PROXY-ERROR + +The "PROXY-ERROR" message is used to signal that the upstream +proxy is malformed/unsupported or otherwise unusable. + +PT proxies MUST terminate immediately after outputting a +"PROXY-ERROR" message. + +Example: + +PROXY-ERROR SOCKS 4 upstream proxies unsupported. + +After the upstream proxy (if any) is configured, PT clients then +iterate over the requested transports in "TOR_PT_CLIENT_TRANSPORTS" +and initialize the listeners. + +For each transport initialized, the PT proxy reports the listener +status back to the parent via messages to stdout. + +CMETHOD <'socks4','socks5'> + +The "CMETHOD" message is used to signal that a requested +PT transport has been launched, the protocol which the parent +should use to make outgoing connections, and the IP address +and port that the PT transport's forward proxy is listening on. + +Example: + +CMETHOD trebuchet socks5 127.0.0.1:19999 + +CMETHOD-ERROR + +The "CMETHOD-ERROR" message is used to signal that +requested PT transport was unable to be launched. + +Example: + +CMETHOD-ERROR trebuchet no rocks available + +Once all PT transports have been initialized (or have failed), the +PT proxy MUST send a final message indicating that it has finished +initializing. + +CMETHODS DONE + +The "CMETHODS DONE" message signals that the PT proxy has +finished initializing all of the transports that it is capable +of handling. + +Upon sending the "CMETHODS DONE" message, the PT proxy +initialization is complete. + +Notes: + +```text + - Unknown transports in "TOR_PT_CLIENT_TRANSPORTS" are ignored + entirely, and MUST NOT result in a "CMETHOD-ERROR" message. + Thus it is entirely possible for a given PT proxy to + immediately output "CMETHODS DONE". + + - Parent processes MUST handle "CMETHOD"/"CMETHOD-ERROR" + messages in any order, regardless of ordering in + "TOR_PT_CLIENT_TRANSPORTS". +``` + + +### Pluggable Transport Server Messages + +PT server reverse proxies iterate over the requested transports +in "TOR_PT_CLIENT_TRANSPORTS" and initialize the listeners. + +For each transport initialized, the PT proxy reports the listener +status back to the parent via messages to stdout. + +SMETHOD [options] + +The "SMETHOD" message is used to signal that a requested +PT transport has been launched, the protocol which will be +used to handle incoming connections, and the IP address and +port that clients should use to reach the reverse-proxy. + +If there is a specific provided for a given +PT transport via "TOR_PT_SERVER_BINDADDR", the transport +MUST be initialized using that as the server address. + +The OPTIONAL 'options' field is used to pass additional +per-transport information back to the parent process. + +The currently recognized 'options' are: + +ARGS:[=,]+[=] + +```text + The "ARGS" option is used to pass additional key/value + formatted information that clients will require to use + the reverse proxy. + + Equal signs and commas MUST be escaped with a backslash. + + Tor: The ARGS are included in the transport line of the + Bridge's extra-info document. + + Examples: + + SMETHOD trebuchet 198.51.100.1:19999 + SMETHOD rot_by_N 198.51.100.1:2323 ARGS:N=13 + + SMETHOD-ERROR +``` + +The "SMETHOD-ERROR" message is used to signal that +requested PT transport reverse proxy was unable to be +launched. + +Example: + +SMETHOD-ERROR trebuchet no cows available + +Once all PT transports have been initialized (or have failed), the +PT proxy MUST send a final message indicating that it has finished +initializing. + +SMETHODS DONE + +The "SMETHODS DONE" message signals that the PT proxy has +finished initializing all of the transports that it is capable +of handling. + +Upon sending the "SMETHODS DONE" message, the PT proxy +initialization is complete. + + +### Pluggable Transport Log Message + +This message is for a client or server PT to be able to signal back to the +parent process via stdout or stderr any log messages. + +A log message can be any kind of messages (human readable) that the PT +sends back so the parent process can gather information about what is going +on in the child process. It is not intended for the parent process to parse +and act accordingly but rather a message used for plain logging. + +For example, the tor daemon logs those messages at the Severity level and +sends them onto the control port using the PT_LOG (see control-spec.txt) +event so any third party can pick them up for debugging. + +The format of the message: + +LOG SEVERITY=Severity MESSAGE=Message + +The SEVERITY value indicate at which logging level the message applies. +The accepted values for are: error, warning, notice, info, debug + +The MESSAGE value is a human readable string formatted by the PT. The + contains the log message which can be a String or CString (see +section 2 in control-spec.txt). + +Example: + +LOG SEVERITY=debug MESSAGE="Connected to bridge A" + + +### Pluggable Transport Status Message + +This message is for a client or server PT to be able to signal back to the +parent process via stdout or stderr any status messages. + +The format of the message: + +STATUS TRANSPORT=Transport = [= ...] + +The TRANSPORT value indicates a hint on what the PT is such has the name or +the protocol used for instance. As an example, obfs4proxy would use +"obfs4". Thus, the Transport value can be anything the PT itself defines +and it can be a String or CString (see section 2 in control-spec.txt). + +The = values are specific to the PT and there has to be at least +one. They are messages that reflects the status that the PT wants to +report. can be a String or CString. + +Examples (fictional): + +STATUS TRANSPORT=obfs4 ADDRESS=198.51.100.123:1234 CONNECT=Success +STATUS TRANSPORT=obfs4 ADDRESS=198.51.100.222:2222 CONNECT=Failed FINGERPRINT= ERRSTR="Connection refused" +STATUS TRANSPORT=trebuchet ADDRESS=198.51.100.15:443 PERCENT=42 + diff --git a/spec/pt-spec/references.md b/spec/pt-spec/references.md new file mode 100644 index 0000000..6956c00 --- /dev/null +++ b/spec/pt-spec/references.md @@ -0,0 +1,22 @@ + +# References + +```text + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + + [RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., + Koblas, D., Jones, L., "SOCKS Protocol Version 5", + RFC 1928, March 1996. + + [EXTORPORT] Kadianakis, G., Mathewson, N., "Extended ORPort and + TransportControlPort", Tor Proposal 196, March 2012. + + [RFC3986] Berners-Lee, T., Fielding, R., Masinter, L., "Uniform + Resource Identifier (URI): Generic Syntax", RFC 3986, + January 2005. + + [RFC1929] Leech, M., "Username/Password Authentication for + SOCKS V5", RFC 1929, March 1996. +``` + diff --git a/spec/pt-spec/specification.md b/spec/pt-spec/specification.md new file mode 100644 index 0000000..0c10377 --- /dev/null +++ b/spec/pt-spec/specification.md @@ -0,0 +1,52 @@ + +# Specification + +Pluggable Transport proxies follow the following workflow +throughout their lifespan. + +```text + 1) Parent process sets the required environment values (3.2) + and launches the PT proxy as a sub-process (fork()/exec()). + + 2) The PT Proxy determines the versions of the PT specification + supported by the parent"TOR_PT_MANAGED_TRANSPORT_VER" (3.2.1) + + 2.1) If there are no compatible versions, the PT proxy + writes a "VERSION-ERROR" message (3.3.1) to stdout and + terminates. + + 2.2) If there is a compatible version, the PT proxy writes + a "VERSION" message (3.3.1) to stdout. + + 3) The PT Proxy parses the rest of the environment values. + + 3.1) If the environment values are malformed, or otherwise + invalid, the PT proxy writes a "ENV-ERROR" message + (3.3.1) to stdout and terminates. + + 3.2) Determining if it is a client side forward proxy or + a server side reverse proxy can be done via examining + the "TOR_PT_CLIENT_TRANSPORTS" and "TOR_PT_SERVER_TRANSPORTS" + environment variables. + + 4) (Client only) If there is an upstream proxy specified via + "TOR_PT_PROXY" (3.2.2), the PT proxy validates the URI + provided. + + 4.1) If the upstream proxy is unusable, the PT proxy writes + a "PROXY-ERROR" message (3.3.2) to stdout and + terminates. + + 4.2) If there is a supported and well-formed upstream proxy + the PT proxy writes a "PROXY DONE" message (3.3.2) to + stdout. + + 5) The PT Proxy initializes the transports and reports the + status via stdout (3.3.2, 3.3.3) + + 6) The PT Proxy forwards and transforms traffic as appropriate. + + 7) Upon being signaled to terminate by the parent process (3.4), + the PT Proxy gracefully shuts down. +``` + diff --git a/spec/rend-spec-v3-intro.md b/spec/rend-spec-v3-intro.md new file mode 100644 index 0000000..5881eaa --- /dev/null +++ b/spec/rend-spec-v3-intro.md @@ -0,0 +1,98 @@ +Tor Rendezvous Specification - Version 3 + +This document specifies how the hidden service version 3 protocol works. This +text used to be proposal 224-rend-spec-ng.txt. + +Table of contents: + +```text + 0. Hidden services: overview and preliminaries. + 0.1. Improvements over previous versions. + 0.2. Notation and vocabulary + 0.3. Cryptographic building blocks + 0.4. Protocol building blocks [BUILDING-BLOCKS] + 0.5. Assigned relay cell types + 0.6. Acknowledgments + 1. Protocol overview + 1.1. View from 10,000 feet + 1.2. In more detail: naming hidden services [NAMING] + 1.3. In more detail: Access control [IMD:AC] + 1.4. In more detail: Distributing hidden service descriptors. [IMD:DIST] + 1.5. In more detail: Scaling to multiple hosts + 1.6. In more detail: Backward compatibility with older hidden service + 1.7. In more detail: Keeping crypto keys offline + 1.8. In more detail: Encryption Keys And Replay Resistance + 1.9. In more detail: A menagerie of keys + 1.9.1. In even more detail: Client authorization [CLIENT-AUTH] + 2. Generating and publishing hidden service descriptors [HSDIR] + 2.1. Deriving blinded keys and subcredentials [SUBCRED] + 2.2. Locating, uploading, and downloading hidden service descriptors + 2.2.1. Dividing time into periods [TIME-PERIODS] + 2.2.2. When to publish a hidden service descriptor [WHEN-HSDESC] + 2.2.3. Where to publish a hidden service descriptor [WHERE-HSDESC] + 2.2.4. Using time periods and SRVs to fetch/upload HS descriptors + 2.2.5. Expiring hidden service descriptors [EXPIRE-DESC] + 2.2.6. URLs for anonymous uploading and downloading + 2.3. Publishing shared random values [PUB-SHAREDRANDOM] + 2.3.1. Client behavior in the absence of shared random values + 2.3.2. Hidden services and changing shared random values + 2.4. Hidden service descriptors: outer wrapper [DESC-OUTER] + 2.5. Hidden service descriptors: encryption format [HS-DESC-ENC] + 2.5.1. First layer of encryption [HS-DESC-FIRST-LAYER] + 2.5.1.1. First layer encryption logic + 2.5.1.2. First layer plaintext format + 2.5.1.3. Client behavior + 2.5.1.4. Obfuscating the number of authorized clients + 2.5.2. Second layer of encryption [HS-DESC-SECOND-LAYER] + 2.5.2.1. Second layer encryption keys + 2.5.2.2. Second layer plaintext format + 2.5.3. Deriving hidden service descriptor encryption keys [HS-DESC-ENCRYPTION-KEYS] + 3. The introduction protocol [INTRO-PROTOCOL] + 3.1. Registering an introduction point [REG_INTRO_POINT] + 3.1.1. Extensible ESTABLISH_INTRO protocol. [EST_INTRO] + 3.1.1.1. Denial-of-Server Defense Extension. [EST_INTRO_DOS_EXT] + 3.1.2. Registering an introduction point on a legacy Tor node [LEGACY_EST_INTRO] + 3.1.3. Acknowledging establishment of introduction point [INTRO_ESTABLISHED] + 3.2. Sending an INTRODUCE1 cell to the introduction point. [SEND_INTRO1] + 3.2.1. INTRODUCE1 cell format [FMT_INTRO1] + 3.2.2. INTRODUCE_ACK cell format. [INTRO_ACK] + 3.3. Processing an INTRODUCE2 cell at the hidden service. [PROCESS_INTRO2] + 3.3.1. Introduction handshake encryption requirements [INTRO-HANDSHAKE-REQS] + 3.3.2. Example encryption handshake: ntor with extra data [NTOR-WITH-EXTRA-DATA] + 3.4. Authentication during the introduction phase. [INTRO-AUTH] + 3.4.1. Ed25519-based authentication. + 4. The rendezvous protocol + 4.1. Establishing a rendezvous point [EST_REND_POINT] + 4.2. Joining to a rendezvous point [JOIN_REND] + 4.2.1. Key expansion + 4.3. Using legacy hosts as rendezvous points + 5. Encrypting data between client and host + 6. Encoding onion addresses [ONIONADDRESS] + 7. Open Questions: + +-1. Draft notes +``` + +This document describes a proposed design and specification for +hidden services in Tor version 0.2.5.x or later. It's a replacement +for the current rend-spec.txt, rewritten for clarity and for improved +design. + +Look for the string "TODO" below: it describes gaps or uncertainties +in the design. + +Change history: + +2013-11-29: Proposal first numbered. Some TODO and XXX items remain. + +2014-01-04: Clarify some unclear sections. + +2014-01-21: Fix a typo. + +```text + 2014-02-20: Move more things to the revised certificate format in the + new updated proposal 220. + + 2015-05-26: Fix two typos. +``` + diff --git a/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md b/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md new file mode 100644 index 0000000..586d9a5 --- /dev/null +++ b/spec/rend-spec-v3/deriving-blinded-keys-subcredentials-subcred.md @@ -0,0 +1,405 @@ + +## Deriving blinded keys and subcredentials [SUBCRED] + +In each time period (see [TIME-PERIODS] for a definition of time +periods), a hidden service host uses a different blinded private key +to sign its directory information, and clients use a different +blinded public key as the index for fetching that information. + +For a candidate for a key derivation method, see Appendix [KEYBLIND]. + +Additionally, clients and hosts derive a subcredential for each +period. Knowledge of the subcredential is needed to decrypt hidden +service descriptors for each period and to authenticate with the +hidden service host in the introduction process. Unlike the +credential, it changes each period. Knowing the subcredential, even +in combination with the blinded private key, does not enable the +hidden service host to derive the main credential--therefore, it is +safe to put the subcredential on the hidden service host while +leaving the hidden service's private key offline. + +The subcredential for a period is derived as: + +N_hs_subcred = H("subcredential" | N_hs_cred | blinded-public-key). + +In the above formula, credential corresponds to: + +N_hs_cred = H("credential" | public-identity-key) + +where public-identity-key is the public identity master key of the hidden +service. + +```text +2.2. Locating, uploading, and downloading hidden service descriptors + [HASHRING] +``` + +To avoid attacks where a hidden service's descriptor is easily +targeted for censorship, we store them at different directories over +time, and use shared random values to prevent those directories from +being predictable far in advance. + +Which Tor servers hosts a hidden service depends on: + +```text + * the current time period, + * the daily subcredential, + * the hidden service directories' public keys, + * a shared random value that changes in each time period, + shared_random_value. + * a set of network-wide networkstatus consensus parameters. + (Consensus parameters are integer values voted on by authorities + and published in the consensus documents, described in + dir-spec.txt, section 3.3.) + + Below we explain in more detail. +``` + + +### Dividing time into periods [TIME-PERIODS] + +To prevent a single set of hidden service directory from becoming a +target by adversaries looking to permanently censor a hidden service, +hidden service descriptors are uploaded to different locations that +change over time. + +The length of a "time period" is controlled by the consensus +parameter 'hsdir-interval', and is a number of minutes between 30 and +14400 (10 days). The default time period length is 1440 (one day). + +Time periods start at the Unix epoch (Jan 1, 1970), and are computed by +taking the number of minutes since the epoch and dividing by the time +period. However, we want our time periods to start at a regular offset +from the SRV voting schedule, so we subtract a "rotation time offset" +of 12 voting periods from the number of minutes since the epoch, before +dividing by the time period (effectively making "our" epoch start at Jan +1, 1970 12:00UTC when the voting period is 1 hour.) + +Example: If the current time is 2016-04-13 11:15:01 UTC, making the seconds +since the epoch 1460546101, and the number of minutes since the epoch +24342435. We then subtract the "rotation time offset" of 12*60 minutes from +the minutes since the epoch, to get 24341715. If the current time period +length is 1440 minutes, by doing the division we see that we are currently +in time period number 16903. + +Specifically, time period #16903 began 16903*1440*60 + (12*60*60) seconds +after the epoch, at 2016-04-12 12:00 UTC, and ended at 16904*1440*60 + +(12*60*60) seconds after the epoch, at 2016-04-13 12:00 UTC. + + +### When to publish a hidden service descriptor [WHEN-HSDESC] + +Hidden services periodically publish their descriptor to the responsible +HSDirs. The set of responsible HSDirs is determined as specified in +[WHERE-HSDESC]. + +Specifically, every time a hidden service publishes its descriptor, it also +sets up a timer for a random time between 60 minutes and 120 minutes in the +future. When the timer triggers, the hidden service needs to publish its +descriptor again to the responsible HSDirs for that time period. +[TODO: Control republish period using a consensus parameter?] + + +#### Overlapping descriptors + +Hidden services need to upload multiple descriptors so that they can be +reachable to clients with older or newer consensuses than them. Services +need to upload their descriptors to the HSDirs _before_ the beginning of +each upcoming time period, so that they are readily available for clients to +fetch them. Furthermore, services should keep uploading their old descriptor +even after the end of a time period, so that they can be reachable by +clients that still have consensuses from the previous time period. + +Hence, services maintain two active descriptors at every point. Clients on +the other hand, don't have a notion of overlapping descriptors, and instead +always download the descriptor for the current time period and shared random +value. It's the job of the service to ensure that descriptors will be +available for all clients. See section [FETCHUPLOADDESC] for how this is +achieved. + +[TODO: What to do when we run multiple hidden services in a single host?] + + +### Where to publish a hidden service descriptor [WHERE-HSDESC] + +This section specifies how the HSDir hash ring is formed at any given +time. Whenever a time value is needed (e.g. to get the current time period +number), we assume that clients and services use the valid-after time from +their latest live consensus. + +The following consensus parameters control where a hidden service +descriptor is stored; + +```text + hsdir_n_replicas = an integer in range [1,16] with default value 2. + hsdir_spread_fetch = an integer in range [1,128] with default value 3. + hsdir_spread_store = an integer in range [1,128] with default value 4. + (Until 0.3.2.8-rc, the default was 3.) +``` + +To determine where a given hidden service descriptor will be stored +in a given period, after the blinded public key for that period is +derived, the uploading or downloading party calculates: + +```text + for replicanum in 1...hsdir_n_replicas: + hs_service_index(replicanum) = H("store-at-idx" | + blinded_public_key | + INT_8(replicanum) | + INT_8(period_length) | + INT_8(period_num) ) +``` + +where blinded_public_key is specified in section [KEYBLIND], period_length +is the length of the time period in minutes, and period_num is calculated +using the current consensus "valid-after" as specified in section +[TIME-PERIODS]. + +Then, for each node listed in the current consensus with the HSDir flag, +we compute a directory index for that node as: + +```text + hs_relay_index(node) = H("node-idx" | node_identity | + shared_random_value | + INT_8(period_num) | + INT_8(period_length) ) +``` + +where shared_random_value is the shared value generated by the authorities +in section [PUB-SHAREDRANDOM], and node_identity is the ed25519 identity +key of the node. + +Finally, for replicanum in 1...hsdir_n_replicas, the hidden service +host uploads descriptors to the first hsdir_spread_store nodes whose +indices immediately follow hs_service_index(replicanum). If any of those +nodes have already been selected for a lower-numbered replica of the +service, any nodes already chosen are disregarded (i.e. skipped over) +when choosing a replica's hsdir_spread_store nodes. + +When choosing an HSDir to download from, clients choose randomly from +among the first hsdir_spread_fetch nodes after the indices. (Note +that, in order to make the system better tolerate disappearing +HSDirs, hsdir_spread_fetch may be less than hsdir_spread_store.) +Again, nodes from lower-numbered replicas are disregarded when +choosing the spread for a replica. + + +### Using time periods and SRVs to fetch/upload HS descriptors [FETCHUPLOADDESC] + +Hidden services and clients need to make correct use of time periods (TP) +and shared random values (SRVs) to successfully fetch and upload +descriptors. Furthermore, to avoid problems with skewed clocks, both clients +and services use the 'valid-after' time of a live consensus as a way to take +decisions with regards to uploading and fetching descriptors. By using the +consensus times as the ground truth here, we minimize the desynchronization +of clients and services due to system clock. Whenever time-based decisions +are taken in this section, assume that they are consensus times and not +system times. + +As [PUB-SHAREDRANDOM] specifies, consensuses contain two shared random +values (the current one and the previous one). Hidden services and clients +are asked to match these shared random values with descriptor time periods +and use the right SRV when fetching/uploading descriptors. This section +attempts to precisely specify how this works. + +Let's start with an illustration of the system: + +```text + +------------------------------------------------------------------+ + | | + | 00:00 12:00 00:00 12:00 00:00 12:00 | + | SRV#1 TP#1 SRV#2 TP#2 SRV#3 TP#3 | + | | + | $==========|-----------$===========|-----------$===========| | + | | + | | + +------------------------------------------------------------------+ + + Legend: [TP#1 = Time Period #1] + [SRV#1 = Shared Random Value #1] + ["$" = descriptor rotation moment] +``` + + +#### Client behavior for fetching descriptors [CLIENTFETCH] + +And here is how clients use TPs and SRVs to fetch descriptors: + +Clients always aim to synchronize their TP with SRV, so they always want to +use TP#N with SRV#N: To achieve this wrt time periods, clients always use +the current time period when fetching descriptors. Now wrt SRVs, if a client +is in the time segment between a new time period and a new SRV (i.e. the +segments drawn with "-") it uses the current SRV, else if the client is in a +time segment between a new SRV and a new time period (i.e. the segments +drawn with "="), it uses the previous SRV. + +Example: + ++------------------------------------------------------------------+ +| | +| 00:00 12:00 00:00 12:00 00:00 12:00 | +| SRV#1 TP#1 SRV#2 TP#2 SRV#3 TP#3 | +| | +| $==========|-----------$===========|-----------$===========| | +| ^ ^ | +| C1 C2 | ++------------------------------------------------------------------+ + +If a client (C1) is at 13:00 right after TP#1, then it will use TP#1 and +SRV#1 for fetching descriptors. Also, if a client (C2) is at 01:00 right +after SRV#2, it will still use TP#1 and SRV#1. + + +#### Service behavior for uploading descriptors [SERVICEUPLOAD] + +As discussed above, services maintain two active descriptors at any time. We +call these the "first" and "second" service descriptors. Services rotate +their descriptor every time they receive a consensus with a valid_after time +past the next SRV calculation time. They rotate their descriptors by +discarding their first descriptor, pushing the second descriptor to the +first, and rebuilding their second descriptor with the latest data. + +Services like clients also employ a different logic for picking SRV and TP +values based on their position in the graph above. Here is the logic: + + +##### First descriptor upload logic [FIRSTDESCUPLOAD] + +Here is the service logic for uploading its first descriptor: + +When a service is in the time segment between a new time period a new SRV +(i.e. the segments drawn with "-"), it uses the previous time period and +previous SRV for uploading its first descriptor: that's meant to cover +for clients that have a consensus that is still in the previous time period. + +Example: Consider in the above illustration that the service is at 13:00 +right after TP#1. It will upload its first descriptor using TP#0 and SRV#0. +So if a client still has a 11:00 consensus it will be able to access it +based on the client logic above. + +Now if a service is in the time segment between a new SRV and a new time +period (i.e. the segments drawn with "=") it uses the current time period +and the previous SRV for its first descriptor: that's meant to cover clients +with an up-to-date consensus in the same time period as the service. + +Example: + ++------------------------------------------------------------------+ +| | +| 00:00 12:00 00:00 12:00 00:00 12:00 | +| SRV#1 TP#1 SRV#2 TP#2 SRV#3 TP#3 | +| | +| $==========|-----------$===========|-----------$===========| | +| ^ | +| S | ++------------------------------------------------------------------+ + +Consider that the service is at 01:00 right after SRV#2: it will upload its +first descriptor using TP#1 and SRV#1. + + +##### Second descriptor upload logic [SECONDDESCUPLOAD] + +Here is the service logic for uploading its second descriptor: + +When a service is in the time segment between a new time period a new SRV +(i.e. the segments drawn with "-"), it uses the current time period and +current SRV for uploading its second descriptor: that's meant to cover for +clients that have an up-to-date consensus on the same TP as the service. + +Example: Consider in the above illustration that the service is at 13:00 +right after TP#1: it will upload its second descriptor using TP#1 and SRV#1. + +Now if a service is in the time segment between a new SRV and a new time +period (i.e. the segments drawn with "=") it uses the next time period and +the current SRV for its second descriptor: that's meant to cover clients +with a newer consensus than the service (in the next time period). + +Example: + ++------------------------------------------------------------------+ +| | +| 00:00 12:00 00:00 12:00 00:00 12:00 | +| SRV#1 TP#1 SRV#2 TP#2 SRV#3 TP#3 | +| | +| $==========|-----------$===========|-----------$===========| | +| ^ | +| S | ++------------------------------------------------------------------+ + +Consider that the service is at 01:00 right after SRV#2: it will upload its +second descriptor using TP#2 and SRV#2. + + +#### Directory behavior for handling descriptor uploads [DIRUPLOAD] + +Upon receiving a hidden service descriptor publish request, directories MUST +check the following: + +```text + * The outer wrapper of the descriptor can be parsed according to + [DESC-OUTER] + * The version-number of the descriptor is "3" + * If the directory has already cached a descriptor for this hidden service, + the revision-counter of the uploaded descriptor must be greater than the + revision-counter of the cached one + * The descriptor signature is valid +``` + +If any of these basic validity checks fails, the directory MUST reject the +descriptor upload. + +NOTE: Even if the descriptor passes the checks above, its first and second +layers could still be invalid: directories cannot validate the encrypted +layers of the descriptor, as they do not have access to the public key of the +service (required for decrypting the first layer of encryption), or the +necessary client credentials (for decrypting the second layer). + + +### Expiring hidden service descriptors [EXPIRE-DESC] + +Hidden services set their descriptor's "descriptor-lifetime" field to 180 +minutes (3 hours). Hidden services ensure that their descriptor will remain +valid in the HSDir caches, by republishing their descriptors periodically as +specified in [WHEN-HSDESC]. + +Hidden services MUST also keep their introduction circuits alive for as long +as descriptors including those intro points are valid (even if that's after +the time period has changed). + + +### URLs for anonymous uploading and downloading + +Hidden service descriptors conforming to this specification are uploaded +with an HTTP POST request to the URL /tor/hs//publish relative to +the hidden service directory's root, and downloaded with an HTTP GET +request for the URL /tor/hs// where is a base64 encoding of +the hidden service's blinded public key and is the protocol +version which is "3" in this case. + +These requests must be made anonymously, on circuits not used for +anything else. + + +### Client-side validation of onion addresses + +When a Tor client receives a prop224 onion address from the user, it +MUST first validate the onion address before attempting to connect or +fetch its descriptor. If the validation fails, the client MUST +refuse to connect. + +As part of the address validation, Tor clients should check that the +underlying ed25519 key does not have a torsion component. If Tor accepted +ed25519 keys with torsion components, attackers could create multiple +equivalent onion addresses for a single ed25519 key, which would map to the +same service. We want to avoid that because it could lead to phishing +attacks and surprising behaviors (e.g. imagine a browser plugin that blocks +onion addresses, but could be bypassed using an equivalent onion address +with a torsion component). + +The right way for clients to detect such fraudulent addresses (which should +only occur malevolently and never naturally) is to extract the ed25519 +public key from the onion address and multiply it by the ed25519 group order +and ensure that the result is the ed25519 identity element. For more +details, please see [TORSION-REFS]. + diff --git a/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md b/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md new file mode 100644 index 0000000..674f584 --- /dev/null +++ b/spec/rend-spec-v3/encoding-onion-addresses-onionaddress.md @@ -0,0 +1,27 @@ + +# Encoding onion addresses [ONIONADDRESS] + +The onion address of a hidden service includes its identity public key, a +version field and a basic checksum. All this information is then base32 +encoded as shown below: + +```text + onion_address = base32(PUBKEY | CHECKSUM | VERSION) + ".onion" + CHECKSUM = H(".onion checksum" | PUBKEY | VERSION)[:2] + + where: + - PUBKEY is the 32 bytes ed25519 master pubkey of the hidden service. + - VERSION is a one byte version field (default value '\x03') + - ".onion checksum" is a constant string + - CHECKSUM is truncated to two bytes before inserting it in onion_address + + Here are a few example addresses: + + pg6mmjiyjmcrsslvykfwnntlaru7p5svn6y2ymmju6nubxndf4pscryd.onion + sp3k262uwy4r2k3ycr5awluarykdpag6a7y33jxop4cs2lu5uz5sseqd.onion + xa4r2iadxm55fbnqgwwi5mymqdcofiu3w6rpbtqn7b2dyn7mgwj64jyd.onion +``` + +For more information about this encoding, please see our discussion thread +at [ONIONADDRESS-REFS]. + diff --git a/spec/rend-spec-v3/encrypting-data-between-client-host.md b/spec/rend-spec-v3/encrypting-data-between-client-host.md new file mode 100644 index 0000000..41cb1fa --- /dev/null +++ b/spec/rend-spec-v3/encrypting-data-between-client-host.md @@ -0,0 +1,12 @@ + +# Encrypting data between client and host + +A successfully completed handshake, as embedded in the +INTRODUCE/RENDEZVOUS cells, gives the client and hidden service host +a shared set of keys Kf, Kb, Df, Db, which they use for sending +end-to-end traffic encryption and authentication as in the regular +Tor relay encryption protocol, applying encryption with these keys +before other encryption, and decrypting with these keys before other +decryption. The client encrypts with Kf and decrypts with Kb; the +service host does the opposite. + diff --git a/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md b/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md new file mode 100644 index 0000000..403ef31 --- /dev/null +++ b/spec/rend-spec-v3/generating-publishing-hidden-service-descriptors.md @@ -0,0 +1,8 @@ + +# Generating and publishing hidden service descriptors [HSDIR] + +Hidden service descriptors follow the same metaformat as other Tor +directory objects. They are published anonymously to Tor servers with the +HSDir flag, HSDir=2 protocol version and tor version >= 0.3.0.8 (because a +bug was fixed in this version). + diff --git a/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md b/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md new file mode 100644 index 0000000..8c6982b --- /dev/null +++ b/spec/rend-spec-v3/hidden-service-descriptors-encryption-format-hs-desc-enc.md @@ -0,0 +1,446 @@ + +## Hidden service descriptors: encryption format [HS-DESC-ENC] + +Hidden service descriptors are protected by two layers of encryption. +Clients need to decrypt both layers to connect to the hidden service. + +The first layer of encryption provides confidentiality against entities who +don't know the public key of the hidden service (e.g. HSDirs), while the +second layer of encryption is only useful when client authorization is enabled +and protects against entities that do not possess valid client credentials. + + +### First layer of encryption [HS-DESC-FIRST-LAYER] + +The first layer of HS descriptor encryption is designed to protect +descriptor confidentiality against entities who don't know the public +identity key of the hidden service. + + +#### First layer encryption logic + +The encryption keys and format for the first layer of encryption are +generated as specified in [HS-DESC-ENCRYPTION-KEYS] with customization +parameters: + +```text + SECRET_DATA = blinded-public-key + STRING_CONSTANT = "hsdir-superencrypted-data" +``` + +The encryption scheme in [HS-DESC-ENCRYPTION-KEYS] uses the service +credential which is derived from the public identity key (see [SUBCRED]) to +ensure that only entities who know the public identity key can decrypt the +first descriptor layer. + +The ciphertext is placed on the "superencrypted" field of the descriptor. + +Before encryption the plaintext is padded with NUL bytes to the nearest +multiple of 10k bytes. + + +#### First layer plaintext format + +After clients decrypt the first layer of encryption, they need to parse the +plaintext to get to the second layer ciphertext which is contained in the +"encrypted" field. + +If client auth is enabled, the hidden service generates a fresh +descriptor_cookie key (`N_hs_desc_enc`, 32 random bytes) and encrypts +it using each authorized client's identity x25519 key. Authorized +clients can use the descriptor cookie (`N_hs_desc_enc`) to decrypt +the second (inner) layer of encryption. Our encryption scheme +requires the hidden service to also generate an ephemeral x25519 +keypair for each new descriptor. + +If client auth is disabled, fake data is placed in each of the fields below +to obfuscate whether client authorization is enabled. + +Here are all the supported fields: + +"desc-auth-type" SP type NL + +[Exactly once] + +```text + This field contains the type of authorization used to protect the + descriptor. The only recognized type is "x25519" and specifies the + encryption scheme described in this section. + + If client authorization is disabled, the value here should be "x25519". + + "desc-auth-ephemeral-key" SP KP_hs_desc_ephem NL + + [Exactly once] + + This field contains `KP_hss_desc_enc`, an ephemeral x25519 public + key generated by the hidden service and encoded in base64. The key + is used by the encryption scheme below. + + If client authorization is disabled, the value here should be a fresh + x25519 pubkey that will remain unused. + + "auth-client" SP client-id SP iv SP encrypted-cookie + + [At least once] + + When client authorization is enabled, the hidden service inserts an + "auth-client" line for each of its authorized clients. If client + authorization is disabled, the fields here can be populated with random + data of the right size (that's 8 bytes for 'client-id', 16 bytes for 'iv' + and 16 bytes for 'encrypted-cookie' all encoded with base64). + + When client authorization is enabled, each "auth-client" line + contains the descriptor cookie `N_hs_desc_enc` encrypted to each + individual client. We assume that each authorized client possesses + a pre-shared x25519 keypair (`KP_hsc_desc_enc`) which is used to + decrypt the descriptor cookie. + + We now describe the descriptor cookie encryption scheme. Here is what + the hidden service computes: + + SECRET_SEED = x25519(KS_hs_desc_ephem, KP_hsc_desc_enc) + KEYS = KDF(N_hs_subcred | SECRET_SEED, 40) + CLIENT-ID = fist 8 bytes of KEYS + COOKIE-KEY = last 32 bytes of KEYS + + Here is a description of the fields in the "auth-client" line: + + - The "client-id" field is CLIENT-ID from above encoded in base64. + + - The "iv" field is 16 random bytes encoded in base64. + + - The "encrypted-cookie" field contains the descriptor cookie ciphertext + as follows and is encoded in base64: + encrypted-cookie = STREAM(iv, COOKIE-KEY) XOR N_hs_desc_enc. + + See section [FIRST-LAYER-CLIENT-BEHAVIOR] for the client-side logic of + how to decrypt the descriptor cookie. + + "encrypted" NL encrypted-string + + [Exactly once] + + An encrypted blob containing the second layer ciphertext, whose format is + discussed in [HS-DESC-SECOND-LAYER] below. The blob is base64 encoded + and enclosed in -----BEGIN MESSAGE---- and ----END MESSAGE---- wrappers. + + Compatibility note: The C Tor implementation does not include a final + newline when generating this first-layer-plaintext section; other + implementations MUST accept this section even if it is missing its final + newline. Other implementations MAY generate this section without a final + newline themselves, to avoid being distinguishable from C tor. +``` + + +#### Client behavior [FIRST-LAYER-CLIENT-BEHAVIOR] + +```text + The goal of clients at this stage is to decrypt the "encrypted" field as + described in [HS-DESC-SECOND-LAYER]. + + If client authorization is enabled, authorized clients need to extract the + descriptor cookie to proceed with decryption of the second layer as + follows: + + An authorized client parsing the first layer of an encrypted descriptor, + extracts the ephemeral key from "desc-auth-ephemeral-key" and calculates + CLIENT-ID and COOKIE-KEY as described in the section above using their + x25519 private key. The client then uses CLIENT-ID to find the right + "auth-client" field which contains the ciphertext of the descriptor + cookie. The client then uses COOKIE-KEY and the iv to decrypt the + descriptor_cookie, which is used to decrypt the second layer of descriptor + encryption as described in [HS-DESC-SECOND-LAYER]. +``` + + +#### Hiding client authorization data + +```text + Hidden services should avoid leaking whether client authorization is + enabled or how many authorized clients there are. + + Hence even when client authorization is disabled, the hidden service adds + fake "desc-auth-type", "desc-auth-ephemeral-key" and "auth-client" lines to + the descriptor, as described in [HS-DESC-FIRST-LAYER]. + + The hidden service also avoids leaking the number of authorized clients by + adding fake "auth-client" entries to its descriptor. Specifically, + descriptors always contain a number of authorized clients that is a + multiple of 16 by adding fake "auth-client" entries if needed. + [XXX consider randomization of the value 16] + + Clients MUST accept descriptors with any number of "auth-client" lines as + long as the total descriptor size is within the max limit of 50k (also + controlled with a consensus parameter). +``` + + +### Second layer of encryption [HS-DESC-SECOND-LAYER] + +The second layer of descriptor encryption is designed to protect descriptor +confidentiality against unauthorized clients. If client authorization is +enabled, it's encrypted using the descriptor_cookie, and contains needed +information for connecting to the hidden service, like the list of its +introduction points. + +If client authorization is disabled, then the second layer of HS encryption +does not offer any additional security, but is still used. + + +#### Second layer encryption keys + +The encryption keys and format for the second layer of encryption are +generated as specified in [HS-DESC-ENCRYPTION-KEYS] with customization +parameters as follows: + +```text + SECRET_DATA = blinded-public-key | descriptor_cookie + STRING_CONSTANT = "hsdir-encrypted-data" + + If client authorization is disabled the 'descriptor_cookie' field is left blank. + + The ciphertext is placed on the "encrypted" field of the descriptor. +``` + + +#### Second layer plaintext format + +After decrypting the second layer ciphertext, clients can finally learn the +list of intro points etc. The plaintext has the following format: + +"create2-formats" SP formats NL + +[Exactly once] + +```text + A space-separated list of integers denoting CREATE2 cell HTYPEs + (handshake types) that the server recognizes. Must include at least + ntor as described in tor-spec.txt. See tor-spec section 5.1 for a list + of recognized handshake types. + + "intro-auth-required" SP types NL + + [At most once] + + A space-separated list of introduction-layer authentication types; see + section [INTRO-AUTH] for more info. A client that does not support at + least one of these authentication types will not be able to contact the + host. Recognized types are: 'ed25519'. + + "single-onion-service" + + [None or at most once] + + If present, this line indicates that the service is a Single Onion + Service (see prop260 for more details about that type of service). This + field has been introduced in 0.3.0 meaning 0.2.9 service don't include + this. + + Followed by zero or more introduction points as follows (see section + [NUM_INTRO_POINT] below for accepted values): + + "introduction-point" SP link-specifiers NL + + [Exactly once per introduction point at start of introduction + point section] + + The link-specifiers is a base64 encoding of a link specifier + block in the format described in [BUILDING-BLOCKS] above. + + As of 0.4.1.1-alpha, services include both IPv4 and IPv6 link + specifiers in descriptors. All available addresses SHOULD be + included in the descriptor, regardless of the address that the + onion service actually used to connect/extend to the intro + point. + + The client SHOULD NOT reject any LSTYPE fields which it doesn't + recognize; instead, it should use them verbatim in its EXTEND + request to the introduction point. + + The client SHOULD perform the basic validity checks on the link + specifiers in the descriptor, described in `tor-spec.txt` + section 5.1.2. These checks SHOULD NOT leak + detailed information about the client's version, configuration, + or consensus. (See 3.3 for service link specifier handling.) + + When connecting to the introduction point, the client SHOULD send + this list of link specifiers verbatim, in the same order as given + here. + + The client MAY reject the list of link specifiers if it is + inconsistent with relay information from the directory, but SHOULD + NOT modify it. + + "onion-key" SP "ntor" SP key NL + + [Exactly once per introduction point] + + The key is a base64 encoded curve25519 public key which is the onion + key of the introduction point Tor node used for the ntor handshake + when a client extends to it. + + "onion-key" SP KeyType SP key.. NL + + [Any number of times] + + Implementations should accept other types of onion keys using this + syntax (where "KeyType" is some string other than "ntor"); + unrecognized key types should be ignored. + + "auth-key" NL certificate NL + + [Exactly once per introduction point] + + The certificate is a proposal 220 certificate wrapped in + "-----BEGIN ED25519 CERT-----". It contains the introduction + point authentication key (`KP_hs_ipt_sid`), signed by + the descriptor signing key (`KP_hs_desc_sign`). The + certificate type must be [09], and the signing key extension + is mandatory. + + NOTE: This certificate was originally intended to be + constructed the other way around: the signing and signed keys + are meant to be reversed. However, C tor implemented it + backwards, and other implementations now need to do the same + in order to conform. (Since this section is inside the + descriptor, which is _already_ signed by `KP_hs_desc_sign`, + the verification aspect of this certificate serves no point in + its current form.) + + "enc-key" SP "ntor" SP key NL + + [Exactly once per introduction point] + + The key is a base64 encoded curve25519 public key used to encrypt + the introduction request to service. (`KP_hss_ntor`) + + "enc-key" SP KeyType SP key.. NL + + [Any number of times] + + Implementations should accept other types of onion keys using this + syntax (where "KeyType" is some string other than "ntor"); + unrecognized key types should be ignored. + + "enc-key-cert" NL certificate NL + + [Exactly once per introduction point] + + Cross-certification of the encryption key using the descriptor + signing key. + + For "ntor" keys, certificate is a proposal 220 certificate + wrapped in "-----BEGIN ED25519 CERT-----" armor. The subject + key is the the ed25519 equivalent of a curve25519 public + encryption key (`KP_hss_ntor`), with the ed25519 key + derived using the process in proposal 228 appendix A. The + signing key is the descriptor signing key (`KP_hs_desc_sign`). + The certificate type must be [0B], and the signing-key + extension is mandatory. + + NOTE: As with "auth-key", this certificate was intended to be + constructed the other way around. However, for compatibility + with C tor, implementations need to construct it this way. It + serves even less point than "auth-key", however, since the + encryption key `KP_hss_ntor` is already available from + the `enc-key` entry. + + "legacy-key" NL key NL + + [None or at most once per introduction point] + [This field is obsolete and should never be generated; it + is included for historical reasons only.] + + The key is an ASN.1 encoded RSA public key in PEM format used for a + legacy introduction point as described in [LEGACY_EST_INTRO]. + + This field is only present if the introduction point only supports + legacy protocol (v2) that is <= 0.2.9 or the protocol version value + "HSIntro 3". + + "legacy-key-cert" NL certificate NL + + [None or at most once per introduction point] + [This field is obsolete and should never be generated; it + is included for historical reasons only.] + + MUST be present if "legacy-key" is present. + + The certificate is a proposal 220 RSA->Ed cross-certificate wrapped + in "-----BEGIN CROSSCERT-----" armor, cross-certifying the RSA + public key found in "legacy-key" using the descriptor signing key. +``` + +To remain compatible with future revisions to the descriptor format, +clients should ignore unrecognized lines in the descriptor. +Other encryption and authentication key formats are allowed; clients +should ignore ones they do not recognize. + +Clients who manage to extract the introduction points of the hidden service +can proceed with the introduction protocol as specified in [INTRO-PROTOCOL]. + +Compatibility note: At least some versions of OnionBalance do not include +a final newline when generating this inner plaintext section; other +implementations MUST accept this section even if it is missing its final +newline. + + +### Deriving hidden service descriptor encryption keys [HS-DESC-ENCRYPTION-KEYS] + +In this section we present the generic encryption format for hidden service +descriptors. We use the same encryption format in both encryption layers, +hence we introduce two customization parameters SECRET_DATA and +STRING_CONSTANT which vary between the layers. + +The SECRET_DATA parameter specifies the secret data that are used during +encryption key generation, while STRING_CONSTANT is merely a string constant +that is used as part of the KDF. + +Here is the key generation logic: + +```text + SALT = 16 bytes from H(random), changes each time we rebuild the + descriptor even if the content of the descriptor hasn't changed. + (So that we don't leak whether the intro point list etc. changed) + + secret_input = SECRET_DATA | N_hs_subcred | INT_8(revision_counter) + + keys = KDF(secret_input | salt | STRING_CONSTANT, S_KEY_LEN + S_IV_LEN + MAC_KEY_LEN) + + SECRET_KEY = first S_KEY_LEN bytes of keys + SECRET_IV = next S_IV_LEN bytes of keys + MAC_KEY = last MAC_KEY_LEN bytes of keys + + The encrypted data has the format: + + SALT hashed random bytes from above [16 bytes] + ENCRYPTED The ciphertext [variable] + MAC D_MAC of both above fields [32 bytes] + + The final encryption format is ENCRYPTED = STREAM(SECRET_IV,SECRET_KEY) XOR Plaintext . + + Where D_MAC = H(mac_key_len | MAC_KEY | salt_len | SALT | ENCRYPTED) + and + mac_key_len = htonll(len(MAC_KEY)) + and + salt_len = htonll(len(SALT)). +``` + + +### Number of introduction points [NUM_INTRO_POINT] + +This section defines how many introduction points an hidden service +descriptor can have at minimum, by default and the maximum: + +Minimum: 0 - Default: 3 - Maximum: 20 + +A value of 0 would means that the service is still alive but doesn't want +to be reached by any client at the moment. Note that the descriptor size +increases considerably as more introduction points are added. + +The reason for a maximum value of 20 is to give enough scalability to tools +like OnionBalance to be able to load balance up to 120 servers (20 x 6 +HSDirs) but also in order for the descriptor size to not overwhelmed hidden +service directories with user defined values that could be gigantic. + diff --git a/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md b/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md new file mode 100644 index 0000000..df6a790 --- /dev/null +++ b/spec/rend-spec-v3/hidden-service-descriptors-outer-wrapper-desc-outer.md @@ -0,0 +1,74 @@ + +## Hidden service descriptors: outer wrapper [DESC-OUTER] + +The format for a hidden service descriptor is as follows, using the +meta-format from dir-spec.txt. + +"hs-descriptor" SP version-number NL + +[At start, exactly once.] + +```text + The version-number is a 32 bit unsigned integer indicating the version + of the descriptor. Current version is "3". + + "descriptor-lifetime" SP LifetimeMinutes NL + + [Exactly once] + + The lifetime of a descriptor in minutes. An HSDir SHOULD expire the + hidden service descriptor at least LifetimeMinutes after it was + uploaded. + + The LifetimeMinutes field can take values between 30 and 720 (12 + hours). + + "descriptor-signing-key-cert" NL certificate NL + + [Exactly once.] + + The 'certificate' field contains a certificate in the format from + proposal 220, wrapped with "-----BEGIN ED25519 CERT-----". The + certificate cross-certifies the short-term descriptor signing key with + the blinded public key. The certificate type must be [08], and the + blinded public key must be present as the signing-key extension. + + "revision-counter" SP Integer NL + + [Exactly once.] + + The revision number of the descriptor. If an HSDir receives a + second descriptor for a key that it already has a descriptor for, + it should retain and serve the descriptor with the higher + revision-counter. + + (Checking for monotonically increasing revision-counter values + prevents an attacker from replacing a newer descriptor signed by + a given key with a copy of an older version.) + + Implementations MUST be able to parse 64-bit values for these + counters. + + "superencrypted" NL encrypted-string + + [Exactly once.] + + An encrypted blob, whose format is discussed in [HS-DESC-ENC] below. The + blob is base64 encoded and enclosed in -----BEGIN MESSAGE---- and + ----END MESSAGE---- wrappers. (The resulting document does not end with + a newline character.) + + "signature" SP signature NL + + [exactly once, at end.] + + A signature of all previous fields, using the signing key in the + descriptor-signing-key-cert line, prefixed by the string "Tor onion + service descriptor sig v3". We use a separate key for signing, so that + the hidden service host does not need to have its private blinded key + online. +``` + +HSDirs accept hidden service descriptors of up to 50k bytes (a consensus +parameter should also be introduced to control this value). + diff --git a/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md b/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md new file mode 100644 index 0000000..6c56bba --- /dev/null +++ b/spec/rend-spec-v3/hidden-service-directory-format-hidservdir-format.md @@ -0,0 +1,30 @@ + +# Appendix F: Hidden service directory format [HIDSERVDIR-FORMAT] + +This appendix section specifies the contents of the HiddenServiceDir directory: + +- "hostname" [FILE] + +This file contains the onion address of the onion service. + +- "private_key_ed25519" [FILE] + +This file contains the private master ed25519 key of the onion service. +[TODO: Offline keys] + +```text + - "./authorized_clients/" [DIRECTORY] + "./authorized_clients/alice.auth" [FILE] + "./authorized_clients/bob.auth" [FILE] + "./authorized_clients/charlie.auth" [FILE] +``` + +If client authorization is enabled, this directory MUST contain a ".auth" +file for each authorized client. Each such file contains the public key of +the respective client. The files are transmitted to the service operator by +the client. + +See section [CLIENT-AUTH-MGMT] for more details and the format of the client file. + +(NOTE: client authorization is implemented as of 0.3.5.1-alpha.) + diff --git a/spec/rend-spec-v3/hidden-services-overview-preliminaries.md b/spec/rend-spec-v3/hidden-services-overview-preliminaries.md new file mode 100644 index 0000000..a71790c --- /dev/null +++ b/spec/rend-spec-v3/hidden-services-overview-preliminaries.md @@ -0,0 +1,309 @@ + +# Hidden services: overview and preliminaries. + +Hidden services aim to provide responder anonymity for bidirectional +stream-based communication on the Tor network. Unlike regular Tor +connections, where the connection initiator receives anonymity but +the responder does not, hidden services attempt to provide +bidirectional anonymity. + +Participants: + +Operator -- A person running a hidden service + +```text + Host, "Server" -- The Tor software run by the operator to provide + a hidden service. + + User -- A person contacting a hidden service. + + Client -- The Tor software running on the User's computer + + Hidden Service Directory (HSDir) -- A Tor node that hosts signed + statements from hidden service hosts so that users can make + contact with them. + + Introduction Point -- A Tor node that accepts connection requests + for hidden services and anonymously relays those requests to the + hidden service. + + Rendezvous Point -- A Tor node to which clients and servers + connect and which relays traffic between them. +``` + + +## Improvements over previous versions. + +Here is a list of improvements of this proposal over the legacy hidden +services: + +a) Better crypto (replaced SHA1/DH/RSA1024 with SHA3/ed25519/curve25519) +b) Improved directory protocol leaking less to directory servers. +c) Improved directory protocol with smaller surface for targeted attacks. +d) Better onion address security against impersonation. +e) More extensible introduction/rendezvous protocol. +f) Offline keys for onion services +g) Advanced client authorization + + +## Notation and vocabulary + +Unless specified otherwise, all multi-octet integers are big-endian. + +We write sequences of bytes in two ways: + +```text + 1. A sequence of two-digit hexadecimal values in square brackets, + as in [AB AD 1D EA]. + + 2. A string of characters enclosed in quotes, as in "Hello". The + characters in these strings are encoded in their ascii + representations; strings are NOT nul-terminated unless + explicitly described as NUL terminated. + + We use the words "byte" and "octet" interchangeably. + + We use the vertical bar | to denote concatenation. +``` + +We use INT_N(val) to denote the network (big-endian) encoding of the +unsigned integer "val" in N bytes. For example, INT_4(1337) is [00 00 +05 39]. Values are truncated like so: val % (2 ^ (N * 8)). For example, +INT_4(42) is 42 % 4294967296 (32 bit). + + +## Cryptographic building blocks + +This specification uses the following cryptographic building blocks: + +```text + * A pseudorandom number generator backed by a strong entropy source. + The output of the PRNG should always be hashed before being posted on + the network to avoid leaking raw PRNG bytes to the network + (see [PRNG-REFS]). + + * A stream cipher STREAM(iv, k) where iv is a nonce of length + S_IV_LEN bytes and k is a key of length S_KEY_LEN bytes. + + * A public key signature system SIGN_KEYGEN()->seckey, pubkey; + SIGN_SIGN(seckey,msg)->sig; and SIGN_CHECK(pubkey, sig, msg) -> + { "OK", "BAD" }; where secret keys are of length SIGN_SECKEY_LEN + bytes, public keys are of length SIGN_PUBKEY_LEN bytes, and + signatures are of length SIGN_SIG_LEN bytes. + + This signature system must also support key blinding operations + as discussed in appendix [KEYBLIND] and in section [SUBCRED]: + SIGN_BLIND_SECKEY(seckey, blind)->seckey2 and + SIGN_BLIND_PUBKEY(pubkey, blind)->pubkey2 . + + * A public key agreement system "PK", providing + PK_KEYGEN()->seckey, pubkey; PK_VALID(pubkey) -> {"OK", "BAD"}; + and PK_HANDSHAKE(seckey, pubkey)->output; where secret keys are + of length PK_SECKEY_LEN bytes, public keys are of length + PK_PUBKEY_LEN bytes, and the handshake produces outputs of + length PK_OUTPUT_LEN bytes. + + * A cryptographic hash function H(d), which should be preimage and + collision resistant. It produces hashes of length HASH_LEN + bytes. + + * A cryptographic message authentication code MAC(key,msg) that + produces outputs of length MAC_LEN bytes. + + * A key derivation function KDF(message, n) that outputs n bytes. + + As a first pass, I suggest: + + * Instantiate STREAM with AES256-CTR. + + * Instantiate SIGN with Ed25519 and the blinding protocol in + [KEYBLIND]. + + * Instantiate PK with Curve25519. + + * Instantiate H with SHA3-256. + + * Instantiate KDF with SHAKE-256. + + * Instantiate MAC(key=k, message=m) with H(k_len | k | m), + where k_len is htonll(len(k)). +``` + + When we need a particular MAC key length below, we choose + MAC_KEY_LEN=32 (256 bits). + +For legacy purposes, we specify compatibility with older versions of +the Tor introduction point and rendezvous point protocols. These used +RSA1024, DH1024, AES128, and SHA1, as discussed in +rend-spec.txt. + +As in [proposal 220], all signatures are generated not over strings +themselves, but over those strings prefixed with a distinguishing +value. + + +## Protocol building blocks [BUILDING-BLOCKS] + +In sections below, we need to transmit the locations and identities +of Tor nodes. We do so in the link identification format used by +EXTEND2 cells in the Tor protocol. + +```text + NSPEC (Number of link specifiers) [1 byte] + NSPEC times: + LSTYPE (Link specifier type) [1 byte] + LSLEN (Link specifier length) [1 byte] + LSPEC (Link specifier) [LSLEN bytes] +``` + +Link specifier types are as described in tor-spec.txt. Every set of +link specifiers SHOULD include at minimum specifiers of type [00] +(TLS-over-TCP, IPv4), [02] (legacy node identity) and [03] (ed25519 +identity key). Sets of link specifiers without these three types +SHOULD be rejected. + +As of 0.4.1.1-alpha, Tor includes both IPv4 and IPv6 link specifiers +in v3 onion service protocol link specifier lists. All available +addresses SHOULD be included as link specifiers, regardless of the +address that Tor actually used to connect/extend to the remote relay. + +We also incorporate Tor's circuit extension handshakes, as used in +the CREATE2 and CREATED2 cells described in tor-spec.txt. In these +handshakes, a client who knows a public key for a server sends a +message and receives a message from that server. Once the exchange is +done, the two parties have a shared set of forward-secure key +material, and the client knows that nobody else shares that key +material unless they control the secret key corresponding to the +server's public key. + + +## Assigned relay cell types + +These relay cell types are reserved for use in the hidden service +protocol. + +32 -- RELAY_COMMAND_ESTABLISH_INTRO + +```text + Sent from hidden service host to introduction point; + establishes introduction point. Discussed in + [REG_INTRO_POINT]. + + 33 -- RELAY_COMMAND_ESTABLISH_RENDEZVOUS + + Sent from client to rendezvous point; creates rendezvous + point. Discussed in [EST_REND_POINT]. + + 34 -- RELAY_COMMAND_INTRODUCE1 + + Sent from client to introduction point; requests + introduction. Discussed in [SEND_INTRO1] + + 35 -- RELAY_COMMAND_INTRODUCE2 + + Sent from introduction point to hidden service host; requests + introduction. Same format as INTRODUCE1. Discussed in + [FMT_INTRO1] and [PROCESS_INTRO2] + + 36 -- RELAY_COMMAND_RENDEZVOUS1 + + Sent from hidden service host to rendezvous point; + attempts to join host's circuit to + client's circuit. Discussed in [JOIN_REND] + + 37 -- RELAY_COMMAND_RENDEZVOUS2 + + Sent from rendezvous point to client; + reports join of host's circuit to + client's circuit. Discussed in [JOIN_REND] + + 38 -- RELAY_COMMAND_INTRO_ESTABLISHED + + Sent from introduction point to hidden service host; + reports status of attempt to establish introduction + point. Discussed in [INTRO_ESTABLISHED] + + 39 -- RELAY_COMMAND_RENDEZVOUS_ESTABLISHED + + Sent from rendezvous point to client; acknowledges + receipt of ESTABLISH_RENDEZVOUS cell. Discussed in + [EST_REND_POINT] + + 40 -- RELAY_COMMAND_INTRODUCE_ACK + + Sent from introduction point to client; acknowledges + receipt of INTRODUCE1 cell and reports success/failure. + Discussed in [INTRO_ACK] +``` + + +## Acknowledgments + +This design includes ideas from many people, including + +```text + Christopher Baines, + Daniel J. Bernstein, + Matthew Finkel, + Ian Goldberg, + George Kadianakis, + Aniket Kate, + Tanja Lange, + Robert Ransom, + Roger Dingledine, + Aaron Johnson, + Tim Wilson-Brown ("teor"), + special (John Brooks), + s7r +``` + +It's based on Tor's original hidden service design by Roger +Dingledine, Nick Mathewson, and Paul Syverson, and on improvements to +that design over the years by people including + +```text + Tobias Kamm, + Thomas Lauterbach, + Karsten Loesing, + Alessandro Preite Martinez, + Robert Ransom, + Ferdinand Rieger, + Christoph Weingarten, + Christian Wilms, +``` + +We wouldn't be able to do any of this work without good attack +designs from researchers including + +```text + Alex Biryukov, + Lasse Øverlier, + Ivan Pustogarov, + Paul Syverson, + Ralf-Philipp Weinmann, + + See [ATTACK-REFS] for their papers. + + Several of these ideas have come from conversations with + + Christian Grothoff, + Brian Warner, + Zooko Wilcox-O'Hearn, +``` + +And if this document makes any sense at all, it's thanks to +editing help from + +```text + Matthew Finkel, + George Kadianakis, + Peter Palfrader, + Tim Wilson-Brown ("teor"), +``` + +[XXX Acknowledge the huge bunch of people working on 8106.] +[XXX Acknowledge the huge bunch of people working on 8244.] + +Please forgive me if I've missed you; please forgive me if I've +misunderstood your best ideas here too. + diff --git a/spec/rend-spec-v3/introduction-protocol-intro-protocol.md b/spec/rend-spec-v3/introduction-protocol-intro-protocol.md new file mode 100644 index 0000000..a9813c2 --- /dev/null +++ b/spec/rend-spec-v3/introduction-protocol-intro-protocol.md @@ -0,0 +1,621 @@ + +# The introduction protocol [INTRO-PROTOCOL] + +The introduction protocol proceeds in three steps. + +First, a hidden service host builds an anonymous circuit to a Tor +node and registers that circuit as an introduction point. + +Single Onion Services attempt to build a non-anonymous single-hop circuit, +but use an anonymous 3-hop circuit if: + +```text + * the intro point is on an address that is configured as unreachable via + a direct connection, or + * the initial attempt to connect to the intro point over a single-hop + circuit fails, and they are retrying the intro point connection. + + [After 'First' and before 'Second', the hidden service publishes its + introduction points and associated keys, and the client fetches + them as described in section [HSDIR] above.] +``` + +Second, a client builds an anonymous circuit to the introduction +point, and sends an introduction request. + +Third, the introduction point relays the introduction request along +the introduction circuit to the hidden service host, and acknowledges +the introduction request to the client. + + +## Registering an introduction point [REG_INTRO_POINT] + + +### Extensible ESTABLISH_INTRO protocol. [EST_INTRO] + +When a hidden service is establishing a new introduction point, it +sends an ESTABLISH_INTRO cell with the following contents: + +```text + AUTH_KEY_TYPE [1 byte] + AUTH_KEY_LEN [2 bytes] + AUTH_KEY [AUTH_KEY_LEN bytes] + N_EXTENSIONS [1 byte] + N_EXTENSIONS times: + EXT_FIELD_TYPE [1 byte] + EXT_FIELD_LEN [1 byte] + EXT_FIELD [EXT_FIELD_LEN bytes] + HANDSHAKE_AUTH [MAC_LEN bytes] + SIG_LEN [2 bytes] + SIG [SIG_LEN bytes] +``` + +The AUTH_KEY_TYPE field indicates the type of the introduction point +authentication key and the type of the MAC to use in +HANDSHAKE_AUTH. Recognized types are: + +```text + [00, 01] -- Reserved for legacy introduction cells; see + [LEGACY_EST_INTRO below] + [02] -- Ed25519; SHA3-256. +``` + +The AUTH_KEY_LEN field determines the length of the AUTH_KEY +field. The AUTH_KEY field contains the public introduction point +authentication key, KP_hs_ipt_sid. + +The EXT_FIELD_TYPE, EXT_FIELD_LEN, EXT_FIELD entries are reserved for +extensions to the introduction protocol. Extensions with +unrecognized EXT_FIELD_TYPE values must be ignored. +(`EXT_FIELD_LEN` may be zero, in which case EXT_FIELD is absent.) + +```text + Unless otherwise specified in the documentation for an extension type: + * Each extension type SHOULD be sent only once in a message. + * Parties MUST ignore any occurrences all occurrences of an extension + with a given type after the first such occurrence. + * Extensions SHOULD be sent in numerically ascending order by type. + (The above extension sorting and multiplicity rules are only defaults; + they may be overridden in the descriptions of individual extensions.) +``` + +The HANDSHAKE_AUTH field contains the MAC of all earlier fields in +the cell using as its key the shared per-circuit material ("KH") +generated during the circuit extension protocol; see tor-spec.txt +section 5.2, "Setting circuit keys". It prevents replays of +ESTABLISH_INTRO cells. + +SIG_LEN is the length of the signature. + +SIG is a signature, using AUTH_KEY, of all contents of the cell, up +to but not including SIG_LEN and SIG. These contents are prefixed +with the string "Tor establish-intro cell v1". + +Upon receiving an ESTABLISH_INTRO cell, a Tor node first decodes the +key and the signature, and checks the signature. The node must reject +the ESTABLISH_INTRO cell and destroy the circuit in these cases: + +```text + * If the key type is unrecognized + * If the key is ill-formatted + * If the signature is incorrect + * If the HANDSHAKE_AUTH value is incorrect + + * If the circuit is already a rendezvous circuit. + * If the circuit is already an introduction circuit. + [TODO: some scalability designs fail there.] + * If the key is already in use by another circuit. +``` + +Otherwise, the node must associate the key with the circuit, for use +later in INTRODUCE1 cells. + + +#### Denial-of-Service Defense Extension. [EST_INTRO_DOS_EXT] + +This extension can be used to send Denial-of-Service (DoS) parameters to +the introduction point in order for it to apply them for the introduction +circuit. + +If used, it needs to be encoded within the N_EXTENSIONS field of the +ESTABLISH_INTRO cell defined in the previous section. The content is +defined as follow: + +EXT_FIELD_TYPE: + +[01] -- Denial-of-Service Parameters. + +```text + If this flag is set, the extension should be used by the introduction + point to learn what values the denial of service subsystem should be + using. + + EXT_FIELD content format is: + + N_PARAMS [1 byte] + N_PARAMS times: + PARAM_TYPE [1 byte] + PARAM_VALUE [8 byte] + + The PARAM_TYPE possible values are: + + [01] -- DOS_INTRODUCE2_RATE_PER_SEC + The rate per second of INTRODUCE2 cell relayed to the + service. + + [02] -- DOS_INTRODUCE2_BURST_PER_SEC + The burst per second of INTRODUCE2 cell relayed to the + service. + + The PARAM_VALUE size is 8 bytes in order to accommodate 64bit values. + It MUST match the specified limit for the following PARAM_TYPE: + + [01] -- Min: 0, Max: 2147483647 + [02] -- Min: 0, Max: 2147483647 + + A value of 0 means the defense is disabled. If the rate per second is + set to 0 (param 0x01) then the burst value should be ignored. And + vice-versa, if the burst value is 0 (param 0x02), then the rate value + should be ignored. In other words, setting one single parameter to 0 + disables the defense. + + The burst can NOT be smaller than the rate. If so, the parameters + should be ignored by the introduction point. + + Any valid value does have precedence over the network wide consensus + parameter. +``` + +Using this extension extends the payload of the ESTABLISH_INTRO cell by 19 +bytes bringing it from 134 bytes to 155 bytes. + +This extension can only be used with relays supporting the protocol version +"HSIntro=5". + +Introduced in tor-0.4.2.1-alpha. + +```text +3.1.2. Registering an introduction point on a legacy Tor node + [LEGACY_EST_INTRO] + + [This section is obsolete and refers to a workaround for now-obsolete Tor + relay versions. It is included for historical reasons.] +``` + +Tor nodes should also support an older version of the ESTABLISH_INTRO +cell, first documented in rend-spec.txt. New hidden service hosts +must use this format when establishing introduction points at older +Tor nodes that do not support the format above in [EST_INTRO]. + +In this older protocol, an ESTABLISH_INTRO cell contains: + +```text + KEY_LEN [2 bytes] + KEY [KEY_LEN bytes] + HANDSHAKE_AUTH [20 bytes] + SIG [variable, up to end of relay payload] + + The KEY_LEN variable determines the length of the KEY field. +``` + +The KEY field is the ASN1-encoded legacy RSA public key that was also +included in the hidden service descriptor. + +The HANDSHAKE_AUTH field contains the SHA1 digest of (KH | "INTRODUCE"). + +The SIG field contains an RSA signature, using PKCS1 padding, of all +earlier fields. + +Older versions of Tor always use a 1024-bit RSA key for these introduction +authentication keys. + + +### Acknowledging establishment of introduction point [INTRO_ESTABLISHED] + +After setting up an introduction circuit, the introduction point reports its +status back to the hidden service host with an INTRO_ESTABLISHED cell. + +The INTRO_ESTABLISHED cell has the following contents: + +```text + N_EXTENSIONS [1 byte] + N_EXTENSIONS times: + EXT_FIELD_TYPE [1 byte] + EXT_FIELD_LEN [1 byte] + EXT_FIELD [EXT_FIELD_LEN bytes] +``` + +Older versions of Tor send back an empty INTRO_ESTABLISHED cell instead. +Services must accept an empty INTRO_ESTABLISHED cell from a legacy relay. +[The above paragraph is obsolete and refers to a workaround for +now-obsolete Tor relay versions. It is included for historical reasons.] + +The same rules for multiplicity, ordering, and handling unknown types +apply to the extension fields here as described [EST_INTRO] above. + + +## Sending an INTRODUCE1 cell to the introduction point. [SEND_INTRO1] + +In order to participate in the introduction protocol, a client must +know the following: + +```text + * An introduction point for a service. + * The introduction authentication key for that introduction point. + * The introduction encryption key for that introduction point. +``` + +The client sends an INTRODUCE1 cell to the introduction point, +containing an identifier for the service, an identifier for the +encryption key that the client intends to use, and an opaque blob to +be relayed to the hidden service host. + +In reply, the introduction point sends an INTRODUCE_ACK cell back to +the client, either informing it that its request has been delivered, +or that its request will not succeed. + +```text + [TODO: specify what tor should do when receiving a malformed cell. Drop it? + Kill circuit? This goes for all possible cells.] +``` + + +### INTRODUCE1 cell format [FMT_INTRO1] + +When a client is connecting to an introduction point, INTRODUCE1 cells +should be of the form: + +```text + LEGACY_KEY_ID [20 bytes] + AUTH_KEY_TYPE [1 byte] + AUTH_KEY_LEN [2 bytes] + AUTH_KEY [AUTH_KEY_LEN bytes] + N_EXTENSIONS [1 byte] + N_EXTENSIONS times: + EXT_FIELD_TYPE [1 byte] + EXT_FIELD_LEN [1 byte] + EXT_FIELD [EXT_FIELD_LEN bytes] + ENCRYPTED [Up to end of relay payload] +``` + +AUTH_KEY_TYPE is defined as in [EST_INTRO]. Currently, the only value of +AUTH_KEY_TYPE for this cell is an Ed25519 public key [02]. + +The LEGACY_KEY_ID field is used to distinguish between legacy and new style +INTRODUCE1 cells. In new style INTRODUCE1 cells, LEGACY_KEY_ID is 20 zero +bytes. Upon receiving an INTRODUCE1 cell, the introduction point checks the +LEGACY_KEY_ID field. If LEGACY_KEY_ID is non-zero, the INTRODUCE1 cell +should be handled as a legacy INTRODUCE1 cell by the intro point. + +Upon receiving a INTRODUCE1 cell, the introduction point checks +whether AUTH_KEY matches the introduction point authentication key for an +active introduction circuit. If so, the introduction point sends an +INTRODUCE2 cell with exactly the same contents to the service, and sends an +INTRODUCE_ACK response to the client. + +(Note that the introduction point does not "clean up" the +INTRODUCE1 cells that it retransmits. Specifically, it does not +change the order or multiplicity of the extensions sent by the +client.) + +The same rules for multiplicity, ordering, and handling unknown types +apply to the extension fields here as described [EST_INTRO] above. + + +### INTRODUCE_ACK cell format. [INTRO_ACK] + +An INTRODUCE_ACK cell has the following fields: + +```text + STATUS [2 bytes] + N_EXTENSIONS [1 bytes] + N_EXTENSIONS times: + EXT_FIELD_TYPE [1 byte] + EXT_FIELD_LEN [1 byte] + EXT_FIELD [EXT_FIELD_LEN bytes] + + Recognized status values are: + + [00 00] -- Success: cell relayed to hidden service host. + [00 01] -- Failure: service ID not recognized + [00 02] -- Bad message format + [00 03] -- Can't relay cell to service +``` + +The same rules for multiplicity, ordering, and handling unknown types +apply to the extension fields here as described [EST_INTRO] above. + + +## Processing an INTRODUCE2 cell at the hidden service. [PROCESS_INTRO2] + +Upon receiving an INTRODUCE2 cell, the hidden service host checks whether +the AUTH_KEY or LEGACY_KEY_ID field matches the keys for this +introduction circuit. + +The service host then checks whether it has received a cell with these +contents or rendezvous cookie before. If it has, it silently drops it as a +replay. (It must maintain a replay cache for as long as it accepts cells +with the same encryption key. Note that the encryption format below should +be non-malleable.) + +If the cell is not a replay, it decrypts the ENCRYPTED field, +establishes a shared key with the client, and authenticates the whole +contents of the cell as having been unmodified since they left the +client. There may be multiple ways of decrypting the ENCRYPTED field, +depending on the chosen type of the encryption key. Requirements for +an introduction handshake protocol are described in +[INTRO-HANDSHAKE-REQS]. We specify one below in section +[NTOR-WITH-EXTRA-DATA]. + +The decrypted plaintext must have the form: + +```text + RENDEZVOUS_COOKIE [20 bytes] + N_EXTENSIONS [1 byte] + N_EXTENSIONS times: + EXT_FIELD_TYPE [1 byte] + EXT_FIELD_LEN [1 byte] + EXT_FIELD [EXT_FIELD_LEN bytes] + ONION_KEY_TYPE [1 bytes] + ONION_KEY_LEN [2 bytes] + ONION_KEY [ONION_KEY_LEN bytes] + NSPEC (Number of link specifiers) [1 byte] + NSPEC times: + LSTYPE (Link specifier type) [1 byte] + LSLEN (Link specifier length) [1 byte] + LSPEC (Link specifier) [LSLEN bytes] + PAD (optional padding) [up to end of plaintext] +``` + +Upon processing this plaintext, the hidden service makes sure that +any required authentication is present in the extension fields, and +then extends a rendezvous circuit to the node described in the LSPEC +fields, using the ONION_KEY to complete the extension. As mentioned +in [BUILDING-BLOCKS], the "TLS-over-TCP, IPv4" and "Legacy node +identity" specifiers must be present. + +As of 0.4.1.1-alpha, clients include both IPv4 and IPv6 link specifiers +in INTRODUCE1 cells. All available addresses SHOULD be included in the +cell, regardless of the address that the client actually used to extend +to the rendezvous point. + +The hidden service should handle invalid or unrecognised link specifiers +the same way as clients do in section 2.5.2.2. In particular, services +SHOULD perform basic validity checks on link specifiers, and SHOULD NOT +reject unrecognised link specifiers, to avoid information leaks. +The list of link specifiers received here SHOULD either be rejected, or +sent verbatim when extending to the rendezvous point, in the same order +received. + +The service MAY reject the list of link specifiers if it is +inconsistent with relay information from the directory, but SHOULD +NOT modify it. + +The ONION_KEY_TYPE field is: + +[01] NTOR: ONION_KEY is 32 bytes long. + +The ONION_KEY field describes the onion key that must be used when +extending to the rendezvous point. It must be of a type listed as +supported in the hidden service descriptor. + +The PAD field should be filled with zeros; its size should be chosen +so that the INTRODUCE2 message occupies a fixed maximum size, in +order to hide the length of the encrypted data. (This maximum size is +490, since we assume that a future Tor implementations will implement +proposal 340 and thus lower the number of bytes that can be contained +in a single relay message.) Note also that current versions of Tor +only pad the INTRODUCE2 message up to 246 bytes. + +Upon receiving a well-formed INTRODUCE2 cell, the hidden service host +will have: + +```text + * The information needed to connect to the client's chosen + rendezvous point. + * The second half of a handshake to authenticate and establish a + shared key with the hidden service client. + * A set of shared keys to use for end-to-end encryption. +``` + +The same rules for multiplicity, ordering, and handling unknown types +apply to the extension fields here as described [EST_INTRO] above. + + +### Introduction handshake encryption requirements [INTRO-HANDSHAKE-REQS] + +When decoding the encrypted information in an INTRODUCE2 cell, a +hidden service host must be able to: + +```text + * Decrypt additional information included in the INTRODUCE2 cell, + to include the rendezvous token and the information needed to + extend to the rendezvous point. + + * Establish a set of shared keys for use with the client. + + * Authenticate that the cell has not been modified since the client + generated it. +``` + +Note that the old TAP-derived protocol of the previous hidden service +design achieved the first two requirements, but not the third. + +```text +3.3.2. Example encryption handshake: ntor with extra data + [NTOR-WITH-EXTRA-DATA] + + [TODO: relocate this] +``` + +This is a variant of the ntor handshake (see tor-spec.txt, section +5.1.4; see proposal 216; and see "Anonymity and one-way +authentication in key-exchange protocols" by Goldberg, Stebila, and +Ustaoglu). + +It behaves the same as the ntor handshake, except that, in addition +to negotiating forward secure keys, it also provides a means for +encrypting non-forward-secure data to the server (in this case, to +the hidden service host) as part of the handshake. + +Notation here is as in section 5.1.4 of tor-spec.txt, which defines +the ntor handshake. + +The PROTOID for this variant is "tor-hs-ntor-curve25519-sha3-256-1". +We also use the following tweak values: + +```text + t_hsenc = PROTOID | ":hs_key_extract" + t_hsverify = PROTOID | ":hs_verify" + t_hsmac = PROTOID | ":hs_mac" + m_hsexpand = PROTOID | ":hs_key_expand" +``` + +To make an INTRODUCE1 cell, the client must know a public encryption +key B for the hidden service on this introduction circuit. The client +generates a single-use keypair: + +x,X = KEYGEN() + +and computes: + +```text + intro_secret_hs_input = EXP(B,x) | AUTH_KEY | X | B | PROTOID + info = m_hsexpand | N_hs_subcred + hs_keys = KDF(intro_secret_hs_input | t_hsenc | info, S_KEY_LEN+MAC_LEN) + ENC_KEY = hs_keys[0:S_KEY_LEN] + MAC_KEY = hs_keys[S_KEY_LEN:S_KEY_LEN+MAC_KEY_LEN] + + and sends, as the ENCRYPTED part of the INTRODUCE1 cell: + + CLIENT_PK [PK_PUBKEY_LEN bytes] + ENCRYPTED_DATA [Padded to length of plaintext] + MAC [MAC_LEN bytes] +``` + +Substituting those fields into the INTRODUCE1 cell body format +described in [FMT_INTRO1] above, we have + +```text + LEGACY_KEY_ID [20 bytes] + AUTH_KEY_TYPE [1 byte] + AUTH_KEY_LEN [2 bytes] + AUTH_KEY [AUTH_KEY_LEN bytes] + N_EXTENSIONS [1 bytes] + N_EXTENSIONS times: + EXT_FIELD_TYPE [1 byte] + EXT_FIELD_LEN [1 byte] + EXT_FIELD [EXT_FIELD_LEN bytes] + ENCRYPTED: + CLIENT_PK [PK_PUBKEY_LEN bytes] + ENCRYPTED_DATA [Padded to length of plaintext] + MAC [MAC_LEN bytes] +``` + +(This format is as documented in [FMT_INTRO1] above, except that here +we describe how to build the ENCRYPTED portion.) + +Here, the encryption key plays the role of B in the regular ntor +handshake, and the AUTH_KEY field plays the role of the node ID. +The CLIENT_PK field is the public key X. The ENCRYPTED_DATA field is +the message plaintext, encrypted with the symmetric key ENC_KEY. The +MAC field is a MAC of all of the cell from the AUTH_KEY through the +end of ENCRYPTED_DATA, using the MAC_KEY value as its key. + +To process this format, the hidden service checks PK_VALID(CLIENT_PK) +as necessary, and then computes ENC_KEY and MAC_KEY as the client did +above, except using EXP(CLIENT_PK,b) in the calculation of +intro_secret_hs_input. The service host then checks whether the MAC is +correct. If it is invalid, it drops the cell. Otherwise, it computes +the plaintext by decrypting ENCRYPTED_DATA. + +The hidden service host now completes the service side of the +extended ntor handshake, as described in tor-spec.txt section 5.1.4, +with the modified PROTOID as given above. To be explicit, the hidden +service host generates a keypair of y,Y = KEYGEN(), and uses its +introduction point encryption key 'b' to compute: + +```text + intro_secret_hs_input = EXP(X,b) | AUTH_KEY | X | B | PROTOID + info = m_hsexpand | N_hs_subcred + hs_keys = KDF(intro_secret_hs_input | t_hsenc | info, S_KEY_LEN+MAC_LEN) + HS_DEC_KEY = hs_keys[0:S_KEY_LEN] + HS_MAC_KEY = hs_keys[S_KEY_LEN:S_KEY_LEN+MAC_KEY_LEN] + + (The above are used to check the MAC and then decrypt the + encrypted data.) + + rend_secret_hs_input = EXP(X,y) | EXP(X,b) | AUTH_KEY | B | X | Y | PROTOID + NTOR_KEY_SEED = MAC(rend_secret_hs_input, t_hsenc) + verify = MAC(rend_secret_hs_input, t_hsverify) + auth_input = verify | AUTH_KEY | B | Y | X | PROTOID | "Server" + AUTH_INPUT_MAC = MAC(auth_input, t_hsmac) + + (The above are used to finish the ntor handshake.) + + The server's handshake reply is: + + SERVER_PK Y [PK_PUBKEY_LEN bytes] + AUTH AUTH_INPUT_MAC [MAC_LEN bytes] +``` + +These fields will be sent to the client in a RENDEZVOUS1 cell using the +HANDSHAKE_INFO element (see [JOIN_REND]). + +The hidden service host now also knows the keys generated by the +handshake, which it will use to encrypt and authenticate data +end-to-end between the client and the server. These keys are as +computed in tor-spec.txt section 5.1.4, except that instead of using +AES-128 and SHA1 for this hop, we use AES-256 and SHA3-256. + + +## Authentication during the introduction phase. [INTRO-AUTH] + +Hidden services may restrict access only to authorized users. +One mechanism to do so is the credential mechanism, where only users who +know the credential for a hidden service may connect at all. + +There is one defined authentication type: `ed25519`. + + +### Ed25519-based authentication `ed25519`. + +(NOTE: This section is not implemented by Tor. It is likely +that we would want to change its design substantially before +deploying any implementation. At the very least, we would +want to bind these extensions to a single onion service, to +prevent replays. We might also want to look for ways to limit +the number of keys a user needs to have.) + +To authenticate with an Ed25519 private key, the user must include an +extension field in the encrypted part of the INTRODUCE1 cell with an +EXT_FIELD_TYPE type of [02] and the contents: + +```text + Nonce [16 bytes] + Pubkey [32 bytes] + Signature [64 bytes] +``` + +Nonce is a random value. Pubkey is the public key that will be used +to authenticate. [TODO: should this be an identifier for the public +key instead?] Signature is the signature, using Ed25519, of: + +```text + "hidserv-userauth-ed25519" + Nonce (same as above) + Pubkey (same as above) + AUTH_KEY (As in the INTRODUCE1 cell) +``` + +The hidden service host checks this by seeing whether it recognizes +and would accept a signature from the provided public key. If it +would, then it checks whether the signature is correct. If it is, +then the correct user has authenticated. + +Replay prevention on the whole cell is sufficient to prevent replays +on the authentication. + +Users SHOULD NOT use the same public key with multiple hidden +services. + diff --git a/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md b/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md new file mode 100644 index 0000000..6802d4b --- /dev/null +++ b/spec/rend-spec-v3/managing-authorized-client-data-client-auth-mgmt.md @@ -0,0 +1,105 @@ + +# Appendix G: Managing authorized client data [CLIENT-AUTH-MGMT] + +Hidden services and clients can configure their authorized client data either +using the torrc, or using the control port. This section presents a suggested +scheme for configuring client authorization. Please see appendix +[HIDSERVDIR-FORMAT] for more information about relevant hidden service files. + +(NOTE: client authorization is implemented as of 0.3.5.1-alpha.) + +G.1. Configuring client authorization using torrc + +G.1.1. Hidden Service side configuration + +```text + A hidden service that wants to enable client authorization, needs to + populate the "authorized_clients/" directory of its HiddenServiceDir + directory with the ".auth" files of its authorized clients. + + When Tor starts up with a configured onion service, Tor checks its + /authorized_clients/ directory for ".auth" files, and if + any recognized and parseable such files are found, then client + authorization becomes activated for that service. + + G.1.2. Service-side bookkeeping + + This section contains more details on how onion services should be keeping + track of their client ".auth" files. + + For the "descriptor" authentication type, the ".auth" file MUST contain + the x25519 public key of that client. Here is a suggested file format: + + :: + + Here is an an example: + + descriptor:x25519:OM7TGIVRYMY6PFX6GAC6ATRTA5U6WW6U7A4ZNHQDI6OVL52XVV2Q + + Tor SHOULD ignore lines it does not recognize. + Tor SHOULD ignore files that don't use the ".auth" suffix. + + G.1.3. Client side configuration + + A client who wants to register client authorization data for onion + services needs to add the following line to their torrc to indicate the + directory which hosts ".auth_private" files containing client-side + credentials for onion services: + + ClientOnionAuthDir + + The contains a file with the suffix ".auth_private" for each onion + service the client is authorized with. Tor should scan the directory for + ".auth_private" files to find which onion services require client + authorization from this client. + + For the "descriptor" auth-type, a ".auth_private" file contains the + private x25519 key: + + :descriptor:x25519: + + The keypair used for client authorization is created by a third party tool + for which the public key needs to be transferred to the service operator + in a secure out-of-band way. The third party tool SHOULD add appropriate + headers to the private key file to ensure that users won't accidentally + give out their private key. + + G.2. Configuring client authorization using the control port + + G.2.1. Service side + + A hidden service also has the option to configure authorized clients + using the control port. The idea is that hidden service operators can use + controller utilities that manage their access control instead of using + the filesystem to register client keys. + + Specifically, we require a new control port command ADD_ONION_CLIENT_AUTH + which is able to register x25519/ed25519 public keys tied to a specific + authorized client. + [XXX figure out control port command format] + + Hidden services who use the control port interface for client auth need + to perform their own key management. + + G.2.2. Client side + + There should also be a control port interface for clients to register + authorization data for hidden services without having to use the + torrc. It should allow both generation of client authorization private + keys, and also to import client authorization data provided by a hidden + service + + This way, Tor Browser can present "Generate client auth keys" and "Import + client auth keys" dialogs to users when they try to visit a hidden service + that is protected by client authorization. + + Specifically, we require two new control port commands: + IMPORT_ONION_CLIENT_AUTH_DATA + GENERATE_ONION_CLIENT_AUTH_DATA + which import and generate client authorization data respectively. + + [XXX how does key management work here?] + [XXX what happens when people use both the control port interface and the + filesystem interface?] +``` + diff --git a/spec/rend-spec-v3/numeric-values-reserved-this-document.md b/spec/rend-spec-v3/numeric-values-reserved-this-document.md new file mode 100644 index 0000000..82eca4c --- /dev/null +++ b/spec/rend-spec-v3/numeric-values-reserved-this-document.md @@ -0,0 +1,5 @@ + +# Appendix D: Numeric values reserved in this document + +[TODO: collect all the lists of commands and values mentioned above] + diff --git a/spec/rend-spec-v3/open-questions.md b/spec/rend-spec-v3/open-questions.md new file mode 100644 index 0000000..af2b347 --- /dev/null +++ b/spec/rend-spec-v3/open-questions.md @@ -0,0 +1,80 @@ + +# Open Questions: + +Scaling hidden services is hard. There are on-going discussions that +you might be able to help with. See [SCALING-REFS]. + +How can we improve the HSDir unpredictability design proposed in +[SHAREDRANDOM]? See [SHAREDRANDOM-REFS] for discussion. + +How can hidden service addresses become memorable while retaining +their self-authenticating and decentralized nature? See +[HUMANE-HSADDRESSES-REFS] for some proposals; many more are possible. + +Hidden Services are pretty slow. Both because of the lengthy setup +procedure and because the final circuit has 6 hops. How can we make +the Hidden Service protocol faster? See [PERFORMANCE-REFS] for some +suggestions. + +References: + +```text +[KEYBLIND-REFS]: + https://trac.torproject.org/projects/tor/ticket/8106 + https://lists.torproject.org/pipermail/tor-dev/2012-September/004026.html + +[KEYBLIND-PROOF]: + https://lists.torproject.org/pipermail/tor-dev/2013-December/005943.html + +[SHAREDRANDOM-REFS]: + https://gitweb.torproject.org/torspec.git/tree/proposals/250-commit-reveal-consensus.txt + https://trac.torproject.org/projects/tor/ticket/8244 + +[SCALING-REFS]: + https://lists.torproject.org/pipermail/tor-dev/2013-October/005556.html + +[HUMANE-HSADDRESSES-REFS]: + https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/ideas/xxx-onion-nyms.txt + http://archives.seul.org/or/dev/Dec-2011/msg00034.html + +[PERFORMANCE-REFS]: + "Improving Efficiency and Simplicity of Tor circuit + establishment and hidden services" by Overlier, L., and + P. Syverson + + [TODO: Need more here! Do we have any? :( ] + +[ATTACK-REFS]: + "Trawling for Tor Hidden Services: Detection, Measurement, + Deanonymization" by Alex Biryukov, Ivan Pustogarov, + Ralf-Philipp Weinmann + + "Locating Hidden Servers" by Lasse Øverlier and Paul + Syverson + +[ED25519-REFS]: + "High-speed high-security signatures" by Daniel + J. Bernstein, Niels Duif, Tanja Lange, Peter Schwabe, and + Bo-Yin Yang. http://cr.yp.to/papers.html#ed25519 + +[ED25519-B-REF]: + https://tools.ietf.org/html/draft-josefsson-eddsa-ed25519-03#section-5: + +[PRNG-REFS]: + http://projectbullrun.org/dual-ec/ext-rand.html + https://lists.torproject.org/pipermail/tor-dev/2015-November/009954.html + +[SRV-TP-REFS]: + https://lists.torproject.org/pipermail/tor-dev/2016-April/010759.html + +[VANITY-REFS]: + https://github.com/Yawning/horse25519 + +[ONIONADDRESS-REFS]: + https://lists.torproject.org/pipermail/tor-dev/2017-January/011816.html + +[TORSION-REFS]: + https://lists.torproject.org/pipermail/tor-dev/2017-April/012164.html + https://getmonero.org/2017/05/17/disclosure-of-a-major-bug-in-cryptonote-based-currencies.html +``` + diff --git a/spec/rend-spec-v3/protocol-overview.md b/spec/rend-spec-v3/protocol-overview.md new file mode 100644 index 0000000..cb47184 --- /dev/null +++ b/spec/rend-spec-v3/protocol-overview.md @@ -0,0 +1,319 @@ + +# Protocol overview + +In this section, we outline the hidden service protocol. This section +omits some details in the name of simplicity; those are given more +fully below, when we specify the protocol in more detail. + + +## View from 10,000 feet + +A hidden service host prepares to offer a hidden service by choosing +several Tor nodes to serve as its introduction points. It builds +circuits to those nodes, and tells them to forward introduction +requests to it using those circuits. + +Once introduction points have been picked, the host builds a set of +documents called "hidden service descriptors" (or just "descriptors" +for short) and uploads them to a set of HSDir nodes. These documents +list the hidden service's current introduction points and describe +how to make contact with the hidden service. + +When a client wants to connect to a hidden service, it first chooses +a Tor node at random to be its "rendezvous point" and builds a +circuit to that rendezvous point. If the client does not have an +up-to-date descriptor for the service, it contacts an appropriate +HSDir and requests such a descriptor. + +The client then builds an anonymous circuit to one of the hidden +service's introduction points listed in its descriptor, and gives the +introduction point an introduction request to pass to the hidden +service. This introduction request includes the target rendezvous +point and the first part of a cryptographic handshake. + +Upon receiving the introduction request, the hidden service host +makes an anonymous circuit to the rendezvous point and completes the +cryptographic handshake. The rendezvous point connects the two +circuits, and the cryptographic handshake gives the two parties a +shared key and proves to the client that it is indeed talking to the +hidden service. + +Once the two circuits are joined, the client can send Tor RELAY cells +to the server. RELAY_BEGIN cells open streams to an external process +or processes configured by the server; RELAY_DATA cells are used to +communicate data on those streams, and so forth. + + +## In more detail: naming hidden services [NAMING] + +A hidden service's name is its long term master identity key. This is +encoded as a hostname by encoding the entire key in Base 32, including a +version byte and a checksum, and then appending the string ".onion" at the +end. The result is a 56-character domain name. + +(This is a change from older versions of the hidden service protocol, +where we used an 80-bit truncated SHA1 hash of a 1024 bit RSA key.) + +The names in this format are distinct from earlier names because of +their length. An older name might look like: + +```text + unlikelynamefora.onion + yyhws9optuwiwsns.onion + + And a new name following this specification might look like: + + l5satjgud6gucryazcyvyvhuxhr74u6ygigiuyixe3a6ysis67ororad.onion + + Please see section [ONIONADDRESS] for the encoding specification. +``` + + +## In more detail: Access control [IMD:AC] + +Access control for a hidden service is imposed at multiple points through +the process above. Furthermore, there is also the option to impose +additional client authorization access control using pre-shared secrets +exchanged out-of-band between the hidden service and its clients. + +The first stage of access control happens when downloading HS descriptors. +Specifically, in order to download a descriptor, clients must know which +blinded signing key was used to sign it. (See the next section for more info +on key blinding.) + +To learn the introduction points, clients must decrypt the body of the +hidden service descriptor. To do so, clients must know the _unblinded_ +public key of the service, which makes the descriptor unusable by entities +without that knowledge (e.g. HSDirs that don't know the onion address). + +Also, if optional client authorization is enabled, hidden service +descriptors are superencrypted using each authorized user's identity x25519 +key, to further ensure that unauthorized entities cannot decrypt it. + +In order to make the introduction point send a rendezvous request to the +service, the client needs to use the per-introduction-point authentication +key found in the hidden service descriptor. + +The final level of access control happens at the server itself, which may +decide to respond or not respond to the client's request depending on the +contents of the request. The protocol is extensible at this point: at a +minimum, the server requires that the client demonstrate knowledge of the +contents of the encrypted portion of the hidden service descriptor. If +optional client authorization is enabled, the service may additionally +require the client to prove knowledge of a pre-shared private key. + + +## In more detail: Distributing hidden service descriptors. [IMD:DIST] + +Periodically, hidden service descriptors become stored at different +locations to prevent a single directory or small set of directories +from becoming a good DoS target for removing a hidden service. + +For each period, the Tor directory authorities agree upon a +collaboratively generated random value. (See section 2.3 for a +description of how to incorporate this value into the voting +practice; generating the value is described in other proposals, +including [SHAREDRANDOM-REFS].) That value, combined with hidden service +directories' public identity keys, determines each HSDir's position +in the hash ring for descriptors made in that period. + +Each hidden service's descriptors are placed into the ring in +positions based on the key that was used to sign them. Note that +hidden service descriptors are not signed with the services' public +keys directly. Instead, we use a key-blinding system [KEYBLIND] to +create a new key-of-the-day for each hidden service. Any client that +knows the hidden service's public identity key can derive these blinded +signing keys for a given period. It should be impossible to derive +the blinded signing key lacking that knowledge. + +This is achieved using two nonces: + +```text + * A "credential", derived from the public identity key KP_hs_id. + N_hs_cred. + + * A "subcredential", derived from the credential N_hs_cred + and information which various with the current time period. + N_hs_subcred. +``` + +The body of each descriptor is also encrypted with a key derived from +the public signing key. + +To avoid a "thundering herd" problem where every service generates +and uploads a new descriptor at the start of each period, each +descriptor comes online at a time during the period that depends on +its blinded signing key. The keys for the last period remain valid +until the new keys come online. + + +## In more detail: Scaling to multiple hosts + +This design is compatible with our current approaches for scaling hidden +services. Specifically, hidden service operators can use onionbalance to +achieve high availability between multiple nodes on the HSDir +layer. Furthermore, operators can use proposal 255 to load balance their +hidden services on the introduction layer. See [SCALING-REFS] for further +discussions on this topic and alternative designs. + +```text +1.6. In more detail: Backward compatibility with older hidden service + protocols +``` + +This design is incompatible with the clients, server, and hsdir node +protocols from older versions of the hidden service protocol as +described in rend-spec.txt. On the other hand, it is designed to +enable the use of older Tor nodes as rendezvous points and +introduction points. + + +## In more detail: Keeping crypto keys offline + +In this design, a hidden service's secret identity key may be +stored offline. It's used only to generate blinded signing keys, +which are used to sign descriptor signing keys. + +In order to operate a hidden service, the operator can generate in +advance a number of blinded signing keys and descriptor signing +keys (and their credentials; see [DESC-OUTER] and [HS-DESC-ENC] +below), and their corresponding descriptor encryption keys, and +export those to the hidden service hosts. + +As a result, in the scenario where the Hidden Service gets +compromised, the adversary can only impersonate it for a limited +period of time (depending on how many signing keys were generated +in advance). + +It's important to not send the private part of the blinded signing +key to the Hidden Service since an attacker can derive from it the +secret master identity key. The secret blinded signing key should +only be used to create credentials for the descriptor signing keys. + +(NOTE: although the protocol allows them, offline keys are not +implemented as of 0.3.2.1-alpha.) + + +## In more detail: Encryption Keys And Replay Resistance + +To avoid replays of an introduction request by an introduction point, +a hidden service host must never accept the same request +twice. Earlier versions of the hidden service design used an +authenticated timestamp here, but including a view of the current +time can create a problematic fingerprint. (See proposal 222 for more +discussion.) + + +## In more detail: A menagerie of keys + +[In the text below, an "encryption keypair" is roughly "a keypair you +can do Diffie-Hellman with" and a "signing keypair" is roughly "a +keypair you can do ECDSA with."] + +Public/private keypairs defined in this document: + +```text + Master (hidden service) identity key -- A master signing keypair + used as the identity for a hidden service. This key is long + term and not used on its own to sign anything; it is only used + to generate blinded signing keys as described in [KEYBLIND] + and [SUBCRED]. The public key is encoded in the ".onion" + address according to [NAMING]. + KP_hs_id, KS_hs_id. + + Blinded signing key -- A keypair derived from the identity key, + used to sign descriptor signing keys. It changes periodically for + each service. Clients who know a 'credential' consisting of the + service's public identity key and an optional secret can derive + the public blinded identity key for a service. This key is used + as an index in the DHT-like structure of the directory system + (see [SUBCRED]). + KP_hs_blind_id, KS_hs_blind_id. + + Descriptor signing key -- A key used to sign hidden service + descriptors. This is signed by blinded signing keys. Unlike + blinded signing keys and master identity keys, the secret part + of this key must be stored online by hidden service hosts. The + public part of this key is included in the unencrypted section + of HS descriptors (see [DESC-OUTER]). + KP_hs_desc_sign, KS_hs_desc_sign. + + Introduction point authentication key -- A short-term signing + keypair used to identify a hidden service's session at a given + introduction point. The service makes a fresh keypair for each + introduction point; these are used to sign the request that a + hidden service host makes when establishing an introduction + point, so that clients who know the public component of this key + can get their introduction requests sent to the right + service. No keypair is ever used with more than one introduction + point. (previously called a "service key" in rend-spec.txt) + KP_hs_ipt_sid, KS_hs_ipt_sid + ("hidden service introduction point session id"). + + Introduction point encryption key -- A short-term encryption + keypair used when establishing connections via an introduction + point. Plays a role analogous to Tor nodes' onion keys. The service + makes a fresh keypair for each introduction point. + KP_hss_ntor, KS_hss_ntor. + + Ephemeral descriptor encryption key -- A short-lived encryption + keypair made by the service, and used to encrypt the inner layer + of hidden service descriptors when client authentication is in + use. + KP_hss_desc_enc, KS_hss_desc_enc + + Nonces defined in this document: + + N_hs_desc_enc -- a nonce used to derive keys to decrypt the inner + encryption layer of hidden service descriptors. This is + sometimes also called a "descriptor cookie". + + Public/private keypairs defined elsewhere: + + Onion key -- Short-term encryption keypair (KS_ntor, KP_ntor). + + (Node) identity key (KP_relayid). + + Symmetric key-like things defined elsewhere: + + KH from circuit handshake -- An unpredictable value derived as + part of the Tor circuit extension handshake, used to tie a request + to a particular circuit. +``` + + +### In even more detail: Client authorization keys [CLIENT-AUTH] + +When client authorization is enabled, each authorized client of a hidden +service has two more asymmetric keypairs which are shared with the hidden +service. An entity without those keys is not able to use the hidden +service. Throughout this document, we assume that these pre-shared keys are +exchanged between the hidden service and its clients in a secure out-of-band +fashion. + +Specifically, each authorized client possesses: + +```text + - An x25519 keypair used to compute decryption keys that allow the client to + decrypt the hidden service descriptor. See [HS-DESC-ENC]. This is + the client's counterpart to KP_hss_desc_enc. + KP_hsc_desc_enc, KS_hsd_desc_enc. + + - An ed25519 keypair which allows the client to compute signatures which + prove to the hidden service that the client is authorized. These + signatures are inserted into the INTRODUCE1 cell, and without them the + introduction to the hidden service cannot be completed. See [INTRO-AUTH]. + KP_hsc_intro_auth, KS_hsc_intro_auth. +``` + +The right way to exchange these keys is to have the client generate keys and +send the corresponding public keys to the hidden service out-of-band. An +easier but less secure way of doing this exchange would be to have the +hidden service generate the keypairs and pass the corresponding private keys +to its clients. See section [CLIENT-AUTH-MGMT] for more details on how these +keys should be managed. + +[TODO: Also specify stealth client authorization.] + +(NOTE: client authorization is implemented as of 0.3.5.1-alpha.) + diff --git a/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md b/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md new file mode 100644 index 0000000..51f3624 --- /dev/null +++ b/spec/rend-spec-v3/publishing-shared-random-values-pub-sharedrandom.md @@ -0,0 +1,47 @@ + +## Publishing shared random values [PUB-SHAREDRANDOM] + +Our design for limiting the predictability of HSDir upload locations +relies on a shared random value (SRV) that isn't predictable in advance or +too influenceable by an attacker. The authorities must run a protocol +to generate such a value at least once per hsdir period. Here we +describe how they publish these values; the procedure they use to +generate them can change independently of the rest of this +specification. For more information see [SHAREDRANDOM-REFS]. + +According to proposal 250, we add two new lines in consensuses: + +```text + "shared-rand-previous-value" SP NUM_REVEALS SP VALUE NL + "shared-rand-current-value" SP NUM_REVEALS SP VALUE NL +``` + + +### Client behavior in the absence of shared random values + +If the previous or current shared random value cannot be found in a +consensus, then Tor clients and services need to generate their own random +value for use when choosing HSDirs. + +To do so, Tor clients and services use: + +SRV = H("shared-random-disaster" | INT_8(period_length) | INT_8(period_num)) + +where period_length is the length of a time period in minutes, +rounded down; period_num is calculated as specified in +[TIME-PERIODS] for the wanted shared random value that could not be +found originally. + + +### Hidden services and changing shared random values + +It's theoretically possible that the consensus shared random values will +change or disappear in the middle of a time period because of directory +authorities dropping offline or misbehaving. + +To avoid client reachability issues in this rare event, hidden services +should use the new shared random values to find the new responsible HSDirs +and upload their descriptors there. + +XXX How long should they upload descriptors there for? + diff --git a/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md b/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md new file mode 100644 index 0000000..9cbd8b9 --- /dev/null +++ b/spec/rend-spec-v3/recommendations-for-searching-for-vanity-onions.md @@ -0,0 +1,45 @@ + +# Appendix C: Recommendations for searching for vanity .onions [VANITY] + +EDITORIAL NOTE: The author thinks that it's silly to brute-force the +keyspace for a key that, when base-32 encoded, spells out the name of +your website. It also feels a bit dangerous to me. If you train your +users to connect to + +llamanymityx4fi3l6x2gyzmtmgxjyqyorj9qsb5r543izcwymle.onion + +I worry that you're making it easier for somebody to trick them into +connecting to + +llamanymityb4sqi0ta0tsw6uovyhwlezkcrmczeuzdvfauuemle.onion + +Nevertheless, people are probably going to try to do this, so here's a +decent algorithm to use. + +To search for a public key with some criterion X: + +Generate a random (sk,pk) pair. + +While pk does not satisfy X: + +```text + Add the number 8 to sk + Add the point 8*B to pk + + Return sk, pk. +``` + +We add 8 and 8*B, rather than 1 and B, so that sk is always a valid +Curve25519 private key, with the lowest 3 bits equal to 0. + +This algorithm is safe [source: djb, personal communication] [TODO: +Make sure I understood correctly!] so long as only the final (sk,pk) +pair is used, and all previous values are discarded. + +To parallelize this algorithm, start with an independent (sk,pk) pair +generated for each independent thread, and let each search proceed +independently. + +See [VANITY-REFS] for a reference implementation of this vanity .onion +search scheme. + diff --git a/spec/rend-spec-v3/rendezvous-protocol.md b/spec/rend-spec-v3/rendezvous-protocol.md new file mode 100644 index 0000000..65a70aa --- /dev/null +++ b/spec/rend-spec-v3/rendezvous-protocol.md @@ -0,0 +1,134 @@ + +# The rendezvous protocol + +Before connecting to a hidden service, the client first builds a +circuit to an arbitrarily chosen Tor node (known as the rendezvous +point), and sends an ESTABLISH_RENDEZVOUS cell. The hidden service +later connects to the same node and sends a RENDEZVOUS cell. Once +this has occurred, the relay forwards the contents of the RENDEZVOUS +cell to the client, and joins the two circuits together. + +Single Onion Services attempt to build a non-anonymous single-hop circuit, +but use an anonymous 3-hop circuit if: + +```text + * the rend point is on an address that is configured as unreachable via + a direct connection, or + * the initial attempt to connect to the rend point over a single-hop + circuit fails, and they are retrying the rend point connection. +``` + + +## Establishing a rendezvous point [EST_REND_POINT] + +The client sends the rendezvous point a RELAY_COMMAND_ESTABLISH_RENDEZVOUS +cell containing a 20-byte value. + +RENDEZVOUS_COOKIE [20 bytes] + +Rendezvous points MUST ignore any extra bytes in an +ESTABLISH_RENDEZVOUS cell. (Older versions of Tor did not.) + +The rendezvous cookie is an arbitrary 20-byte value, chosen randomly +by the client. The client SHOULD choose a new rendezvous cookie for +each new connection attempt. If the rendezvous cookie is already in +use on an existing circuit, the rendezvous point should reject it and +destroy the circuit. + +Upon receiving an ESTABLISH_RENDEZVOUS cell, the rendezvous point associates +the cookie with the circuit on which it was sent. It replies to the client +with an empty RENDEZVOUS_ESTABLISHED cell to indicate success. Clients MUST +ignore any extra bytes in a RENDEZVOUS_ESTABLISHED cell. + +The client MUST NOT use the circuit which sent the cell for any +purpose other than rendezvous with the given location-hidden service. + +The client should establish a rendezvous point BEFORE trying to +connect to a hidden service. + + +## Joining to a rendezvous point [JOIN_REND] + +To complete a rendezvous, the hidden service host builds a circuit to +the rendezvous point and sends a RENDEZVOUS1 cell containing: + +```text + RENDEZVOUS_COOKIE [20 bytes] + HANDSHAKE_INFO [variable; depends on handshake type + used.] +``` + +where RENDEZVOUS_COOKIE is the cookie suggested by the client during the +introduction (see [PROCESS_INTRO2]) and HANDSHAKE_INFO is defined in +[NTOR-WITH-EXTRA-DATA]. + +If the cookie matches the rendezvous cookie set on any +not-yet-connected circuit on the rendezvous point, the rendezvous +point connects the two circuits, and sends a RENDEZVOUS2 cell to the +client containing the HANDSHAKE_INFO field of the RENDEZVOUS1 cell. + +Upon receiving the RENDEZVOUS2 cell, the client verifies that HANDSHAKE_INFO +correctly completes a handshake. To do so, the client parses SERVER_PK from +HANDSHAKE_INFO and reverses the final operations of section +[NTOR-WITH-EXTRA-DATA] as shown here: + +```text + rend_secret_hs_input = EXP(Y,x) | EXP(B,x) | AUTH_KEY | B | X | Y | PROTOID + NTOR_KEY_SEED = MAC(ntor_secret_input, t_hsenc) + verify = MAC(ntor_secret_input, t_hsverify) + auth_input = verify | AUTH_KEY | B | Y | X | PROTOID | "Server" + AUTH_INPUT_MAC = MAC(auth_input, t_hsmac) +``` + +Finally the client verifies that the received AUTH field of HANDSHAKE_INFO +is equal to the computed AUTH_INPUT_MAC. + +Now both parties use the handshake output to derive shared keys for use on +the circuit as specified in the section below: + + +### Key expansion + +The hidden service and its client need to derive crypto keys from the +NTOR_KEY_SEED part of the handshake output. To do so, they use the KDF +construction as follows: + +K = KDF(NTOR_KEY_SEED | m_hsexpand, HASH_LEN * 2 + S_KEY_LEN * 2) + +The first HASH_LEN bytes of K form the forward digest Df; the next HASH_LEN +bytes form the backward digest Db; the next S_KEY_LEN bytes form Kf, and the +final S_KEY_LEN bytes form Kb. Excess bytes from K are discarded. + +Subsequently, the rendezvous point passes relay cells, unchanged, from each +of the two circuits to the other. When Alice's OP sends RELAY cells along +the circuit, it authenticates with Df, and encrypts them with the Kf, then +with all of the keys for the ORs in Alice's side of the circuit; and when +Alice's OP receives RELAY cells from the circuit, it decrypts them with the +keys for the ORs in Alice's side of the circuit, then decrypts them with Kb, +and checks integrity with Db. Bob's OP does the same, with Kf and Kb +interchanged. + +[TODO: Should we encrypt HANDSHAKE_INFO as we did INTRODUCE2 +contents? It's not necessary, but it could be wise. Similarly, we +should make it extensible.] + + +## Using legacy hosts as rendezvous points + +[This section is obsolete and refers to a workaround for now-obsolete Tor +relay versions. It is included for historical reasons.] + +The behavior of ESTABLISH_RENDEZVOUS is unchanged from older versions +of this protocol, except that relays should now ignore unexpected +bytes at the end. + +Old versions of Tor required that RENDEZVOUS cell payloads be exactly +168 bytes long. All shorter rendezvous payloads should be padded to +this length with random bytes, to make them difficult to distinguish from +older protocols at the rendezvous point. + +Relays older than 0.2.9.1 should not be used for rendezvous points by next +generation onion services because they enforce too-strict length checks to +rendezvous cells. Hence the "HSRend" protocol from proposal#264 should be +used to select relays for rendezvous points. + diff --git a/spec/rend-spec-v3/reserved-numbers.md b/spec/rend-spec-v3/reserved-numbers.md new file mode 100644 index 0000000..2fe9fba --- /dev/null +++ b/spec/rend-spec-v3/reserved-numbers.md @@ -0,0 +1,17 @@ + +# Appendix E: Reserved numbers + +We reserve these certificate type values for Ed25519 certificates: + +```text + [08] short-term descriptor signing key, signed with blinded + public key. (Section 2.4) + [09] intro point authentication key, cross-certifying the descriptor + signing key. (Section 2.5) + [0B] ed25519 key derived from the curve25519 intro point encryption key, + cross-certifying the descriptor signing key. (Section 2.5) + + Note: The value "0A" is skipped because it's reserved for the onion key + cross-certifying ntor identity key from proposal 228. +``` + diff --git a/spec/rend-spec-v3/selecting-nodes-picknodes.md b/spec/rend-spec-v3/selecting-nodes-picknodes.md new file mode 100644 index 0000000..cb4cb13 --- /dev/null +++ b/spec/rend-spec-v3/selecting-nodes-picknodes.md @@ -0,0 +1,10 @@ + +# Appendix B: Selecting nodes [PICKNODES] + +Picking introduction points +Picking rendezvous points +Building paths +Reusing circuits + +(TODO: This needs a writeup) + diff --git a/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md b/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md new file mode 100644 index 0000000..f4f950d --- /dev/null +++ b/spec/rend-spec-v3/signature-scheme-with-key-blinding-keyblind.md @@ -0,0 +1,102 @@ + +# Appendix A: Signature scheme with key blinding [KEYBLIND] + + +## Key derivation overview + +As described in [IMD:DIST] and [SUBCRED] above, we require a "key +blinding" system that works (roughly) as follows: + +There is a master keypair (sk, pk). + +```text + Given the keypair and a nonce n, there is a derivation function + that gives a new blinded keypair (sk_n, pk_n). This keypair can + be used for signing. + + Given only the public key and the nonce, there is a function + that gives pk_n. + + Without knowing pk, it is not possible to derive pk_n; without + knowing sk, it is not possible to derive sk_n. + + It's possible to check that a signature was made with sk_n while + knowing only pk_n. + + Someone who sees a large number of blinded public keys and + signatures made using those public keys can't tell which + signatures and which blinded keys were derived from the same + master keypair. + + You can't forge signatures. + + [TODO: Insert a more rigorous definition and better references.] +``` + + +## Tor's key derivation scheme + +We propose the following scheme for key blinding, based on Ed25519. + +(This is an ECC group, so remember that scalar multiplication is the +trapdoor function, and it's defined in terms of iterated point +addition. See the Ed25519 paper [Reference ED25519-REFS] for a fairly +clear writeup.) + +Let B be the ed25519 basepoint as found in section 5 of [ED25519-B-REF]: + +```text + B = (15112221349535400772501151409588531511454012693041857206046113283949847762202, + 46316835694926478169428394003475163141307993866256225615783033603165251855960) +``` + +Assume B has prime order l, so lB=0. Let a master keypair be written as +(a,A), where a is the private key and A is the public key (A=aB). + +To derive the key for a nonce N and an optional secret s, compute the +blinding factor like this: + +```text + h = H(BLIND_STRING | A | s | B | N) + BLIND_STRING = "Derive temporary signing key" | INT_1(0) + N = "key-blind" | INT_8(period-number) | INT_8(period_length) + B = "(1511[...]2202, 4631[...]5960)" + + then clamp the blinding factor 'h' according to the ed25519 spec: + + h[0] &= 248; + h[31] &= 63; + h[31] |= 64; + + and do the key derivation as follows: + + private key for the period: + + a' = h a mod l + RH' = SHA-512(RH_BLIND_STRING | RH)[:32] + RH_BLIND_STRING = "Derive temporary signing key hash input" + + public key for the period: + + A' = h A = (ha)B +``` + +Generating a signature of M: given a deterministic random-looking r +(see EdDSA paper), take R=rB, S=r+hash(R,A',M)ah mod l. Send signature +(R,S) and public key A'. + +Verifying the signature: Check whether SB = R+hash(R,A',M)A'. + +```text + (If the signature is valid, + SB = (r + hash(R,A',M)ah)B + = rB + (hash(R,A',M)ah)B + = R + hash(R,A',M)A' ) + + This boils down to regular Ed25519 with key pair (a', A'). +``` + +See [KEYBLIND-REFS] for an extensive discussion on this scheme and +possible alternatives. Also, see [KEYBLIND-PROOF] for a security +proof of this scheme. + diff --git a/spec/rend-spec-v3/text-vectors.md b/spec/rend-spec-v3/text-vectors.md new file mode 100644 index 0000000..71897e8 --- /dev/null +++ b/spec/rend-spec-v3/text-vectors.md @@ -0,0 +1,101 @@ + +# Appendix G: Text vectors + +G.1. Test vectors for hs-ntor / NTOR-WITH-EXTRA-DATA + +```text + Here is a set of test values for the hs-ntor handshake, called + [NTOR-WITH-EXTRA-DATA] in this document. They were generated by + instrumenting Tor's code to dump the values for an INTRODUCE/RENDEZVOUS + handshake, and then by running that code on a Chutney network. + + We assume an onion service with: + + KP_hs_ipd_sid = 34E171E4358E501BFF21ED907E96AC6B + FEF697C779D040BBAF49ACC30FC5D21F + KP_hss_ntor = 8E5127A40E83AABF6493E41F142B6EE3 + 604B85A3961CD7E38D247239AFF71979 + KS_hss_ntor = A0ED5DBF94EEB2EDB3B514E4CF6ABFF6 + 022051CC5F103391F1970A3FCD15296A + N_hs_subcred = 0085D26A9DEBA252263BF0231AEAC59B + 17CA11BAD8A218238AD6487CBAD68B57 + + The client wants to make in INTRODUCE request. It generates + the following header (everything before the ENCRYPTED portion) + of its INTRODUCE1 cell: + + H = 000000000000000000000000000000000000000002002034E171E4358E501BFF + 21ED907E96AC6BFEF697C779D040BBAF49ACC30FC5D21F00 + + It generates the following plaintext body to encrypt. (This + is the "decrypted plaintext body" from [PROCESS_INTRO2]. + + P = 6BD364C12638DD5C3BE23D76ACA05B04E6CE932C0101000100200DE6130E4FCA + C4EDDA24E21220CC3EADAE403EF6B7D11C8273AC71908DE565450300067F0000 + 0113890214F823C4F8CC085C792E0AEE0283FE00AD7520B37D0320728D5DF39B + 7B7077A0118A900FF4456C382F0041300ACF9C58E51C392795EF870000000000 + 0000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000 + + (Note! This should in fact be padded to be longer; when these + test vectors were generated, the target INTRODUCE1 length in C + Tor was needlessly short.) + + The client now begins the hs-ntor handshake. It generates + a curve25519 keypair: + + x = 60B4D6BF5234DCF87A4E9D7487BDF3F4 + A69B6729835E825CA29089CFDDA1E341 + X = BF04348B46D09AED726F1D66C618FDEA + 1DE58E8CB8B89738D7356A0C59111D5D + + Then it calculates: + + ENC_KEY = 9B8917BA3D05F3130DACCE5300C3DC27 + F6D012912F1C733036F822D0ED238706 + MAC_KEY = FC4058DA59D4DF61E7B40985D122F502 + FD59336BC21C30CAF5E7F0D4A2C38FD5 + + With these, it encrypts the plaintext body P with ENC_KEY, getting + an encrypted value C. It computes MAC(MAC_KEY, H | X | C), + getting a MAC value M. It then assembles the final INTRODUCE1 + body as H | X | C | M: + + 000000000000000000000000000000000000000002002034E171E4358E501BFF + 21ED907E96AC6BFEF697C779D040BBAF49ACC30FC5D21F00BF04348B46D09AED + 726F1D66C618FDEA1DE58E8CB8B89738D7356A0C59111D5DADBECCCB38E37830 + 4DCC179D3D9E437B452AF5702CED2CCFEC085BC02C4C175FA446525C1B9D5530 + 563C362FDFFB802DAB8CD9EBC7A5EE17DA62E37DEEB0EB187FBB48C63298B0E8 + 3F391B7566F42ADC97C46BA7588278273A44CE96BC68FFDAE31EF5F0913B9A9C + 7E0F173DBC0BDDCD4ACB4C4600980A7DDD9EAEC6E7F3FA3FC37CD95E5B8BFB3E + 35717012B78B4930569F895CB349A07538E42309C993223AEA77EF8AEA64F25D + DEE97DA623F1AEC0A47F150002150455845C385E5606E41A9A199E7111D54EF2 + D1A51B7554D8B3692D85AC587FB9E69DF990EFB776D8 +``` + +Later the service receives that body in an INTRODUCE2 cell. It +processes it according to the hs-ntor handshake, and recovers +the client's plaintext P. To continue the hs-ntor handshake, +the service chooses a curve25519 keypair: + +```text + y = 68CB5188CA0CD7924250404FAB54EE13 + 92D3D2B9C049A2E446513875952F8F55 + Y = 8FBE0DB4D4A9C7FF46701E3E0EE7FD05 + CD28BE4F302460ADDEEC9E93354EE700 + + From this and the client's input, it computes: + + AUTH_INPUT_MAC = 4A92E8437B8424D5E5EC279245D5C72B + 25A0327ACF6DAF902079FCB643D8B208 + NTOR_KEY_SEED = 4D0C72FE8AFF35559D95ECC18EB5A368 + 83402B28CDFD48C8A530A5A3D7D578DB +``` + +The service sends back Y | AUTH_INPUT_MAC in its RENDEZVOUS1 cell +body. From these, the client finishes the handshake, validates +AUTH_INPUT_MAC, and computes the same NTOR_KEY_SEED. + +Now that both parties have the same NTOR_KEY_SEED, they can derive +the shared key material they will use for their circuit. + diff --git a/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md b/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md new file mode 100644 index 0000000..45ebc10 --- /dev/null +++ b/spec/rend-spec-v3/two-methods-for-managing-revision-counters.md @@ -0,0 +1,81 @@ + +# Appendix F: Two methods for managing revision counters. + +Implementations MAY generate revision counters in any way they please, +so long as they are monotonically increasing over the lifetime of each +blinded public key. But to avoid fingerprinting, implementors SHOULD +choose a strategy also used by other Tor implementations. Here we +describe two, and additionally list some strategies that implementors +should NOT use. + +F.1. Increment-on-generation + +```text + This is the simplest strategy, and the one used by Tor through at + least version 0.3.4.0-alpha. + + Whenever using a new blinded key, the service records the + highest revision counter it has used with that key. When generating + a descriptor, the service uses the smallest non-negative number + higher than any number it has already used. + + In other words, the revision counters under this system start fresh + with each blinded key as 0, 1, 2, 3, and so on. + + F.2. Encrypted time in period + + This scheme is what we recommend for situations when multiple + service instances need to coordinate their revision counters, + without an actual coordination mechanism. + + Let T be the number of seconds that have elapsed since the descriptor + became valid, plus 1. (T must be at least 1.) Implementations can use the + number of seconds since the start time of the shared random protocol run + that corresponds to this descriptor. + + Let S be a secret that all the service providers share. For + example, it could be the private signing key corresponding to the + current blinded key. + + Let K be an AES-256 key, generated as + K = H("rev-counter-generation" | S) + + Use K, and AES in counter mode with IV=0, to generate a stream of T + * 2 bytes. Consider these bytes as a sequence of T 16-bit + little-endian words. Add these words. + + Let the sum of these words be the revision counter. + + Cryptowiki attributes roughly this scheme to G. Bebek in: + + G. Bebek. Anti-tamper database research: Inference control + techniques. Technical Report EECS 433 Final Report, Case + Western Reserve University, November 2002. + + Although we believe it is suitable for use in this application, it + is not a perfect order-preserving encryption algorithm (and all + order-preserving encryption has weaknesses). Please think twice + before using it for anything else. + + (This scheme can be optimized pretty easily by caching the encryption of + X*1, X*2, X*3, etc for some well chosen X.) + + For a slow reference implementation, see src/test/ope_ref.py in the + Tor source repository. [XXXX for now, see the same file in Nick's + "ope_hax" branch -- it isn't merged yet.] + + This scheme is not currently implemented in Tor. + + F.X. Some revision-counter strategies to avoid + + Though it might be tempting, implementations SHOULD NOT use the + current time or the current time within the period directly as their + revision counter -- doing so leaks their view of the current time, + which can be used to link the onion service to other services run on + the same host. + + Similarly, implementations SHOULD NOT let the revision counter + increase forever without resetting it -- doing so links the service + across changes in the blinded public key. +``` + diff --git a/spec/socks-extensions.md b/spec/socks-extensions.md new file mode 100644 index 0000000..011c119 --- /dev/null +++ b/spec/socks-extensions.md @@ -0,0 +1,188 @@ +Tor's extensions to the SOCKS protocol + +Table of Contents + +```text + 1. Overview + 1.1. Extent of support + 2. Name lookup + 3. Other command extensions. + 4. HTTP-resistance + 5. Optimistic data + 6. Extended error codes +``` + + +# Overview + +The SOCKS protocol provides a generic interface for TCP proxies. Client +software connects to a SOCKS server via TCP, and requests a TCP connection +to another address and port. The SOCKS server establishes the connection, +and reports success or failure to the client. After the connection has +been established, the client application uses the TCP stream as usual. + +Tor supports SOCKS4 as defined in [1], SOCKS4A as defined in [2], and +SOCKS5 as defined in [3] and [4]. + +The stickiest issue for Tor in supporting clients, in practice, is forcing +DNS lookups to occur at the OR side: if clients do their own DNS lookup, +the DNS server can learn which addresses the client wants to reach. +SOCKS4 supports addressing by IPv4 address; SOCKS4A is a kludge on top of +SOCKS4 to allow addressing by hostname; SOCKS5 supports IPv4, IPv6, and +hostnames. + + +## Extent of support + +Tor supports the SOCKS4, SOCKS4A, and SOCKS5 standards, except as follows: + +BOTH: +- The BIND command is not supported. + +SOCKS4,4A: +- SOCKS4 usernames are used to implement stream isolation. + +```text + SOCKS5: + - The (SOCKS5) "UDP ASSOCIATE" command is not supported. + - SOCKS5 BIND command is not supported. + - IPv6 is not supported in CONNECT commands. + - SOCKS5 GSSAPI subnegotiation is not supported. + - The "NO AUTHENTICATION REQUIRED" (SOCKS5) authentication method [00] is + supported; and as of Tor 0.2.3.2-alpha, the "USERNAME/PASSWORD" (SOCKS5) + authentication method [02] is supported too, and used as a method to + implement stream isolation. As an extension to support some broken clients, + we allow clients to pass "USERNAME/PASSWORD" authentication message to us + even if no authentication was selected. Furthermore, we allow + username/password fields of this message to be empty. This technically + violates RFC1929 [4], but ensures interoperability with somewhat broken + SOCKS5 client implementations. + - Custom reply error code. The "REP" fields, as per the RFC[3], has + unassigned values which are used to describe Tor internal errors. See + ExtendedErrors in the tor.1 man page for more details. It is only sent + back if this SocksPort flag is set. +``` + +(For more information on stream isolation, see IsolateSOCKSAuth on the Tor +manpage.) + + +# Name lookup + +As an extension to SOCKS4A and SOCKS5, Tor implements a new command value, +"RESOLVE" [F0]. When Tor receives a "RESOLVE" SOCKS command, it initiates +a remote lookup of the hostname provided as the target address in the SOCKS +request. The reply is either an error (if the address couldn't be +resolved) or a success response. In the case of success, the address is +stored in the portion of the SOCKS response reserved for remote IP address. + +(We support RESOLVE in SOCKS4 too, even though it is unnecessary.) + +For SOCKS5 only, we support reverse resolution with a new command value, +"RESOLVE_PTR" [F1]. In response to a "RESOLVE_PTR" SOCKS5 command with +an IPv4 address as its target, Tor attempts to find the canonical +hostname for that IPv4 record, and returns it in the "server bound +address" portion of the reply. +(This command was not supported before Tor 0.1.2.2-alpha.) + + +# Other command extensions. + +Tor 0.1.2.4-alpha added a new command value: "CONNECT_DIR" [F2]. +In this case, Tor will open an encrypted direct TCP connection to the +directory port of the Tor server specified by address:port (the port +specified should be the ORPort of the server). It uses a one-hop tunnel +and a "BEGIN_DIR" relay cell to accomplish this secure connection. + +The F2 command value was removed in Tor 0.2.0.10-alpha in favor of a +new use_begindir flag in edge_connection_t. + + +# HTTP-resistance + +Tor checks the first byte of each SOCKS request to see whether it looks +more like an HTTP request (that is, it starts with a "G", "H", or "P"). If +so, Tor returns a small webpage, telling the user that his/her browser is +misconfigured. This is helpful for the many users who mistakenly try to +use Tor as an HTTP proxy instead of a SOCKS proxy. + + +# Optimistic data + +Tor allows SOCKS clients to send connection data before Tor has sent a +SOCKS response. When using an exit node that supports "optimistic data", +Tor will send such data to the server without waiting to see whether the +connection attempt succeeds. This behavior can save a single round-trip +time when starting connections with a protocol where the client speaks +first (like HTTP). Clients that do this must be ready to hear that +their connection has succeeded or failed _after_ they have sent the +data. + + +# Extended error codes + +We define a set of additional extension error codes that can be returned +by our SOCKS implementation in response to failed onion service +connections. + +(In the C Tor implementation, these error codes can be disabled +via the ExtendedErrors flag. In Arti, these error codes are enabled +whenever onion services are.) + +* X'F0' Onion Service Descriptor Can Not be Found + +```text + The requested onion service descriptor can't be found on the hashring + and thus not reachable by the client. + + * X'F1' Onion Service Descriptor Is Invalid + + The requested onion service descriptor can't be parsed or signature + validation failed. + + * X'F2' Onion Service Introduction Failed + + Client failed to introduce to the service meaning the descriptor was + found but the service is not anymore at the introduction points. The + service has likely changed its descriptor or is not running. + + * X'F3' Onion Service Rendezvous Failed + + Client failed to rendezvous with the service which means that the client + is unable to finalize the connection. + + * X'F4' Onion Service Missing Client Authorization + + Tor was able to download the requested onion service descriptor but is + unable to decrypt its content because it is missing client authorization + information for it. + + * X'F5' Onion Service Wrong Client Authorization + + Tor was able to download the requested onion service descriptor but is + unable to decrypt its content using the client authorization information + it has. This means the client access were revoked. + + * X'F6' Onion Service Invalid Address + + The given .onion address is invalid. In one of these cases this + error is returned: address checksum doesn't match, ed25519 public + key is invalid or the encoding is invalid. + + * X'F7' Onion Service Introduction Timed Out + + Similar to X'F2' code but in this case, all introduction attempts + have failed due to a time out. +``` + +(Note that not all of the above error codes are currently returned +by Arti as of August 2023.) + +```text +References: + [1] http://en.wikipedia.org/wiki/SOCKS#SOCKS4 + [2] http://en.wikipedia.org/wiki/SOCKS#SOCKS4a + [3] SOCKS5: RFC 1928 https://www.ietf.org/rfc/rfc1928.txt + [4] RFC 1929: https://www.ietf.org/rfc/rfc1929.txt +``` + diff --git a/spec/srv-spec-intro.md b/spec/srv-spec-intro.md new file mode 100644 index 0000000..d818603 --- /dev/null +++ b/spec/srv-spec-intro.md @@ -0,0 +1,50 @@ +Tor Shared Random Subsystem Specification + +This document specifies how the commit-and-reveal shared random subsystem of +Tor works. This text used to be proposal 250-commit-reveal-consensus.txt. + +Table Of Contents: + +```text + 1. Introduction + 1.1. Motivation + 1.2. Previous work + 2. Overview + 2.1. Introduction to our commit-and-reveal protocol + 2.2. Ten thousand feet view of the protocol + 2.3. How we use the consensus [CONS] + 2.3.1. Inserting Shared Random Values in the consensus + 2.4. Persistent State of the Protocol [STATE] + 2.5. Protocol Illustration + 3. Protocol + 3.1 Commitment Phase [COMMITMENTPHASE] + 3.1.1. Voting During Commitment Phase + 3.1.2. Persistent State During Commitment Phase [STATECOMMIT] + 3.2 Reveal Phase + 3.2.1. Voting During Reveal Phase + 3.2.2. Persistent State During Reveal Phase [STATEREVEAL] + 3.3. Shared Random Value Calculation At 00:00UTC + 3.3.1. Shared Randomness Calculation [SRCALC] + 3.4. Bootstrapping Procedure + 3.5. Rebooting Directory Authorities [REBOOT] + 4. Specification [SPEC] + 4.1. Voting + 4.1.1. Computing commitments and reveals [COMMITREVEAL] + 4.1.2. Validating commitments and reveals [VALIDATEVALUES] + 4.1.4. Encoding commit/reveal values in votes [COMMITVOTE] + 4.1.5. Shared Random Value [SRVOTE] + 4.2. Encoding Shared Random Values in the consensus [SRCONSENSUS] + 4.3. Persistent state format [STATEFORMAT] + 5. Security Analysis + 5.1. Security of commit-and-reveal and future directions + 5.2. Predicting the shared random value during reveal phase + 5.3. Partition attacks + 5.3.1. Partition attacks during commit phase + 5.3.2. Partition attacks during reveal phase + 6. Discussion + 6.1. Why the added complexity from proposal 225? + 6.2. Why do you do a commit-and-reveal protocol in 24 rounds? + 6.3. Why can't we recover if the 00:00UTC consensus fails? + 7. Acknowledgements +``` + diff --git a/spec/srv-spec/acknowledgements.md b/spec/srv-spec/acknowledgements.md new file mode 100644 index 0000000..171cfde --- /dev/null +++ b/spec/srv-spec/acknowledgements.md @@ -0,0 +1,29 @@ + +# Acknowledgements + +Thanks to everyone who has contributed to this design with feedback and +discussion. + +Thanks go to arma, ioerror, kernelcorn, nickm, s7r, Sebastian, teor, weasel +and everyone else! + +References: + +```text +[RANDOM-REFS]: + http://projectbullrun.org/dual-ec/ext-rand.html + https://lists.torproject.org/pipermail/tor-dev/2015-November/009954.html + +[RNGMESSAGING]: + https://moderncrypto.org/mail-archive/messaging/2015/002032.html + +[HOPPER]: + https://lists.torproject.org/pipermail/tor-dev/2014-January/006053.html + +[UNICORN]: + https://eprint.iacr.org/2015/366.pdf + +[VDFS]: + https://eprint.iacr.org/2018/601.pdf +``` + diff --git a/spec/srv-spec/discussion.md b/spec/srv-spec/discussion.md new file mode 100644 index 0000000..9d469fc --- /dev/null +++ b/spec/srv-spec/discussion.md @@ -0,0 +1,39 @@ + +# Discussion + + +## Why the added complexity from proposal 225? + +The complexity difference between this proposal and prop225 is in part +because prop225 doesn't specify how the shared random value gets to the +clients. This proposal spends lots of effort specifying how the two shared +random values can always be readily accessible to clients. + + +## Why do you do a commit-and-reveal protocol in 24 rounds? + +The reader might be wondering why we span the protocol over the course of a +whole day (24 hours), when only 3 rounds would be sufficient to generate a +shared random value. + +We decided to do it this way, because we piggyback on the Tor voting +protocol which also happens every hour. + +We could instead only do the shared randomness protocol from 21:00 to 00:00 +every day. Or to do it multiple times a day. + +However, we decided that since the shared random value needs to be in every +consensus anyway, carrying the commitments/reveals as well will not be a +big problem. Also, this way we give more chances for a failing dirauth to +recover and rejoin the protocol. + + +## Why can't we recover if the 00:00UTC consensus fails? + +If the 00:00UTC consensus fails, there will be no shared random value for +the whole day. In theory, we could recover by calculating the shared +randomness of the day at 01:00UTC instead. However, the engineering issues +with adding such recovery logic are too great. For example, it's not easy +for an authority who just booted to learn whether a specific consensus +failed to be created. + diff --git a/spec/srv-spec/introduction.md b/spec/srv-spec/introduction.md new file mode 100644 index 0000000..5ed203b --- /dev/null +++ b/spec/srv-spec/introduction.md @@ -0,0 +1,28 @@ + +# Introduction + + +## Motivation + +For the next generation hidden services project, we need the Tor network to +produce a fresh random value every day in such a way that it cannot be +predicted in advance or influenced by an attacker. + +Currently we need this random value to make the HSDir hash ring +unpredictable (#8244), which should resolve a wide class of hidden service +DoS attacks and should make it harder for people to gauge the popularity +and activity of target hidden services. Furthermore this random value can +be used by other systems in need of fresh global randomness like +Tor-related protocols (e.g. OnioNS) or even non-Tor-related (e.g. warrant +canaries). + + +## Previous work + +Proposal 225 specifies a commit-and-reveal protocol that can be run as an +external script and have the results be fed to the directory authorities. +However, directory authority operators feel unsafe running a third-party +script that opens TCP ports and accepts connections from the Internet. +Hence, this proposal aims to embed the commit-and-reveal idea in the Tor +voting process which should make it smoother to deploy and maintain. + diff --git a/spec/srv-spec/overview.md b/spec/srv-spec/overview.md new file mode 100644 index 0000000..728aad2 --- /dev/null +++ b/spec/srv-spec/overview.md @@ -0,0 +1,158 @@ + +# Overview + +This proposal alters the Tor consensus protocol such that a random number is +generated every midnight by the directory authorities during the regular voting +process. The distributed random generator scheme is based on the +commit-and-reveal technique. + +The proposal also specifies how the final shared random value is embedded +in consensus documents so that clients who need it can get it. + + +## Introduction to our commit-and-reveal protocol + +Every day, before voting for the consensus at 00:00UTC each authority +generates a new random value and keeps it for the whole day. The authority +cryptographically hashes the random value and calls the output its +"commitment" value. The original random value is called the "reveal" value. + +The idea is that given a reveal value you can cryptographically confirm that +it corresponds to a given commitment value (by hashing it). However given a +commitment value you should not be able to derive the underlying reveal +value. The construction of these values is specified in section [COMMITREVEAL]. + + +## Ten thousand feet view of the protocol + +Our commit-and-reveal protocol aims to produce a fresh shared random value +(denoted shared_random_value here and elsewhere) every day at 00:00UTC. The +final fresh random value is embedded in the consensus document at that +time. + +Our protocol has two phases and uses the hourly voting procedure of Tor. +Each phase lasts 12 hours, which means that 12 voting rounds happen in +between. In short, the protocol works as follows: + +Commit phase: + +```text + Starting at 00:00UTC and for a period of 12 hours, authorities every + hour include their commitment in their votes. They also include any + received commitments from other authorities, if available. + + Reveal phase: + + At 12:00UTC, the reveal phase starts and lasts till the end of the + protocol at 00:00UTC. In this stage, authorities must reveal the value + they committed to in the previous phase. The commitment and revealed + values from other authorities, when available, are also added to the + vote. + + Shared Randomness Calculation: + + At 00:00UTC, the shared random value is computed from the agreed + revealed values and added to the consensus. + + This concludes the commit-and-reveal protocol every day at 00:00UTC. +``` + + +## How we use the consensus [CONS] + +The produced shared random values need to be readily available to +clients. For this reason we include them in the consensus documents. + +Every hour the consensus documents need to include the shared random value +of the day, as well as the shared random value of the previous day. That's +because either of these values might be needed at a given time for a Tor +client to access a hidden service according to section [TIME-OVERLAP] of +proposal 224. This means that both of these two values need to be included +in votes as well. + +Hence, consensuses need to include: + +```text + (a) The shared random value of the current time period. + (b) The shared random value of the previous time period. +``` + +For this, a new SR consensus method will be needed to indicate which +authorities support this new protocol. + + +### Inserting Shared Random Values in the consensus + +After voting happens, we need to be careful on how we pick which shared +random values (SRV) to put in the consensus, to avoid breaking the consensus +because of authorities having different views of the commit-and-reveal +protocol (because maybe they missed some rounds of the protocol). + +For this reason, authorities look at the received votes before creating a +consensus and employ the following logic: + +```text + - First of all, they make sure that the agreed upon consensus method is + above the SR consensus method. + + - Authorities include an SRV in the consensus if and only if the SRV has + been voted by at least the majority of authorities. + + - For the consensus at 00:00UTC, authorities include an SRV in the consensus + if and only if the SRV has been voted by at least AuthDirNumAgreements + authorities (where AuthDirNumAgreements is a newly introduced consensus + parameter). +``` + +Authorities include in the consensus the most popular SRV that also +satisfies the above constraints. Otherwise, no SRV should be included. + +The above logic is used to make it harder to break the consensus by natural +partioning causes. + +We use the AuthDirNumAgreements consensus parameter to enforce that a +_supermajority_ of dirauths supports the SR protocol during SRV creation, so +that even if a few of those dirauths drop offline in the middle of the run +the SR protocol does not get disturbed. We go to extra lengths to ensure +this because changing SRVs in the middle of the day has terrible +reachability consequences for hidden service clients. + + +## Persistent State of the Protocol [STATE] + +A directory authority needs to keep a persistent state on disk of the on +going protocol run. This allows an authority to join the protocol seamlessly +in the case of a reboot. + +During the commitment phase, it is populated with the commitments of all +authorities. Then during the reveal phase, the reveal values are also +stored in the state. + +As discussed previously, the shared random values from the current and +previous time period must also be present in the state at all times if they +are available. + + +## Protocol Illustration + +An illustration for better understanding the protocol can be found here: + +https://people.torproject.org/~asn/hs_notes/shared_rand.jpg + +It reads left-to-right. + +The illustration displays what the authorities (A_1, A_2, A_3) put in their +votes. A chain 'A_1 -> c_1 -> r_1' denotes that authority A_1 committed to +the value c_1 which corresponds to the reveal value r_1. + +The illustration depicts only a few rounds of the whole protocol. It starts +with the first three rounds of the commit phase, then it jumps to the last +round of the commit phase. It continues with the first two rounds of the +reveal phase and then it jumps to the final round of the protocol run. It +finally shows the first round of the commit phase of the next protocol run +(00:00UTC) where the final Shared Random Value is computed. In our fictional +example, the SRV was computed with 3 authority contributions and its value +is "a56fg39h". + +We advice you to revisit this after you have read the whole document. + diff --git a/spec/srv-spec/protocol.md b/spec/srv-spec/protocol.md new file mode 100644 index 0000000..4567f89 --- /dev/null +++ b/spec/srv-spec/protocol.md @@ -0,0 +1,155 @@ + +# Protocol + +In this section we give a detailed specification of the protocol. We +describe the protocol participants' logic and the messages they send. The +encoding of the messages is specified in the next section ([SPEC]). + +Now we go through the phases of the protocol: + + +## Commitment Phase [COMMITMENTPHASE] + +The commit phase lasts from 00:00UTC to 12:00UTC. + +During this phase, an authority commits a value in its vote and +saves it to the permanent state as well. + +Authorities also save any received authoritative commits by other authorities +in their permanent state. We call a commit by Alice "authoritative" if it was +included in Alice's vote. + + +### Voting During Commitment Phase + +During the commit phase, each authority includes in its votes: + +```text + - The commitment value for this protocol run. + - Any authoritative commitments received from other authorities. + - The two previous shared random values produced by the protocol (if any). +``` + +The commit phase lasts for 12 hours, so authorities have multiple chances to +commit their values. An authority MUST NOT commit a second value during a +subsequent round of the commit phase. + +If an authority publishes a second commitment value in the same commit +phase, only the first commitment should be taken in account by other +authorities. Any subsequent commitments MUST be ignored. + + +### Persistent State During Commitment Phase [STATECOMMIT] + +During the commitment phase, authorities save in their persistent state the +authoritative commits they have received from each authority. Only one commit +per authority must be considered trusted and active at a given time. + + +## Reveal Phase + +The reveal phase lasts from 12:00UTC to 00:00UTC. + +Now that the commitments have been agreed on, it's time for authorities to +reveal their random values. + + +### Voting During Reveal Phase + +During the reveal phase, each authority includes in its votes: + +```text + - Its reveal value that was previously committed in the commit phase. + - All the commitments and reveals received from other authorities. + - The two previous shared random values produced by the protocol (if any). +``` + +The set of commitments have been decided during the commitment +phase and must remain the same. If an authority tries to change its +commitment during the reveal phase or introduce a new commitment, +the new commitment MUST be ignored. + + +### Persistent State During Reveal Phase [STATEREVEAL] + +During the reveal phase, authorities keep the authoritative commits from the +commit phase in their persistent state. They also save any received reveals +that correspond to authoritative commits and are valid (as specified in +[VALIDATEVALUES]). + +An authority that just received a reveal value from another authority's vote, +MUST wait till the next voting round before including that reveal value in +its votes. + + +## Shared Random Value Calculation At 00:00UTC + +Finally, at 00:00UTC every day, authorities compute a fresh shared random +value and this value must be added to the consensus so clients can use it. + +Authorities calculate the shared random value using the reveal values in +their state as specified in subsection [SRCALC]. + +Authorities at 00:00UTC start including this new shared random value in +their votes, replacing the one from two protocol runs ago. Authorities also +start including this new shared random value in the consensus as well. + +Apart from that, authorities at 00:00UTC proceed voting normally as they +would in the first round of the commitment phase (section [COMMITMENTPHASE]). + + +### Shared Randomness Calculation [SRCALC] + +An authority that wants to derive the shared random value SRV, should use +the appropriate reveal values for that time period and calculate SRV as +follows. + +HASHED_REVEALS = H(ID_a | R_a | ID_b | R_b | ..) + +```text + SRV = SHA3-256("shared-random" | INT_8(REVEAL_NUM) | INT_4(VERSION) | + HASHED_REVEALS | PREVIOUS_SRV) +``` + +where the ID_a value is the identity key fingerprint of authority 'a' and R_a +is the corresponding reveal value of that authority for the current period. + +Also, REVEAL_NUM is the number of revealed values in this construction, +VERSION is the protocol version number and PREVIOUS_SRV is the previous +shared random value. If no previous shared random value is known, then +PREVIOUS_SRV is set to 32 NUL (\x00) bytes. + +To maintain consistent ordering in HASHED_REVEALS, all the ID_a | R_a pairs +are ordered based on the R_a value in ascending order. + + +## Bootstrapping Procedure + +As described in [CONS], two shared random values are required for the HSDir +overlay periods to work properly as specified in proposal 224. Hence +clients MUST NOT use the randomness of this system till it has bootstrapped +completely; that is, until two shared random values are included in a +consensus. This should happen after three 00:00UTC consensuses have been +produced, which takes 48 hours. + + +## Rebooting Directory Authorities [REBOOT] + +The shared randomness protocol must be able to support directory +authorities who leave or join in the middle of the protocol execution. + +An authority that commits in the Commitment Phase and then leaves MUST have +stored its reveal value on disk so that it continues participating in the +protocol if it returns before or during the Reveal Phase. The reveal value +MUST be stored timestamped to avoid sending it on wrong protocol runs. + +An authority that misses the Commitment Phase cannot commit anymore, so it's +unable to participate in the protocol for that run. Same goes for an +authority that misses the Reveal phase. Authorities who do not participate in +the protocol SHOULD still carry commits and reveals of others in their vote. + +Finally, authorities MUST implement their persistent state in such a way that they +will never commit two different values in the same protocol run, even if they +have to reboot in the middle (assuming that their persistent state file is +kept). A suggested way to structure the persistent state is found at [STATEFORMAT]. + diff --git a/spec/srv-spec/security-analysis.md b/spec/srv-spec/security-analysis.md new file mode 100644 index 0000000..698cac5 --- /dev/null +++ b/spec/srv-spec/security-analysis.md @@ -0,0 +1,98 @@ + +# Security Analysis + + +## Security of commit-and-reveal and future directions + +The security of commit-and-reveal protocols is well understood, and has +certain flaws. Basically, the protocol is insecure to the extent that an +adversary who controls b of the authorities gets to choose among 2^b +outcomes for the result of the protocol. However, an attacker who is not a +dirauth should not be able to influence the outcome at all. + +We believe that this system offers sufficient security especially compared +to the current situation. More secure solutions require much more advanced +crypto and more complex protocols so this seems like an acceptable solution +for now. + +Here are some examples of possible future directions: +- Schemes based on threshold signatures (e.g. see [HOPPER]) +- Unicorn scheme by Lenstra et al. [UNICORN] +- Schemes based on Verifiable Delay Functions [VDFS] + +For more alternative approaches on collaborative random number generation +also see the discussion at [RNGMESSAGING]. + + +## Predicting the shared random value during reveal phase + +The reveal phase lasts 12 hours, and most authorities will send their +reveal value on the first round of the reveal phase. This means that an +attacker can predict the final shared random value about 12 hours before +it's generated. + +This does not pose a problem for the HSDir hash ring, since we impose an +higher uptime restriction on HSDir nodes, so 12 hours predictability is not +an issue. + +Any other protocols using the shared random value from this system should +be aware of this property. + + +## Partition attacks + +This design is not immune to certain partition attacks. We believe they +don't offer much gain to an attacker as they are very easy to detect and +difficult to pull off since an attacker would need to compromise a directory +authority at the very least. Also, because of the byzantine general problem, +it's very hard (even impossible in some cases) to protect against all such +attacks. Nevertheless, this section describes all possible partition attack +and how to detect them. + + +### Partition attacks during commit phase + +A malicious directory authority could send only its commit to one single +authority which results in that authority having an extra commit value for +the shared random calculation that the others don't have. Since the +consensus needs majority, this won't affect the final SRV value. However, +the attacker, using this attack, could remove a single directory authority +from the consensus decision at 24:00 when the SRV is computed. + +An attacker could also partition the authorities by sending two different +commitment values to different authorities during the commit phase. + +All of the above is fairly easy to detect. Commitment values in the vote +coming from an authority should NEVER be different between authorities. If +so, this means an attack is ongoing or very bad bug (highly unlikely). + + +### Partition attacks during reveal phase + +Let's consider Alice, a malicious directory authority. Alice could wait +until the last reveal round, and reveal its value to half of the +authorities. That would partition the authorities into two sets: the ones +who think that the shared random value should contain this new reveal, and +the rest who don't know about it. This would result in a tie and two +different shared random value. + +A similar attack is possible. For example, two rounds before the end of the +reveal phase, Alice could advertise her reveal value to only half of the +dirauths. This way, in the last reveal phase round, half of the dirauths +will include that reveal value in their votes and the others will not. In +the end of the reveal phase, half of the dirauths will calculate a +different shared randomness value than the others. + +We claim that this attack is not particularly fruitful: Alice ends up +having two shared random values to choose from which is a fundamental +problem of commit-and-reveal protocols as well (since the last person can +always abort or reveal). The attacker can also sabotage the consensus, but +there are other ways this can be done with the current voting system. + +Furthermore, we claim that such an attack is very noisy and detectable. +First of all, it requires the authority to sabotage two consensuses which +will cause quite some noise. Furthermore, the authority needs to send +different votes to different auths which is detectable. Like the commit +phase attack, the detection here is to make sure that the commitment values +in a vote coming from an authority are always the same for each authority. + diff --git a/spec/srv-spec/specification-spec.md b/spec/srv-spec/specification-spec.md new file mode 100644 index 0000000..1ff7db3 --- /dev/null +++ b/spec/srv-spec/specification-spec.md @@ -0,0 +1,159 @@ + +# Specification [SPEC] + + +## Voting + +This section describes how commitments, reveals and SR values are encoded in +votes. We describe how to encode both the authority's own +commitments/reveals and also the commitments/reveals received from the other +authorities. Commitments and reveals share the same line, but reveals are +optional. + +Participating authorities need to include the line: + +"shared-rand-participate" + +in their votes to announce that they take part in the protocol. + + +### Computing commitments and reveals [COMMITREVEAL] + +A directory authority that wants to participate in this protocol needs to +create a new pair of commitment/reveal values for every protocol +run. Authorities SHOULD generate a fresh pair of such values right before the +first commitment phase of the day (at 00:00UTC). + +The value REVEAL is computed as follows: + +REVEAL = base64-encode( TIMESTAMP || H(RN) ) + +```text + where RN is the SHA3 hashed value of a 256-bit random value. We hash the + random value to avoid exposing raw bytes from our PRNG to the network (see + [RANDOM-REFS]). + + TIMESTAMP is an 8-bytes network-endian time_t value. Authorities SHOULD + set TIMESTAMP to the valid-after time of the vote document they first plan + to publish their commit into (so usually at 00:00UTC, except if they start + up in a later commit round). + + The value COMMIT is computed as follows: + + COMMIT = base64-encode( TIMESTAMP || H(REVEAL) ) +``` + + +### Validating commitments and reveals [VALIDATEVALUES] + +Given a COMMIT message and a REVEAL message it should be possible to verify +that they indeed correspond. To do so, the client extracts the random value +H(RN) from the REVEAL message, hashes it, and compares it with the H(H(RN)) +from the COMMIT message. We say that the COMMIT and REVEAL messages +correspond, if the comparison was successful. + +Participants MUST also check that corresponding COMMIT and REVEAL values +have the same timestamp value. + +Authorities should ignore reveal values during the Reveal Phase that don't +correspond to commit values published during the Commitment Phase. + + +### Encoding commit/reveal values in votes [COMMITVOTE] + +An authority puts in its vote the commitments and reveals it has produced and +seen from the other authorities. To do so, it includes the following in its +votes: + +"shared-rand-commit" SP VERSION SP ALGNAME SP IDENTITY SP COMMIT [SP REVEAL] NL + +where VERSION is the version of the protocol the commit was created with. +IDENTITY is the authority's SHA1 identity fingerprint and COMMIT is the +encoded commit [COMMITREVEAL]. Authorities during the reveal phase can +also optionally include an encoded reveal value REVEAL. There MUST be only +one line per authority else the vote is considered invalid. Finally, the +ALGNAME is the hash algorithm that should be used to compute COMMIT and +REVEAL which is "sha3-256" for version 1. + + +### Shared Random Value [SRVOTE] + +Authorities include a shared random value (SRV) in their votes using the +following encoding for the previous and current value respectively: + +```text + "shared-rand-previous-value" SP NUM_REVEALS SP VALUE NL + "shared-rand-current-value" SP NUM_REVEALS SP VALUE NL +``` + +where VALUE is the actual shared random value encoded in hex (computed as +specified in section [SRCALC]. NUM_REVEALS is the number of reveal values +used to generate this SRV. + +To maintain consistent ordering, the shared random values of the previous +period should be listed before the values of the current period. + + +## Encoding Shared Random Values in the consensus [SRCONSENSUS] + +Authorities insert the two active shared random values in the consensus +following the same encoding format as in [SRVOTE]. + + +## Persistent state format [STATEFORMAT] + +As a way to keep ground truth state in this protocol, an authority MUST +keep a persistent state of the protocol. The next sub-section suggest a +format for this state which is the same as the current state file format. + +It contains a preamble, a commitment and reveal section and a list of +shared random values. + +The preamble (or header) contains the following items. They MUST occur in +the order given here: + +"Version" SP version NL + +[At start, exactly once.] + +A document format version. For this specification, version is "1". + +"ValidUntil" SP YYYY-MM-DD SP HH:MM:SS NL + +[Exactly once] + +```text + After this time, this state is expired and shouldn't be used nor + trusted. The validity time period is till the end of the current + protocol run (the upcoming noon). +``` + +The following details the commitment and reveal section. They are encoded +the same as in the vote. This makes it easier for implementation purposes. + +"Commit" SP version SP algname SP identity SP commit [SP reveal] NL + +[Exactly once per authority] + +The values are the same as detailed in section [COMMITVOTE]. + +This line is also used by an authority to store its own value. + +Finally is the shared random value section. + +"SharedRandPreviousValue" SP num_reveals SP value NL + +[At most once] + +```text + This is the previous shared random value agreed on at the previous + period. The fields are the same as in section [SRVOTE]. + + "SharedRandCurrentValue" SP num_reveals SP value NL + + [At most once] + + This is the latest shared random value. The fields are the same as in + section [SRVOTE]. +``` + diff --git a/spec/tor-spec-intro.md b/spec/tor-spec-intro.md new file mode 100644 index 0000000..eb9a32b --- /dev/null +++ b/spec/tor-spec-intro.md @@ -0,0 +1,91 @@ +Tor Protocol Specification + +```text + Roger Dingledine + Nick Mathewson + +Table of Contents + + 0. Preliminaries + 0.1. Notation and encoding + 0.2. Security parameters + 0.3. Ciphers + 0.4. A bad hybrid encryption algorithm, for legacy purposes + 1. System overview + 1.1. Keys and names + 2. Connections + 2.1. Picking TLS ciphersuites + 2.2. TLS security considerations + 3. Cell Packet format + 4. Negotiating and initializing connections + 4.1. Negotiating versions with VERSIONS cells + 4.2. CERTS cells + 4.3. AUTH_CHALLENGE cells + 4.4. AUTHENTICATE cells + 4.4.1. Link authentication type 1: RSA-SHA256-TLSSecret + 4.4.2. Link authentication type 3: Ed25519-SHA256-RFC5705 + 4.5. NETINFO cells + 5. Circuit management + 5.1. CREATE and CREATED cells + 5.1.1. Choosing circuit IDs in create cells + 5.1.2. EXTEND and EXTENDED cells + 5.1.3. The "TAP" handshake + 5.1.4. The "ntor" handshake + 5.1.4.1. The "ntor-v3" handshake. + 5.1.5. CREATE_FAST/CREATED_FAST cells + 5.1.6. Additional data in CREATE/CREATED cells + 5.2. Setting circuit keys + 5.2.1. KDF-TOR + 5.2.2. KDF-RFC5869 + 5.3. Creating circuits + 5.3.1. Canonical connections + 5.4. Tearing down circuits + 5.5. Routing relay cells + 5.5.1. Circuit ID Checks + 5.5.2. Forward Direction + 5.5.2.1. Routing from the Origin + 5.5.2.2. Relaying Forward at Onion Routers + 5.5.3. Backward Direction + 5.5.3.1. Relaying Backward at Onion Routers + 5.5.4. Routing to the Origin + 5.6. Handling relay_early cells + 6. Application connections and stream management + 6.1. Relay cells + 6.1.1. Calculating the 'Digest' field + 6.2. Opening streams and transferring data + 6.2.1. Opening a directory stream + 6.3. Closing streams + 6.4. Remote hostname lookup + 7. Flow control + 7.1. Link throttling + 7.2. Link padding + 7.3. Circuit-level flow control + 7.3.1. SENDME Cell Format + 7.4. Stream-level flow control + 8. Handling resource exhaustion + 8.1. Memory exhaustion + 9. Subprotocol versioning + 9.1. "Link" + 9.2. "LinkAuth" + 9.3. "Relay" + 9.4. "HSIntro" + 9.5. "HSRend" + 9.6. "HSDir" + 9.7. "DirCache" + 9.8. "Desc" + 9.9. "Microdesc" + 9.10. "Cons" + 9.11. "Padding" + 9.12. "FlowCtrl" +``` + +Note: This document aims to specify Tor as currently implemented, though it +may take it a little time to become fully up to date. Future versions of Tor +may implement improved protocols, and compatibility is not guaranteed. +We may or may not remove compatibility notes for other obsolete versions of +Tor as they become obsolete. + +This specification is not a design document; most design criteria +are not examined. For more information on why Tor acts as it does, +see tor-design.pdf. + diff --git a/spec/tor-spec/application-connections-stream-management.md b/spec/tor-spec/application-connections-stream-management.md new file mode 100644 index 0000000..53b9b97 --- /dev/null +++ b/spec/tor-spec/application-connections-stream-management.md @@ -0,0 +1,3 @@ + +# Application connections and stream management + diff --git a/spec/tor-spec/cell-packet-format.md b/spec/tor-spec/cell-packet-format.md new file mode 100644 index 0000000..b26f012 --- /dev/null +++ b/spec/tor-spec/cell-packet-format.md @@ -0,0 +1,124 @@ + +# Cell Packet format + +The basic unit of communication for onion routers and onion +proxies is a fixed-width "cell". + +On a version 1 connection, each cell contains the following +fields: + +```text + CircID [CIRCID_LEN bytes] + Command [1 byte] + Payload (padded with padding bytes) [PAYLOAD_LEN bytes] +``` + +On a version 2 or higher connection, all cells are as in version 1 +connections, except for variable-length cells, whose format is: + +```text + CircID [CIRCID_LEN octets] + Command [1 octet] + Length [2 octets; big-endian integer] + Payload (some commands MAY pad) [Length bytes] +``` + +Most variable-length cells MAY be padded with padding bytes, except +for VERSIONS cells, which MUST NOT contain any additional bytes. +(The payload of VPADDING cells consists of padding bytes.) + +On a version 2 connection, variable-length cells are indicated by a +command byte equal to 7 ("VERSIONS"). On a version 3 or +higher connection, variable-length cells are indicated by a command +byte equal to 7 ("VERSIONS"), or greater than or equal to 128. + +CIRCID_LEN is 2 for link protocol versions 1, 2, and 3. CIRCID_LEN +is 4 for link protocol version 4 or higher. The first VERSIONS cell, +and any cells sent before the first VERSIONS cell, always have +CIRCID_LEN == 2 for backward compatibility. + +The CircID field determines which circuit, if any, the cell is +associated with. + +The 'Command' field of a fixed-length cell holds one of the following +values: + +```text + 0 -- PADDING (Padding) (See Sec 7.2) + 1 -- CREATE (Create a circuit) (See Sec 5.1) + 2 -- CREATED (Acknowledge create) (See Sec 5.1) + 3 -- RELAY (End-to-end data) (See Sec 5.5 and 6) + 4 -- DESTROY (Stop using a circuit) (See Sec 5.4) + 5 -- CREATE_FAST (Create a circuit, no KP) (See Sec 5.1) + 6 -- CREATED_FAST (Circuit created, no KP) (See Sec 5.1) + 8 -- NETINFO (Time and address info) (See Sec 4.5) + 9 -- RELAY_EARLY (End-to-end data; limited)(See Sec 5.6) + 10 -- CREATE2 (Extended CREATE cell) (See Sec 5.1) + 11 -- CREATED2 (Extended CREATED cell) (See Sec 5.1) + 12 -- PADDING_NEGOTIATE (Padding negotiation) (See Sec 7.2) + + Variable-length command values are: + + 7 -- VERSIONS (Negotiate proto version) (See Sec 4) + 128 -- VPADDING (Variable-length padding) (See Sec 7.2) + 129 -- CERTS (Certificates) (See Sec 4.2) + 130 -- AUTH_CHALLENGE (Challenge value) (See Sec 4.3) + 131 -- AUTHENTICATE (Client authentication)(See Sec 4.5) + 132 -- AUTHORIZE (Client authorization) (Not yet used) + + The interpretation of 'Payload' depends on the type of the cell. + + VPADDING/PADDING: + Payload contains padding bytes. + CREATE/CREATE2: Payload contains the handshake challenge. + CREATED/CREATED2: Payload contains the handshake response. + RELAY/RELAY_EARLY: Payload contains the relay header and relay body. + DESTROY: Payload contains a reason for closing the circuit. + (see 5.4) +``` + +Upon receiving any other value for the command field, an OR must +drop the cell. Since more cell types may be added in the future, ORs +should generally not warn when encountering unrecognized commands. + +The cell is padded up to the cell length with padding bytes. + +Senders set padding bytes depending on the cell's command: + +```text + VERSIONS: Payload MUST NOT contain padding bytes. + AUTHORIZE: Payload is unspecified and reserved for future use. + Other variable-length cells: + Payload MAY contain padding bytes at the end of the cell. + Padding bytes SHOULD be set to NUL. + RELAY/RELAY_EARLY: Payload MUST be padded to PAYLOAD_LEN with padding + bytes. Padding bytes SHOULD be set to random values. + Other fixed-length cells: + Payload MUST be padded to PAYLOAD_LEN with padding bytes. + Padding bytes SHOULD be set to NUL. +``` + +We recommend random padding in RELAY/RELAY_EARLY cells, so that the cell +content is unpredictable. See the format of relay cells in section 6.1 +for detail. + +For other cells, TLS authenticates cell content, so randomized padding +bytes are redundant. + +Receivers MUST ignore padding bytes. + +PADDING cells are currently used to implement connection keepalive. +If there is no other traffic, ORs and OPs send one another a PADDING +cell every few minutes. + +CREATE, CREATE2, CREATED, CREATED2, and DESTROY cells are used to +manage circuits; see section 5 below. + +RELAY cells are used to send commands and data along a circuit; see +section 6 below. + +VERSIONS and NETINFO cells are used to set up connections in link +protocols v2 and higher; in link protocol v3 and higher, CERTS, +AUTH_CHALLENGE, and AUTHENTICATE may also be used. See section 4 +below. + diff --git a/spec/tor-spec/circuit-management.md b/spec/tor-spec/circuit-management.md new file mode 100644 index 0000000..3784bab --- /dev/null +++ b/spec/tor-spec/circuit-management.md @@ -0,0 +1,3 @@ + +# Circuit management + diff --git a/spec/tor-spec/closing-streams.md b/spec/tor-spec/closing-streams.md new file mode 100644 index 0000000..c85b3cc --- /dev/null +++ b/spec/tor-spec/closing-streams.md @@ -0,0 +1,100 @@ + +## Closing streams + +When an anonymized TCP connection is closed, or an edge node +encounters error on any stream, it sends a 'RELAY_END' cell along the +circuit (if possible) and closes the TCP connection immediately. If +an edge node receives a 'RELAY_END' cell for any stream, it closes +the TCP connection completely, and sends nothing more along the +circuit for that stream. + +The payload of a RELAY_END cell begins with a single 'reason' byte to +describe why the stream is closing. For some reasons, it contains +additional data (depending on the reason.) The values are: + +```text + 1 -- REASON_MISC (catch-all for unlisted reasons) + 2 -- REASON_RESOLVEFAILED (couldn't look up hostname) + 3 -- REASON_CONNECTREFUSED (remote host refused connection) [*] + 4 -- REASON_EXITPOLICY (OR refuses to connect to host or port) + 5 -- REASON_DESTROY (Circuit is being destroyed) + 6 -- REASON_DONE (Anonymized TCP connection was closed) + 7 -- REASON_TIMEOUT (Connection timed out, or OR timed out + while connecting) + 8 -- REASON_NOROUTE (Routing error while attempting to + contact destination) + 9 -- REASON_HIBERNATING (OR is temporarily hibernating) + 10 -- REASON_INTERNAL (Internal error at the OR) + 11 -- REASON_RESOURCELIMIT (OR has no resources to fulfill request) + 12 -- REASON_CONNRESET (Connection was unexpectedly reset) + 13 -- REASON_TORPROTOCOL (Sent when closing connection because of + Tor protocol violations.) + 14 -- REASON_NOTDIRECTORY (Client sent RELAY_BEGIN_DIR to a + non-directory relay.) + + [*] Older versions of Tor also send this reason when connections are + reset. +``` + +OPs and ORs MUST accept reasons not on the above list, since future +versions of Tor may provide more fine-grained reasons. + +For most reasons, the format of RELAY_END is: + +Reason [1 byte] + +For REASON_EXITPOLICY, the format of RELAY_END is: + +```text + Reason [1 byte] + IPv4 or IPv6 address [4 bytes or 16 bytes] + TTL [4 bytes] +``` + +(If the TTL is absent, it should be treated as if it were 0xffffffff. +If the address is absent or is the wrong length, the RELAY_END message +should be processed anyway.) + +Tors SHOULD NOT send any reason except REASON_MISC for a stream that they +have originated. + +Implementations SHOULD accept empty RELAY_END messages, and treat them +as if they specified REASON_MISC. + +Upon receiving a RELAY_END cell, the recipient may be sure that no further +cells will arrive on that stream, and can treat such cells as a protocol +violation. + +After sending a RELAY_END cell, the sender needs to give the recipient +time to receive that cell. In the meantime, the sender SHOULD remember +how many cells of which types (CONNECTED, SENDME, DATA) that it would have +accepted on that stream, and SHOULD kill the circuit if it receives more +than permitted. + +--- [The rest of this section describes unimplemented functionality.] + +Because TCP connections can be half-open, we follow an equivalent +to TCP's FIN/FIN-ACK/ACK protocol to close streams. + +An exit (or onion service) connection can have a TCP stream in one of +three states: 'OPEN', 'DONE_PACKAGING', and 'DONE_DELIVERING'. For the +purposes of modeling transitions, we treat 'CLOSED' as a fourth state, +although connections in this state are not, in fact, tracked by the +onion router. + +A stream begins in the 'OPEN' state. Upon receiving a 'FIN' from +the corresponding TCP connection, the edge node sends a 'RELAY_FIN' +cell along the circuit and changes its state to 'DONE_PACKAGING'. +Upon receiving a 'RELAY_FIN' cell, an edge node sends a 'FIN' to +the corresponding TCP connection (e.g., by calling +shutdown(SHUT_WR)) and changing its state to 'DONE_DELIVERING'. + +When a stream in already in 'DONE_DELIVERING' receives a 'FIN', it +also sends a 'RELAY_FIN' along the circuit, and changes its state +to 'CLOSED'. When a stream already in 'DONE_PACKAGING' receives a +'RELAY_FIN' cell, it sends a 'FIN' and changes its state to +'CLOSED'. + +If an edge node encounters an error on any stream, it sends a +'RELAY_END' cell (if possible) and closes the stream immediately. + diff --git a/spec/tor-spec/connections.md b/spec/tor-spec/connections.md new file mode 100644 index 0000000..d540489 --- /dev/null +++ b/spec/tor-spec/connections.md @@ -0,0 +1,229 @@ + +# Connections + +Connections between two Tor relays, or between a client and a relay, +use TLS/SSLv3 for link authentication and encryption. All +implementations MUST support the SSLv3 ciphersuite +"TLS_DHE_RSA_WITH_AES_128_CBC_SHA" if it is available. They SHOULD +support better ciphersuites if available. + +There are three ways to perform TLS handshakes with a Tor server. In +the first way, "certificates-up-front", both the initiator and +responder send a two-certificate chain as part of their initial +handshake. (This is supported in all Tor versions.) In the second +way, "renegotiation", the responder provides a single certificate, +and the initiator immediately performs a TLS renegotiation. (This is +supported in Tor 0.2.0.21 and later.) And in the third way, +"in-protocol", the initial TLS negotiation completes, and the +parties bootstrap themselves to mutual authentication via use of the +Tor protocol without further TLS handshaking. (This is supported in +0.2.3.6-alpha and later.) + +Each of these options provides a way for the parties to learn it is +available: a client does not need to know the version of the Tor +server in order to connect to it properly. + +In "certificates up-front" (a.k.a "the v1 handshake"), +the connection initiator always sends a +two-certificate chain, consisting of an X.509 certificate using a +short-term connection public key and a second, self-signed X.509 +certificate containing its identity key. The other party sends a similar +certificate chain. The initiator's ClientHello MUST NOT include any +ciphersuites other than: + +```text + TLS_DHE_RSA_WITH_AES_256_CBC_SHA + TLS_DHE_RSA_WITH_AES_128_CBC_SHA + SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA +``` + +In "renegotiation" (a.k.a. "the v2 handshake"), +the connection initiator sends no certificates, and +the responder sends a single connection certificate. Once the TLS +handshake is complete, the initiator renegotiates the handshake, with each +party sending a two-certificate chain as in "certificates up-front". +The initiator's ClientHello MUST include at least one ciphersuite not in +the list above -- that's how the initiator indicates that it can +handle this handshake. For other considerations on the initiator's +ClientHello, see section 2.1 below. + +In "in-protocol" (a.k.a. "the v3 handshake"), the initiator sends no +certificates, and the +responder sends a single connection certificate. The choice of +ciphersuites must be as in a "renegotiation" handshake. There are +additionally a set of constraints on the connection certificate, +which the initiator can use to learn that the in-protocol handshake +is in use. Specifically, at least one of these properties must be +true of the certificate: + +```text + * The certificate is self-signed + * Some component other than "commonName" is set in the subject or + issuer DN of the certificate. + * The commonName of the subject or issuer of the certificate ends + with a suffix other than ".net". + * The certificate's public key modulus is longer than 1024 bits. +``` + +The initiator then sends a VERSIONS cell to the responder, which then +replies with a VERSIONS cell; they have then negotiated a Tor +protocol version. Assuming that the version they negotiate is 3 or higher +(the only ones specified for use with this handshake right now), the +responder sends a CERTS cell, an AUTH_CHALLENGE cell, and a NETINFO +cell to the initiator, which may send either CERTS, AUTHENTICATE, +NETINFO if it wants to authenticate, or just NETINFO if it does not. + +For backward compatibility between later handshakes and "certificates +up-front", the ClientHello of an initiator that supports a later +handshake MUST include at least one ciphersuite other than those listed +above. The connection responder examines the initiator's ciphersuite list +to see whether it includes any ciphers other than those included in the +list above. If extra ciphers are included, the responder proceeds as in +"renegotiation" and "in-protocol": it sends a single certificate and +does not request +client certificates. Otherwise (in the case that no extra ciphersuites +are included in the ClientHello) the responder proceeds as in +"certificates up-front": it requests client certificates, and sends a +two-certificate chain. In either case, once the responder has sent its +certificate or certificates, the initiator counts them. If two +certificates have been sent, it proceeds as in "certificates up-front"; +otherwise, it proceeds as in "renegotiation" or "in-protocol". + +To decide whether to do "renegotiation" or "in-protocol", the +initiator checks whether the responder's initial certificate matches +the criteria listed above. + +All new relay implementations of the Tor protocol MUST support +backwards-compatible renegotiation; clients SHOULD do this too. If +this is not possible, new client implementations MUST support both +"renegotiation" and "in-protocol" and use the router's +published link protocols list (see dir-spec.txt on the "protocols" entry) +to decide which to use. + +In all of the above handshake variants, certificates sent in the clear +SHOULD NOT include any strings to identify the host as a Tor relay. In +the "renegotiation" and "backwards-compatible renegotiation" steps, the +initiator SHOULD choose a list of ciphersuites and TLS extensions +to mimic one used by a popular web browser. + +Even though the connection protocol is identical, we will think of the +initiator as either an onion router (OR) if it is willing to relay +traffic for other Tor users, or an onion proxy (OP) if it only handles +local requests. Onion proxies SHOULD NOT provide long-term-trackable +identifiers in their handshakes. + +In all handshake variants, once all certificates are exchanged, all +parties receiving certificates must confirm that the identity key is as +expected. If the key is not as expected, the party must close the +connection. + +(When initiating a connection, if a reasonably live consensus is +available, then the expected identity key is taken from that +consensus. But when initiating a connection otherwise, the expected +identity key is the one given in the hard-coded authority or +fallback list. Finally, when creating a connection because of an +EXTEND/EXTEND2 cell, the expected identity key is the one given in +the cell.) + +When connecting to an OR, all parties SHOULD reject the connection if that +OR has a malformed or missing certificate. When accepting an incoming +connection, an OR SHOULD NOT reject incoming connections from parties with +malformed or missing certificates. (However, an OR should not believe +that an incoming connection is from another OR unless the certificates +are present and well-formed.) + +[Before version 0.1.2.8-rc, ORs rejected incoming connections from ORs and +OPs alike if their certificates were missing or malformed.] + +Once a TLS connection is established, the two sides send cells +(specified below) to one another. Cells are sent serially. Standard +cells are CELL_LEN(link_proto) bytes long, but variable-length cells +also exist; see Section 3. Cells may be sent embedded in TLS records +of any size or divided across TLS records, but the framing of TLS +records MUST NOT leak information about the type or contents of the +cells. + +TLS connections are not permanent. Either side MAY close a connection +if there are no circuits running over it and an amount of time +(KeepalivePeriod, defaults to 5 minutes) has passed since the last time +any traffic was transmitted over the TLS connection. Clients SHOULD +also hold a TLS connection with no circuits open, if it is likely that a +circuit will be built soon using that connection. + +Client-only Tor instances are encouraged to avoid using handshake +variants that include certificates, if those certificates provide +any persistent tags to the relays they contact. If clients do use +certificates, they SHOULD NOT keep using the same certificates when +their IP address changes. Clients MAY send certificates using any +of the above handshake variants. + + +## Picking TLS ciphersuites + +Clients SHOULD send a ciphersuite list chosen to emulate some popular +web browser or other program common on the internet. Clients may send +the "Fixed Cipheruite List" below. If they do not, they MUST NOT +advertise any ciphersuite that they cannot actually support, unless that +cipher is one not supported by OpenSSL 1.0.1. + +The fixed ciphersuite list is: + +```text + TLS1_ECDHE_ECDSA_WITH_AES_256_CBC_SHA + TLS1_ECDHE_RSA_WITH_AES_256_CBC_SHA + TLS1_DHE_RSA_WITH_AES_256_SHA + TLS1_DHE_DSS_WITH_AES_256_SHA + TLS1_ECDH_RSA_WITH_AES_256_CBC_SHA + TLS1_ECDH_ECDSA_WITH_AES_256_CBC_SHA + TLS1_RSA_WITH_AES_256_SHA + TLS1_ECDHE_ECDSA_WITH_RC4_128_SHA + TLS1_ECDHE_ECDSA_WITH_AES_128_CBC_SHA + TLS1_ECDHE_RSA_WITH_RC4_128_SHA + TLS1_ECDHE_RSA_WITH_AES_128_CBC_SHA + TLS1_DHE_RSA_WITH_AES_128_SHA + TLS1_DHE_DSS_WITH_AES_128_SHA + TLS1_ECDH_RSA_WITH_RC4_128_SHA + TLS1_ECDH_RSA_WITH_AES_128_CBC_SHA + TLS1_ECDH_ECDSA_WITH_RC4_128_SHA + TLS1_ECDH_ECDSA_WITH_AES_128_CBC_SHA + SSL3_RSA_RC4_128_MD5 + SSL3_RSA_RC4_128_SHA + TLS1_RSA_WITH_AES_128_SHA + TLS1_ECDHE_ECDSA_WITH_DES_192_CBC3_SHA + TLS1_ECDHE_RSA_WITH_DES_192_CBC3_SHA + SSL3_EDH_RSA_DES_192_CBC3_SHA + SSL3_EDH_DSS_DES_192_CBC3_SHA + TLS1_ECDH_RSA_WITH_DES_192_CBC3_SHA + TLS1_ECDH_ECDSA_WITH_DES_192_CBC3_SHA + SSL3_RSA_FIPS_WITH_3DES_EDE_CBC_SHA + SSL3_RSA_DES_192_CBC3_SHA + [*] The "extended renegotiation is supported" ciphersuite, 0x00ff, is + not counted when checking the list of ciphersuites. +``` + +If the client sends the Fixed Ciphersuite List, the responder MUST NOT +select any ciphersuite besides TLS_DHE_RSA_WITH_AES_256_CBC_SHA, +TLS_DHE_RSA_WITH_AES_128_CBC_SHA, and SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA: +such ciphers might not actually be supported by the client. + +If the client sends a v2+ ClientHello with a list of ciphers other then +the Fixed Ciphersuite List, the responder can trust that the client +supports every cipher advertised in that list, so long as that ciphersuite +is also supported by OpenSSL 1.0.1. + +Responders MUST NOT select any TLS ciphersuite that lacks ephemeral keys, +or whose symmetric keys are less then KEY_LEN bits, or whose digests are +less than HASH_LEN bits. Responders SHOULD NOT select any SSLv3 +ciphersuite other than the DHE+3DES suites listed above. + + +## TLS security considerations + +Implementations MUST NOT allow TLS session resumption -- it can +exacerbate some attacks (e.g. the "Triple Handshake" attack from +Feb 2013), and it plays havoc with forward secrecy guarantees. + +Implementations SHOULD NOT allow TLS compression -- although we don't +know a way to apply a CRIME-style attack to current Tor directly, +it's a waste of resources. + diff --git a/spec/tor-spec/create-created-cells.md b/spec/tor-spec/create-created-cells.md new file mode 100644 index 0000000..a6d36a9 --- /dev/null +++ b/spec/tor-spec/create-created-cells.md @@ -0,0 +1,574 @@ + +## CREATE and CREATED cells + +Users set up circuits incrementally, one hop at a time. To create a +new circuit, OPs send a CREATE/CREATE2 cell to the first node, with +the first half of an authenticated handshake; that node responds with +a CREATED/CREATED2 cell with the second half of the handshake. To +extend a circuit past the first hop, the OP sends an EXTEND/EXTEND2 +relay cell (see section 5.1.2) which instructs the last node in the +circuit to send a CREATE/CREATE2 cell to extend the circuit. + +There are two kinds of CREATE and CREATED cells: The older +"CREATE/CREATED" format, and the newer "CREATE2/CREATED2" format. The +newer format is extensible by design; the older one is not. + +A CREATE2 cell contains: + +```text + HTYPE (Client Handshake Type) [2 bytes] + HLEN (Client Handshake Data Len) [2 bytes] + HDATA (Client Handshake Data) [HLEN bytes] + + A CREATED2 cell contains: + + HLEN (Server Handshake Data Len) [2 bytes] + HDATA (Server Handshake Data) [HLEN bytes] + + Recognized HTYPEs (handshake types) are: + + 0x0000 TAP -- the original Tor handshake; see 5.1.3 + 0x0001 reserved + 0x0002 ntor -- the ntor+curve25519+sha256 handshake; see 5.1.4 + 0x0003 ntor-v3 -- ntor extended with extra data; see 5.1.4.1 + + The format of a CREATE cell is one of the following: + + HDATA (Client Handshake Data) [TAP_C_HANDSHAKE_LEN bytes] + + or + + HTAG (Client Handshake Type Tag) [16 bytes] + HDATA (Client Handshake Data) [TAP_C_HANDSHAKE_LEN-16 bytes] +``` + +The first format is equivalent to a CREATE2 cell with HTYPE of 'tap' +and length of TAP_C_HANDSHAKE_LEN. The second format is a way to +encapsulate new handshake types into the old CREATE cell format for +migration. See 5.1.2 below. Recognized HTAG values are: + +ntor -- 'ntorNTORntorNTOR' + +The format of a CREATED cell is: + +HDATA (Server Handshake Data) [TAP_S_HANDSHAKE_LEN bytes] + +(It's equivalent to a CREATED2 cell with length of TAP_S_HANDSHAKE_LEN.) + +As usual with DH, x and y MUST be generated randomly. + +In general, clients SHOULD use CREATE whenever they are using the TAP +handshake, and CREATE2 otherwise. Clients SHOULD NOT send the +second format of CREATE cells (the one with the handshake type tag) +to a server directly. + +Servers always reply to a successful CREATE with a CREATED, and to a +successful CREATE2 with a CREATED2. On failure, a server sends a +DESTROY cell to tear down the circuit. + +[CREATE2 is handled by Tor 0.2.4.7-alpha and later.] + + +### Choosing circuit IDs in create cells + +The CircID for a CREATE/CREATE2 cell is a nonzero integer, selected +by the node (OP or OR) that sends the CREATE/CREATED2 cell. +Depending on the link protocol version, there are certain rules for +choosing the value of CircID which MUST be obeyed, as implementations +MAY decide to refuse in case of a violation. In link protocol 3 or +lower, CircIDs are 2 bytes long; in protocol 4 or higher, CircIDs are +4 bytes long. + +In link protocol version 3 or lower, the nodes choose from only one +half of the possible values based on the ORs' public identity keys, +in order to avoid collisions. If the sending node has a lower key, +it chooses a CircID with an MSB of 0; otherwise, it chooses a CircID +with an MSB of 1. (Public keys are compared numerically by modulus.) +A client with no public key MAY choose any CircID it wishes, since +clients never need to process CREATE/CREATE2 cells. + +In link protocol version 4 or higher, whichever node initiated the +connection MUST set its MSB to 1, and whichever node didn't initiate +the connection MUST set its MSB to 0. + +The CircID value 0 is specifically reserved for cells that do not +belong to any circuit: CircID 0 MUST not be used for circuits. No +other CircID value, including 0x8000 or 0x80000000, is reserved. + +Existing Tor implementations choose their CircID values at random from +among the available unused values. To avoid distinguishability, new +implementations should do the same. Implementations MAY give up and stop +attempting to build new circuits on a channel, if a certain number of +randomly chosen CircID values are all in use (today's Tor stops after 64). + + +### EXTEND and EXTENDED cells + +To extend an existing circuit, the client sends an EXTEND or EXTEND2 +RELAY_EARLY cell to the last node in the circuit. + +An EXTEND2 cell's relay payload contains: + +```text + NSPEC (Number of link specifiers) [1 byte] + NSPEC times: + LSTYPE (Link specifier type) [1 byte] + LSLEN (Link specifier length) [1 byte] + LSPEC (Link specifier) [LSLEN bytes] + HTYPE (Client Handshake Type) [2 bytes] + HLEN (Client Handshake Data Len) [2 bytes] + HDATA (Client Handshake Data) [HLEN bytes] +``` + +Link specifiers describe the next node in the circuit and how to +connect to it. Recognized specifiers are: + +```text + [00] TLS-over-TCP, IPv4 address + A four-byte IPv4 address plus two-byte ORPort + [01] TLS-over-TCP, IPv6 address + A sixteen-byte IPv6 address plus two-byte ORPort + [02] Legacy identity + A 20-byte SHA1 identity fingerprint. At most one may be listed. + [03] Ed25519 identity + A 32-byte Ed25519 identity fingerprint. At most one may + be listed. +``` + +Nodes MUST ignore unrecognized specifiers, and MUST accept multiple +instances of specifiers other than 'legacy identity' and +'Ed25519 identity'. (Nodes SHOULD reject link specifier lists +that include multiple instances of either one of those specifiers.) + +For purposes of indistinguishability, implementations SHOULD send +these link specifiers, if using them, in this order: [00], [02], [03], +[01]. + +The relay payload for an EXTEND relay cell consists of: + +```text + Address [4 bytes] + Port [2 bytes] + Onion skin [TAP_C_HANDSHAKE_LEN bytes] + Identity fingerprint [HASH_LEN bytes] +``` + +The "legacy identity" and "identity fingerprint" fields are the +SHA1 hash of the PKCS#1 ASN1 encoding of the next onion router's +identity (signing) key. (See 0.3 above.) The "Ed25519 identity" +field is the Ed25519 identity key of the target node. Including +this key information allows the extending OR verify that it is +indeed connected to the correct target OR, and prevents certain +man-in-the-middle attacks. + +Extending ORs MUST check _all_ provided identity keys (if they +recognize the format), and and MUST NOT extend the circuit if the +target OR did not prove its ownership of any such identity key. +If only one identity key is provided, but the extending OR knows +the other (from directory information), then the OR SHOULD also +enforce the key in the directory. + +If an extending OR has a channel with a given Ed25519 ID and RSA +identity, and receives a request for that Ed25519 ID and a +different RSA identity, it SHOULD NOT attempt to make another +connection: it should just fail and DESTROY the circuit. + +The client MAY include multiple IPv4 or IPv6 link specifiers in an +EXTEND cell; current OR implementations only consider the first +of each type. + +After checking relay identities, extending ORs generate a +CREATE/CREATE2 cell from the contents of the EXTEND/EXTEND2 cell. +See section 5.3 for details. + +The payload of an EXTENDED cell is the same as the payload of a +CREATED cell. + +The payload of an EXTENDED2 cell is the same as the payload of a +CREATED2 cell. + +[Support for EXTEND2/EXTENDED2 was added in Tor 0.2.4.8-alpha.] + +Clients SHOULD use the EXTEND format whenever sending a TAP +handshake, and MUST use it whenever the EXTEND cell will be handled +by a node running a version of Tor too old to support EXTEND2. In +other cases, clients SHOULD use EXTEND2. + +When generating an EXTEND2 cell, clients SHOULD include the target's +Ed25519 identity whenever the target has one, and whenever the +target supports LinkAuth subprotocol version "3". (See section 9.2.) + +When encoding a non-TAP handshake in an EXTEND cell, clients SHOULD +use the format with 'client handshake type tag'. + + +### The "TAP" handshake + +This handshake uses Diffie-Hellman in Z_p and RSA to compute a set of +shared keys which the client knows are shared only with a particular +server, and the server knows are shared with whomever sent the +original handshake (or with nobody at all). It's not very fast and +not very good. (See Goldberg's "On the Security of the Tor +Authentication Protocol".) + +Define TAP_C_HANDSHAKE_LEN as DH_LEN+KEY_LEN+KP_PAD_LEN. +Define TAP_S_HANDSHAKE_LEN as DH_LEN+HASH_LEN. + +The payload for a CREATE cell is an 'onion skin', which consists of +the first step of the DH handshake data (also known as g^x). This +value is encrypted using the "legacy hybrid encryption" algorithm +(see 0.4 above) to the server's onion key, giving a client handshake: + +```text + KP-encrypted: + Padding [KP_PAD_LEN bytes] + Symmetric key [KEY_LEN bytes] + First part of g^x [KP_ENC_LEN-KP_PAD_LEN-KEY_LEN bytes] + Symmetrically encrypted: + Second part of g^x [DH_LEN-(KP_ENC_LEN-KP_PAD_LEN-KEY_LEN) + bytes] +``` + +The payload for a CREATED cell, or the relay payload for an +EXTENDED cell, contains: + +```text + DH data (g^y) [DH_LEN bytes] + Derivative key data (KH) [HASH_LEN bytes] +``` + +Once the handshake between the OP and an OR is completed, both can +now calculate g^xy with ordinary DH. Before computing g^xy, both parties +MUST verify that the received g^x or g^y value is not degenerate; +that is, it must be strictly greater than 1 and strictly less than p-1 +where p is the DH modulus. Implementations MUST NOT complete a handshake +with degenerate keys. Implementations MUST NOT discard other "weak" +g^x values. + +(Discarding degenerate keys is critical for security; if bad keys +are not discarded, an attacker can substitute the OR's CREATED +cell's g^y with 0 or 1, thus creating a known g^xy and impersonating +the OR. Discarding other keys may allow attacks to learn bits of +the private key.) + +Once both parties have g^xy, they derive their shared circuit keys +and 'derivative key data' value via the KDF-TOR function in 5.2.1. + + +### The "ntor" handshake + +This handshake uses a set of DH handshakes to compute a set of +shared keys which the client knows are shared only with a particular +server, and the server knows are shared with whomever sent the +original handshake (or with nobody at all). Here we use the +"curve25519" group and representation as specified in "Curve25519: +new Diffie-Hellman speed records" by D. J. Bernstein. + +[The ntor handshake was added in Tor 0.2.4.8-alpha.] + +In this section, define: + +```text + H(x,t) as HMAC_SHA256 with message x and key t. + H_LENGTH = 32. + ID_LENGTH = 20. + G_LENGTH = 32 + PROTOID = "ntor-curve25519-sha256-1" + t_mac = PROTOID | ":mac" + t_key = PROTOID | ":key_extract" + t_verify = PROTOID | ":verify" + G = The preferred base point for curve25519 ([9]) + KEYGEN() = The curve25519 key generation algorithm, returning + a private/public keypair. + m_expand = PROTOID | ":key_expand" + KEYID(A) = A + EXP(a, b) = The ECDH algorithm for establishing a shared secret. +``` + +To perform the handshake, the client needs to know an identity key +digest for the server, and an ntor onion key (a curve25519 public +key) for that server. Call the ntor onion key "B". The client +generates a temporary keypair: + +x,X = KEYGEN() + +and generates a client-side handshake with contents: + +```text + NODEID Server identity digest [ID_LENGTH bytes] + KEYID KEYID(B) [H_LENGTH bytes] + CLIENT_KP X [G_LENGTH bytes] +``` + +The server generates a keypair of y,Y = KEYGEN(), and uses its ntor +private key 'b' to compute: + +```text + secret_input = EXP(X,y) | EXP(X,b) | ID | B | X | Y | PROTOID + KEY_SEED = H(secret_input, t_key) + verify = H(secret_input, t_verify) + auth_input = verify | ID | B | Y | X | PROTOID | "Server" + + The server's handshake reply is: + + SERVER_KP Y [G_LENGTH bytes] + AUTH H(auth_input, t_mac) [H_LENGTH bytes] + + The client then checks Y is in G^* [see NOTE below], and computes + + secret_input = EXP(Y,x) | EXP(B,x) | ID | B | X | Y | PROTOID + KEY_SEED = H(secret_input, t_key) + verify = H(secret_input, t_verify) + auth_input = verify | ID | B | Y | X | PROTOID | "Server" + + The client verifies that AUTH == H(auth_input, t_mac). +``` + +Both parties check that none of the EXP() operations produced the +point at infinity. [NOTE: This is an adequate replacement for +checking Y for group membership, if the group is curve25519.] + +Both parties now have a shared value for KEY_SEED. They expand this +into the keys needed for the Tor relay protocol, using the KDF +described in 5.2.2 and the tag m_expand. + + +#### The "ntor-v3" handshake + +This handshake extends the ntor handshake to include support +for extra data transmitted as part of the handshake. Both +the client and the server can transmit extra data; in both cases, +the extra data is encrypted, but only server data receives +forward secrecy. + +To advertise support for this handshake, servers advertise the +"Relay=4" subprotocol version. To select it, clients use the +'ntor-v3' HTYPE value in their CREATE2 cells. + +In this handshake, we define: + +```text + PROTOID = "ntor3-curve25519-sha3_256-1" + t_msgkdf = PROTOID | ":kdf_phase1" + t_msgmac = PROTOID | ":msg_mac" + t_key_seed = PROTOID | ":key_seed" + t_verify = PROTOID | ":verify" + t_final = PROTOID | ":kdf_final" + t_auth = PROTOID | ":auth_final" + + `ENCAP(s)` -- an encapsulation function. We define this + as `htonll(len(s)) | s`. (Note that `len(ENCAP(s)) = len(s) + 8`). + + `PARTITION(s, n1, n2, n3, ...)` -- a function that partitions a + bytestring `s` into chunks of length `n1`, `n2`, `n3`, and so + on. Extra data is put into a final chunk. If `s` is not long + enough, the function fails. + + H(s, t) = SHA3_256(ENCAP(t) | s) + MAC(k, msg, t) = SHA3_256(ENCAP(t) | ENCAP(k) | s) + KDF(s, t) = SHAKE_256(ENCAP(t) | s) + ENC(k, m) = AES_256_CTR(k, m) + + EXP(pk,sk), KEYGEN: defined as in curve25519 + + DIGEST_LEN = MAC_LEN = MAC_KEY_LEN = ENC_KEY_LEN = PUB_KEY_LEN = 32 + + ID_LEN = 32 (representing an ed25519 identity key) + + For any tag "t_foo": + H_foo(s) = H(s, t_foo) + MAC_foo(k, msg) = MAC(k, msg, t_foo) + KDF_foo(s) = KDF(s, t_foo) + + Other notation is as in the ntor description in 5.1.4 above. + + The client begins by knowing: + + B, ID -- The curve25519 onion key and Ed25519 ID of the server that it + wants to use. + CM -- A message it wants to send as part of its handshake. + VER -- An optional shared verification string: + + The client computes: + + x,X = KEYGEN() + Bx = EXP(B,x) + secret_input_phase1 = Bx | ID | X | B | PROTOID | ENCAP(VER) + phase1_keys = KDF_msgkdf(secret_input_phase1) + (ENC_K1, MAC_K1) = PARTITION(phase1_keys, ENC_KEY_LEN, MAC_KEY_LEN) + encrypted_msg = ENC(ENC_K1, CM) + msg_mac = MAC_msgmac(MAC_K1, ID | B | X | encrypted_msg) + + The client then sends, as its CREATE handshake: + + NODEID ID [ID_LEN bytes] + KEYID B [PUB_KEY_LEN bytes] + CLIENT_PK X [PUB_KEY_LEN bytes] + MSG encrypted_msg [len(CM) bytes] + MAC msg_mac [MAC_LEN bytes] + + The client remembers x, X, B, ID, Bx, and msg_mac. +``` + +When the server receives this handshake, it checks whether NODEID is as +expected, and looks up the (b,B) keypair corresponding to KEYID. If the +keypair is missing or the NODEID is wrong, the handshake fails. + +Now the relay uses `X=CLIENT_PK` to compute: + +```text + Xb = EXP(X,b) + secret_input_phase1 = Xb | ID | X | B | PROTOID | ENCAP(VER) + phase1_keys = KDF_msgkdf(secret_input_phase1) + (ENC_K1, MAC_K1) = PARTITION(phase1_keys, ENC_KEY_LEN, MAC_KEY_LEN) + + expected_mac = MAC_msgmac(MAC_K1, ID | B | X | MSG) +``` + +If `expected_mac` is not `MAC`, the handshake fails. Otherwise +the relay computes `CM` as: + +CM = DEC(MSG, ENC_K1) + +The relay then checks whether `CM` is well-formed, and in response +composes `SM`, the reply that it wants to send as part of the +handshake. It then generates a new ephemeral keypair: + +y,Y = KEYGEN() + +and computes the rest of the handshake: + +```text + Xy = EXP(X,y) + secret_input = Xy | Xb | ID | B | X | Y | PROTOID | ENCAP(VER) + ntor_key_seed = H_key_seed(secret_input) + verify = H_verify(secret_input) + + RAW_KEYSTREAM = KDF_final(ntor_key_seed) + (ENC_KEY, KEYSTREAM) = PARTITION(RAW_KEYSTREAM, ENC_KEY_LKEN, ...) + + encrypted_msg = ENC(ENC_KEY, SM) + + auth_input = verify | ID | B | Y | X | MAC | ENCAP(encrypted_msg) | + PROTOID | "Server" + AUTH = H_auth(auth_input) + + The relay then sends as its CREATED handshake: + + Y Y [PUB_KEY_LEN bytes] + AUTH AUTH [DIGEST_LEN bytes] + MSG encrypted_msg [len(SM) bytes, up to end of the message] + + Upon receiving this handshake, the client computes: + + Yx = EXP(Y, x) + secret_input = Yx | Bx | ID | B | X | Y | PROTOID | ENCAP(VER) + ntor_key_seed = H_key_seed(secret_input) + verify = H_verify(secret_input) + + auth_input = verify | ID | B | Y | X | MAC | ENCAP(MSG) | + PROTOID | "Server" + AUTH_expected = H_auth(auth_input) +``` + +If AUTH_expected is equal to AUTH, then the handshake has +succeeded. The client can then calculate: + +```text + RAW_KEYSTREAM = KDF_final(ntor_key_seed) + (ENC_KEY, KEYSTREAM) = PARTITION(RAW_KEYSTREAM, ENC_KEY_LKEN, ...) + + SM = DEC(ENC_KEY, MSG) +``` + +SM is the message from the relay, and the client uses KEYSTREAM to +generate the shared secrets for the newly created circuit. + +Now both parties share the same KEYSTREAM, and can use it to generate +their circuit keys. + + +### CREATE_FAST/CREATED_FAST cells + +When initializing the first hop of a circuit, the OP has already +established the OR's identity and negotiated a secret key using TLS. +Because of this, it is not always necessary for the OP to perform the +public key operations to create a circuit. In this case, the +OP MAY send a CREATE_FAST cell instead of a CREATE cell for the first +hop only. The OR responds with a CREATED_FAST cell, and the circuit is +created. + +A CREATE_FAST cell contains: + +Key material (X) [HASH_LEN bytes] + +A CREATED_FAST cell contains: + +```text + Key material (Y) [HASH_LEN bytes] + Derivative key data [HASH_LEN bytes] (See 5.2.1 below) + + The values of X and Y must be generated randomly. +``` + +Once both parties have X and Y, they derive their shared circuit keys +and 'derivative key data' value via the KDF-TOR function in 5.2.1. + +The CREATE_FAST handshake is currently deprecated whenever it is not +necessary; the migration is controlled by the "usecreatefast" +networkstatus parameter as described in dir-spec.txt. + +[Tor 0.3.1.1-alpha and later disable CREATE_FAST by default.] + + +### Additional data in CREATE/CREATED cells + +Some handshakes (currently ntor-v3 defined above) allow the client or the +relay to send additional data as part of the handshake. When used in a +CREATE/CREATED handshake, this additional data must have the following +format: + +```text + N_EXTENSIONS [one byte] + N_EXTENSIONS times: + EXT_FIELD_TYPE [one byte] + EXT_FIELD_LEN [one byte] + EXT_FIELD [EXT_FIELD_LEN bytes] + + (`EXT_FIELD_LEN` may be zero, in which case EXT_FIELD is absent.) + + All parties MUST reject messages that are not well-formed per the + rules above. + + We do not specify specific TYPE semantics here; we leave those for + other proposals and specifications. + + Parties MUST ignore extensions with `EXT_FIELD_TYPE` bodies they do not + recognize. + + Unless otherwise specified in the documentation for an extension type: + * Each extension type SHOULD be sent only once in a message. + * Parties MUST ignore any occurrences all occurrences of an extension + with a given type after the first such occurrence. + * Extensions SHOULD be sent in numerically ascending order by type. + + (The above extension sorting and multiplicity rules are only defaults; + they may be overridden in the description of individual extensions.) + + Currently supported extensions are: + + 1 -- CC_FIELD_REQUEST [Client to server] + + Contains an empty payload. Signifies that the client + wants to use the extended congestion control described + in proposal 324. + + 2 -- CC_FIELD_RESPONSE [Server to client] + + Indicates that the relay will use the congestion control + of proposal 324, as requested by the client. One byte + in length: + + sendme_inc [1 byte] +``` + diff --git a/spec/tor-spec/creating-circuits.md b/spec/tor-spec/creating-circuits.md new file mode 100644 index 0000000..31bd6a3 --- /dev/null +++ b/spec/tor-spec/creating-circuits.md @@ -0,0 +1,104 @@ + +## Creating circuits + +When creating a circuit through the network, the circuit creator +(OP) performs the following steps: + +```text + 1. Choose an onion router as an end node (R_N): + * N MAY be 1 for non-anonymous directory mirror, introduction point, + or service rendezvous connections. + * N SHOULD be 3 or more for anonymous connections. + Some end nodes accept streams (see 6.1), others are introduction + or rendezvous points (see rend-spec-{v2,v3}.txt). + + 2. Choose a chain of (N-1) onion routers (R_1...R_N-1) to constitute + the path, such that no router appears in the path twice. + + 3. If not already connected to the first router in the chain, + open a new connection to that router. + + 4. Choose a circID not already in use on the connection with the + first router in the chain; send a CREATE/CREATE2 cell along + the connection, to be received by the first onion router. + + 5. Wait until a CREATED/CREATED2 cell is received; finish the + handshake and extract the forward key Kf_1 and the backward + key Kb_1. + + 6. For each subsequent onion router R (R_2 through R_N), extend + the circuit to R. +``` + +To extend the circuit by a single onion router R_M, the OP performs +these steps: + +1. Create an onion skin, encrypted to R_M's public onion key. + +```text + 2. Send the onion skin in a relay EXTEND/EXTEND2 cell along + the circuit (see sections 5.1.2 and 5.5). + + 3. When a relay EXTENDED/EXTENDED2 cell is received, verify KH, + and calculate the shared keys. The circuit is now extended. +``` + +When an onion router receives an EXTEND relay cell, it sends a CREATE +cell to the next onion router, with the enclosed onion skin as its +payload. + +When an onion router receives an EXTEND2 relay cell, it sends a CREATE2 +cell to the next onion router, with the enclosed HLEN, HTYPE, and HDATA +as its payload. The initiating onion router chooses some circID not yet +used on the connection between the two onion routers. (But see section +5.1.1 above, concerning choosing circIDs.) + +As special cases, if the EXTEND/EXTEND2 cell includes a legacy identity, or +identity fingerprint of all zeroes, or asks to extend back to the relay +that sent the extend cell, the circuit will fail and be torn down. + +Ed25519 identity keys are not required in EXTEND2 cells, so all zero +keys SHOULD be accepted. If the extending relay knows the ed25519 key from +the consensus, it SHOULD also check that key. (See section 5.1.2.) + +If an EXTEND2 cell contains the ed25519 key of the relay that sent the +extend cell, the circuit will fail and be torn down. + +When an onion router receives a CREATE/CREATE2 cell, if it already has a +circuit on the given connection with the given circID, it drops the +cell. Otherwise, after receiving the CREATE/CREATE2 cell, it completes +the specified handshake, and replies with a CREATED/CREATED2 cell. + +Upon receiving a CREATED/CREATED2 cell, an onion router packs it payload +into an EXTENDED/EXTENDED2 relay cell (see section 5.1.2), and sends +that cell up the circuit. Upon receiving the EXTENDED/EXTENDED2 relay +cell, the OP can retrieve the handshake material. + +(As an optimization, OR implementations may delay processing onions +until a break in traffic allows time to do so without harming +network latency too greatly.) + + +### Canonical connections + +It is possible for an attacker to launch a man-in-the-middle attack +against a connection by telling OR Alice to extend to OR Bob at some +address X controlled by the attacker. The attacker cannot read the +encrypted traffic, but the attacker is now in a position to count all +bytes sent between Alice and Bob (assuming Alice was not already +connected to Bob.) + +To prevent this, when an OR gets an extend request, it SHOULD use an +existing OR connection if the ID matches, and ANY of the following +conditions hold: + +```text + - The IP matches the requested IP. + - The OR knows that the IP of the connection it's using is canonical + because it was listed in the NETINFO cell. + + ORs SHOULD NOT check the IPs that are listed in the server descriptor. + Trusting server IPs makes it easier to covertly impersonate a relay, after + stealing its keys. +``` + diff --git a/spec/tor-spec/flow-control.md b/spec/tor-spec/flow-control.md new file mode 100644 index 0000000..4bd8910 --- /dev/null +++ b/spec/tor-spec/flow-control.md @@ -0,0 +1,178 @@ + +# Flow control + + +## Link throttling + +Each client or relay should do appropriate bandwidth throttling to +keep its user happy. + +Communicants rely on TCP's default flow control to push back when they +stop reading. + +The mainline Tor implementation uses token buckets (one for reads, +one for writes) for the rate limiting. + +Since 0.2.0.x, Tor has let the user specify an additional pair of +token buckets for "relayed" traffic, so people can deploy a Tor relay +with strict rate limiting, but also use the same Tor as a client. To +avoid partitioning concerns we combine both classes of traffic over a +given OR connection, and keep track of the last time we read or wrote +a high-priority (non-relayed) cell. If it's been less than N seconds +(currently N=30), we give the whole connection high priority, else we +give the whole connection low priority. We also give low priority +to reads and writes for connections that are serving directory +information. See proposal 111 for details. + + +## Link padding + +Link padding can be created by sending PADDING or VPADDING cells +along the connection; relay cells of type "DROP" can be used for +long-range padding. The payloads of PADDING, VPADDING, or DROP +cells are filled with padding bytes. See Section 3. + +If the link protocol is version 5 or higher, link level padding is +enabled as per padding-spec.txt. On these connections, clients may +negotiate the use of padding with a CELL_PADDING_NEGOTIATE command +whose format is as follows: + +```text + Version [1 byte] + Command [1 byte] + ito_low_ms [2 bytes] + ito_high_ms [2 bytes] +``` + +Currently, only version 0 of this cell is defined. In it, the command +field is either 1 (stop padding) or 2 (start padding). For the start +padding command, a pair of timeout values specifying a low and a high +range bounds for randomized padding timeouts may be specified as unsigned +integer values in milliseconds. The ito_low_ms field should not be lower +than the current consensus parameter value for nf_ito_low (default: +1500). The ito_high_ms field should not be lower than ito_low_ms. +(If any party receives an out-of-range value, they clamp it so +that it is in-range.) + +For the stop padding command, the timeout fields should be sent as +zero (to avoid client distinguishability) and ignored by the recipient. + +For more details on padding behavior, see padding-spec.txt. + + +## Circuit-level flow control + +To control a circuit's bandwidth usage, each OR keeps track of two +'windows', consisting of how many RELAY_DATA cells it is allowed to +originate or willing to consume. + +These two windows are respectively named: the package window (packaged for +transmission) and the deliver window (delivered for local streams). + +Because of our leaky-pipe topology, every relay on the circuit has a pair +of windows, and the OP has a pair of windows for every relay on the +circuit. These windows do not apply to relayed cells, however, and a relay +that is never used for streams will never decrement its window or cause the +client to decrement a window. + +Each 'window' value is initially set based on the consensus parameter +'circwindow' in the directory (see dir-spec.txt), or to 1000 data cells if +no 'circwindow' value is given. In each direction, cells that are not +RELAY_DATA cells do not affect the window. + +An OR or OP (depending on the stream direction) sends a RELAY_SENDME cell +to indicate that it is willing to receive more cells when its deliver +window goes down below a full increment (100). For example, if the window +started at 1000, it should send a RELAY_SENDME when it reaches 900. + +When an OR or OP receives a RELAY_SENDME, it increments its package window +by a value of 100 (circuit window increment) and proceeds to sending the +remaining RELAY_DATA cells. + +If a package window reaches 0, the OR or OP stops reading from TCP +connections for all streams on the corresponding circuit, and sends no more +RELAY_DATA cells until receiving a RELAY_SENDME cell. + +If a deliver window goes below 0, the circuit should be torn down. + +Starting with tor-0.4.1.1-alpha, authenticated SENDMEs are supported +(version 1, see below). This means that both the OR and OP need to remember +the rolling digest of the cell that precedes (triggers) a RELAY_SENDME. +This can be known if the package window gets to a multiple of the circuit +window increment (100). + +When the RELAY_SENDME version 1 arrives, it will contain a digest that MUST +match the one remembered. This represents a proof that the end point of the +circuit saw the sent cells. On failure to match, the circuit should be torn +down. + +To ensure unpredictability, random bytes should be added to at least one +RELAY_DATA cell within one increment window. In other word, every 100 cells +(increment), random bytes should be introduced in at least one cell. + + +### SENDME Cell Format + +A circuit-level RELAY_SENDME cell always has its StreamID=0. + +An OR or OP must obey these two consensus parameters in order to know which +version to emit and accept. + +```text + 'sendme_emit_min_version': Minimum version to emit. + 'sendme_accept_min_version': Minimum version to accept. +``` + +If a RELAY_SENDME version is received that is below the minimum accepted +version, the circuit should be closed. + +The RELAY_SENDME payload contains the following: + +```text + VERSION [1 byte] + DATA_LEN [2 bytes] + DATA [DATA_LEN bytes] +``` + +The VERSION tells us what is expected in the DATA section of length +DATA_LEN and how to handle it. The recognized values are: + +0x00: The rest of the payload should be ignored. + +0x01: Authenticated SENDME. The DATA section MUST contain: + +DIGEST [20 bytes] + +```text + If the DATA_LEN value is less than 20 bytes, the cell should be + dropped and the circuit closed. If the value is more than 20 bytes, + then the first 20 bytes should be read to get the DIGEST value. + + The DIGEST is the rolling digest value from the RELAY_DATA cell that + immediately preceded (triggered) this RELAY_SENDME. This value is + matched on the other side from the previous cell sent that the OR/OP + must remember. + + (Note that if the digest in use has an output length greater than 20 + bytes—as is the case for the hop of an onion service rendezvous + circuit created by the hs_ntor handshake—we truncate the digest + to 20 bytes here.) +``` + +If the VERSION is unrecognized or below the minimum accepted version (taken +from the consensus), the circuit should be torn down. + + +## Stream-level flow control + +Edge nodes use RELAY_SENDME cells to implement end-to-end flow +control for individual connections across circuits. Similarly to +circuit-level flow control, edge nodes begin with a window of cells +(500) per stream, and increment the window by a fixed value (50) +upon receiving a RELAY_SENDME cell. Edge nodes initiate RELAY_SENDME +cells when both a) the window is <= 450, and b) there are less than +ten cell payloads remaining to be flushed at that edge. + +Stream-level RELAY_SENDME cells are distinguished by having nonzero +StreamID. They are still empty; the body still SHOULD be ignored. + diff --git a/spec/tor-spec/handling-relay_early-cells.md b/spec/tor-spec/handling-relay_early-cells.md new file mode 100644 index 0000000..e88c946 --- /dev/null +++ b/spec/tor-spec/handling-relay_early-cells.md @@ -0,0 +1,20 @@ + +## Handling relay_early cells + +A RELAY_EARLY cell is designed to limit the length any circuit can reach. +When an OR receives a RELAY_EARLY cell, and the next node in the circuit +is speaking v2 of the link protocol or later, the OR relays the cell as a +RELAY_EARLY cell. Otherwise, older Tors will relay it as a RELAY cell. + +If a node ever receives more than 8 RELAY_EARLY cells on a given +outbound circuit, it SHOULD close the circuit. If it receives any +inbound RELAY_EARLY cells, it MUST close the circuit immediately. + +When speaking v2 of the link protocol or later, clients MUST only send +EXTEND/EXTEND2 cells inside RELAY_EARLY cells. Clients SHOULD send the first +~8 RELAY cells that are not targeted at the first hop of any circuit as +RELAY_EARLY cells too, in order to partially conceal the circuit length. + +[Starting with Tor 0.2.3.11-alpha, relays should reject any +EXTEND/EXTEND2 cell not received in a RELAY_EARLY cell.] + diff --git a/spec/tor-spec/handling-resource-exhaustion.md b/spec/tor-spec/handling-resource-exhaustion.md new file mode 100644 index 0000000..b8e89eb --- /dev/null +++ b/spec/tor-spec/handling-resource-exhaustion.md @@ -0,0 +1,32 @@ + +# Handling resource exhaustion + + +## Memory exhaustion. + +(See also dos-spec.md.) + +If RAM becomes low, an OR should begin destroying circuits until +more memory is free again. We recommend the following algorithm: + +- Set a threshold amount of RAM to recover at 10% of the total RAM. + +```text + - Sort the circuits by their 'staleness', defined as the age of the + oldest data queued on the circuit. This data can be: + + * Bytes that are waiting to flush to or from a stream on that + circuit. + + * Bytes that are waiting to flush from a connection created with + BEGIN_DIR. + + * Cells that are waiting to flush or be processed. + + - While we have not yet recovered enough RAM: + + * Free all memory held by the most stale circuit, and send DESTROY + cells in both directions on that circuit. Count the amount of + memory we recovered towards the total. +``` + diff --git a/spec/tor-spec/negotiating-initializing-connections.md b/spec/tor-spec/negotiating-initializing-connections.md new file mode 100644 index 0000000..d281761 --- /dev/null +++ b/spec/tor-spec/negotiating-initializing-connections.md @@ -0,0 +1,403 @@ + +# Negotiating and initializing connections + +After Tor instances negotiate handshake with either the "renegotiation" or +"in-protocol" handshakes, they must exchange a set of cells to set up +the Tor connection and make it "open" and usable for circuits. + +When the renegotiation handshake is used, both parties immediately +send a VERSIONS cell (4.1 below), and after negotiating a link +protocol version (which will be 2), each send a NETINFO cell (4.5 +below) to confirm their addresses and timestamps. No other intervening +cell types are allowed. + +When the in-protocol handshake is used, the initiator sends a +VERSIONS cell to indicate that it will not be renegotiating. The +responder sends a VERSIONS cell, a CERTS cell (4.2 below) to give the +initiator the certificates it needs to learn the responder's +identity, an AUTH_CHALLENGE cell (4.3) that the initiator must include +as part of its answer if it chooses to authenticate, and a NETINFO +cell (4.5). As soon as it gets the CERTS cell, the initiator knows +whether the responder is correctly authenticated. At this point the +initiator behaves differently depending on whether it wants to +authenticate or not. If it does not want to authenticate, it MUST +send a NETINFO cell. If it does want to authenticate, it MUST send a +CERTS cell, an AUTHENTICATE cell (4.4), and a NETINFO. When this +handshake is in use, the first cell must be VERSIONS, VPADDING, or +AUTHORIZE, and no other cell type is allowed to intervene besides +those specified, except for VPADDING cells. + +The AUTHORIZE cell type is reserved for future use by scanning-resistance +designs. + +[Tor versions before 0.2.3.11-alpha did not recognize the AUTHORIZE cell, +and did not permit any command other than VERSIONS as the first cell of +the in-protocol handshake.] + + +## Negotiating versions with VERSIONS cells + +There are multiple instances of the Tor link connection protocol. Any +connection negotiated using the "certificates up front" handshake (see +section 2 above) is "version 1". In any connection where both parties +have behaved as in the "renegotiation" handshake, the link protocol +version must be 2. In any connection where both parties have behaved +as in the "in-protocol" handshake, the link protocol must be 3 or higher. + +To determine the version, in any connection where the "renegotiation" +or "in-protocol" handshake was used (that is, where the responder +sent only one certificate at first and where the initiator did not +send any certificates in the first negotiation), both parties MUST +send a VERSIONS cell. In "renegotiation", they send a VERSIONS cell +right after the renegotiation is finished, before any other cells are +sent. In "in-protocol", the initiator sends a VERSIONS cell +immediately after the initial TLS handshake, and the responder +replies immediately with a VERSIONS cell. (As an exception to this rule, +if both sides support the "in-protocol" handshake, either side may send +VPADDING cells at any time.) + +The payload in a VERSIONS cell is a series of big-endian two-byte +integers. Both parties MUST select as the link protocol version the +highest number contained both in the VERSIONS cell they sent and in the +versions cell they received. If they have no such version in common, +they cannot communicate and MUST close the connection. Either party MUST +close the connection if the versions cell is not well-formed (for example, +if the payload contains an odd number of bytes). + +Any VERSIONS cells sent after the first VERSIONS cell MUST be ignored. +(To be interpreted correctly, later VERSIONS cells MUST have a CIRCID_LEN +matching the version negotiated with the first VERSIONS cell.) + +Since the version 1 link protocol does not use the "renegotiation" +handshake, implementations MUST NOT list version 1 in their VERSIONS +cell. When the "renegotiation" handshake is used, implementations +MUST list only the version 2. When the "in-protocol" handshake is +used, implementations MUST NOT list any version before 3, and SHOULD +list at least version 3. + +Link protocols differences are: + +```text + 1 -- The "certs up front" handshake. + 2 -- Uses the renegotiation-based handshake. Introduces + variable-length cells. + 3 -- Uses the in-protocol handshake. + 4 -- Increases circuit ID width to 4 bytes. + 5 -- Adds support for link padding and negotiation (padding-spec.txt). +``` + + +## CERTS cells + +The CERTS cell describes the keys that a Tor instance is claiming +to have. It is a variable-length cell. Its payload format is: + +```text + N: Number of certs in cell [1 octet] + N times: + CertType [1 octet] + CLEN [2 octets] + Certificate [CLEN octets] + + Any extra octets at the end of a CERTS cell MUST be ignored. + + Relevant certType values are: + 1: Link key certificate certified by RSA1024 identity + 2: RSA1024 Identity certificate, self-signed. + 3: RSA1024 AUTHENTICATE cell link certificate, signed with RSA1024 key. + 4: Ed25519 signing key, signed with identity key. + 5: TLS link certificate, signed with ed25519 signing key. + 6: Ed25519 AUTHENTICATE cell key, signed with ed25519 signing key. + 7: Ed25519 identity, signed with RSA identity. +``` + +The certificate format for certificate types 1-3 is DER encoded +X509. For others, the format is as documented in cert-spec.txt. +Note that type 7 uses a different format from types 4-6. + +A CERTS cell may have no more than one certificate of each CertType. + +To authenticate the responder as having a given Ed25519,RSA identity key +combination, the initiator MUST check the following. + +```text + * The CERTS cell contains exactly one CertType 2 "ID" certificate. + * The CERTS cell contains exactly one CertType 4 Ed25519 + "Id->Signing" cert. + * The CERTS cell contains exactly one CertType 5 Ed25519 + "Signing->link" certificate. + * The CERTS cell contains exactly one CertType 7 "RSA->Ed25519" + cross-certificate. + * All X.509 certificates above have validAfter and validUntil dates; + no X.509 or Ed25519 certificates are expired. + * All certificates are correctly signed. + * The certified key in the Signing->Link certificate matches the + SHA256 digest of the certificate that was used to + authenticate the TLS connection. + * The identity key listed in the ID->Signing cert was used to + sign the ID->Signing Cert. + * The Signing->Link cert was signed with the Signing key listed + in the ID->Signing cert. + * The RSA->Ed25519 cross-certificate certifies the Ed25519 + identity, and is signed with the RSA identity listed in the + "ID" certificate. + * The certified key in the ID certificate is a 1024-bit RSA key. + * The RSA ID certificate is correctly self-signed. +``` + +To authenticate the responder as having a given RSA identity only, +the initiator MUST check the following: + +```text + * The CERTS cell contains exactly one CertType 1 "Link" certificate. + * The CERTS cell contains exactly one CertType 2 "ID" certificate. + * Both certificates have validAfter and validUntil dates that + are not expired. + * The certified key in the Link certificate matches the + link key that was used to negotiate the TLS connection. + * The certified key in the ID certificate is a 1024-bit RSA key. + * The certified key in the ID certificate was used to sign both + certificates. + * The link certificate is correctly signed with the key in the + ID certificate + * The ID certificate is correctly self-signed. +``` + +In both cases above, checking these conditions is sufficient to +authenticate that the initiator is talking to the Tor node with the +expected identity, as certified in the ID certificate(s). + +To authenticate the initiator as having a given Ed25519,RSA +identity key combination, the responder MUST check the following: + +```text + * The CERTS cell contains exactly one CertType 2 "ID" certificate. + * The CERTS cell contains exactly one CertType 4 Ed25519 + "Id->Signing" certificate. + * The CERTS cell contains exactly one CertType 6 Ed25519 + "Signing->auth" certificate. + * The CERTS cell contains exactly one CertType 7 "RSA->Ed25519" + cross-certificate. + * All X.509 certificates above have validAfter and validUntil dates; + no X.509 or Ed25519 certificates are expired. + * All certificates are correctly signed. + * The identity key listed in the ID->Signing cert was used to + sign the ID->Signing Cert. + * The Signing->AUTH cert was signed with the Signing key listed + in the ID->Signing cert. + * The RSA->Ed25519 cross-certificate certifies the Ed25519 + identity, and is signed with the RSA identity listed in the + "ID" certificate. + * The certified key in the ID certificate is a 1024-bit RSA key. + * The RSA ID certificate is correctly self-signed. +``` + +To authenticate the initiator as having an RSA identity key only, +the responder MUST check the following: + +```text + * The CERTS cell contains exactly one CertType 3 "AUTH" certificate. + * The CERTS cell contains exactly one CertType 2 "ID" certificate. + * Both certificates have validAfter and validUntil dates that + are not expired. + * The certified key in the AUTH certificate is a 1024-bit RSA key. + * The certified key in the ID certificate is a 1024-bit RSA key. + * The certified key in the ID certificate was used to sign both + certificates. + * The auth certificate is correctly signed with the key in the + ID certificate. + * The ID certificate is correctly self-signed. +``` + +Checking these conditions is NOT sufficient to authenticate that the +initiator has the ID it claims; to do so, the cells in 4.3 and 4.4 +below must be exchanged. + + +## AUTH_CHALLENGE cells + +An AUTH_CHALLENGE cell is a variable-length cell with the following +fields: + +```text + Challenge [32 octets] + N_Methods [2 octets] + Methods [2 * N_Methods octets] +``` + +It is sent from the responder to the initiator. Initiators MUST +ignore unexpected bytes at the end of the cell. Responders MUST +generate every challenge independently using a strong RNG or PRNG. + +The Challenge field is a randomly generated string that the +initiator must sign (a hash of) as part of authenticating. The +methods are the authentication methods that the responder will +accept. Only two authentication methods are defined right now: +see 4.4.1 and 4.4.2 below. + + +## AUTHENTICATE cells + +If an initiator wants to authenticate, it responds to the +AUTH_CHALLENGE cell with a CERTS cell and an AUTHENTICATE cell. +The CERTS cell is as a server would send, except that instead of +sending a CertType 1 (and possibly CertType 5) certs for arbitrary link +certificates, the initiator sends a CertType 3 (and possibly +CertType 6) cert for an RSA/Ed25519 AUTHENTICATE key. + +This difference is because we allow any link key type on a TLS +link, but the protocol described here will only work for specific key +types as described in 4.4.1 and 4.4.2 below. + +An AUTHENTICATE cell contains the following: + +```text + AuthType [2 octets] + AuthLen [2 octets] + Authentication [AuthLen octets] +``` + +Responders MUST ignore extra bytes at the end of an AUTHENTICATE +cell. Recognized AuthTypes are 1 and 3, described in the next +two sections. + +Initiators MUST NOT send an AUTHENTICATE cell before they have +verified the certificates presented in the responder's CERTS +cell, and authenticated the responder. + + +### Link authentication type 1: RSA-SHA256-TLSSecret + +If AuthType is 1 (meaning "RSA-SHA256-TLSSecret"), then the +Authentication field of the AUTHENTICATE cell contains the following: + +```text + TYPE: The characters "AUTH0001" [8 octets] + CID: A SHA256 hash of the initiator's RSA1024 identity key [32 octets] + SID: A SHA256 hash of the responder's RSA1024 identity key [32 octets] + SLOG: A SHA256 hash of all bytes sent from the responder to the + initiator as part of the negotiation up to and including the + AUTH_CHALLENGE cell; that is, the VERSIONS cell, the CERTS cell, + the AUTH_CHALLENGE cell, and any padding cells. [32 octets] + CLOG: A SHA256 hash of all bytes sent from the initiator to the + responder as part of the negotiation so far; that is, the + VERSIONS cell and the CERTS cell and any padding cells. [32 + octets] + SCERT: A SHA256 hash of the responder's TLS link certificate. [32 + octets] + TLSSECRETS: A SHA256 HMAC, using the TLS master secret as the + secret key, of the following: + - client_random, as sent in the TLS Client Hello + - server_random, as sent in the TLS Server Hello + - the NUL terminated ASCII string: + "Tor V3 handshake TLS cross-certification" + [32 octets] + RAND: A 24 byte value, randomly chosen by the initiator. (In an + imitation of SSL3's gmt_unix_time field, older versions of Tor + sent an 8-byte timestamp as the first 8 bytes of this field; + new implementations should not do that.) [24 octets] + SIG: A signature of a SHA256 hash of all the previous fields + using the initiator's "Authenticate" key as presented. (As + always in Tor, we use OAEP-MGF1 padding; see tor-spec.txt + section 0.3.) + [variable length] +``` + +To check the AUTHENTICATE cell, a responder checks that all fields +from TYPE through TLSSECRETS contain their unique +correct values as described above, and then verifies the signature. +The server MUST ignore any extra bytes in the signed data after +the RAND field. + +Responders MUST NOT accept this AuthType if the initiator has +claimed to have an Ed25519 identity. + +(There is no AuthType 2: It was reserved but never implemented.) + + +### Link authentication type 3: Ed25519-SHA256-RFC5705. + +If AuthType is 3, meaning "Ed25519-SHA256-RFC5705", the +Authentication field of the AuthType cell is as below: + +Modified values and new fields below are marked with asterisks. + +```text + TYPE: The characters "AUTH0003" [8 octets] + CID: A SHA256 hash of the initiator's RSA1024 identity key [32 octets] + SID: A SHA256 hash of the responder's RSA1024 identity key [32 octets] + CID_ED: The initiator's Ed25519 identity key [32 octets] + SID_ED: The responder's Ed25519 identity key, or all-zero. [32 octets] + SLOG: A SHA256 hash of all bytes sent from the responder to the + initiator as part of the negotiation up to and including the + AUTH_CHALLENGE cell; that is, the VERSIONS cell, the CERTS cell, + the AUTH_CHALLENGE cell, and any padding cells. [32 octets] + CLOG: A SHA256 hash of all bytes sent from the initiator to the + responder as part of the negotiation so far; that is, the + VERSIONS cell and the CERTS cell and any padding cells. [32 + octets] + SCERT: A SHA256 hash of the responder's TLS link certificate. [32 + octets] + TLSSECRETS: The output of an RFC5705 Exporter function on the + TLS session, using as its inputs: + - The label string "EXPORTER FOR TOR TLS CLIENT BINDING AUTH0003" + - The context value equal to the initiator's Ed25519 identity key. + - The length 32. + [32 octets] + RAND: A 24 byte value, randomly chosen by the initiator. [24 octets] + SIG: A signature of all previous fields using the initiator's + Ed25519 authentication key (as in the cert with CertType 6). + [variable length] +``` + +To check the AUTHENTICATE cell, a responder checks that all fields +from TYPE through TLSSECRETS contain their unique +correct values as described above, and then verifies the signature. +The server MUST ignore any extra bytes in the signed data after +the RAND field. + + +## NETINFO cells + +If version 2 or higher is negotiated, each party sends the other a +NETINFO cell. The cell's payload is: + +```text + TIME (Timestamp) [4 bytes] + OTHERADDR (Other OR's address) [variable] + ATYPE (Address type) [1 byte] + ALEN (Address length) [1 byte] + AVAL (Address value in NBO) [ALEN bytes] + NMYADDR (Number of this OR's addresses) [1 byte] + NMYADDR times: + ATYPE (Address type) [1 byte] + ALEN (Address length) [1 byte] + AVAL (Address value in NBO)) [ALEN bytes] + + Recognized address types (ATYPE) are: + + [04] IPv4. + [06] IPv6. +``` + +ALEN MUST be 4 when ATYPE is 0x04 (IPv4) and 16 when ATYPE is 0x06 +(IPv6). If the ALEN value is wrong for the given ATYPE value, then +the provided address should be ignored. + +The timestamp is a big-endian unsigned integer number of seconds +since the Unix epoch. Implementations MUST ignore unexpected bytes +at the end of the cell. Clients SHOULD send "0" as their timestamp, to +avoid fingerprinting. + +Implementations MAY use the timestamp value to help decide if their +clocks are skewed. Initiators MAY use "other OR's address" to help +learn which address their connections may be originating from, if they do +not know it; and to learn whether the peer will treat the current +connection as canonical. Implementations SHOULD NOT trust these +values unconditionally, especially when they come from non-authorities, +since the other party can lie about the time or IP addresses it sees. + +Initiators SHOULD use "this OR's address" to make sure +that they have connected to another OR at its canonical address. +(See 5.3.1 below.) + diff --git a/spec/tor-spec/opening-streams-transferring-data.md b/spec/tor-spec/opening-streams-transferring-data.md new file mode 100644 index 0000000..dc92d51 --- /dev/null +++ b/spec/tor-spec/opening-streams-transferring-data.md @@ -0,0 +1,117 @@ + +## Opening streams and transferring data + +To open a new anonymized TCP connection, the OP chooses an open +circuit to an exit that may be able to connect to the destination +address, selects an arbitrary StreamID not yet used on that circuit, +and constructs a RELAY_BEGIN cell with a payload encoding the address +and port of the destination host. The payload format is: + +```text + ADDRPORT [nul-terminated string] + FLAGS [4 bytes] + + ADDRPORT is made of ADDRESS | ':' | PORT | [00] +``` + +where ADDRESS can be a DNS hostname, or an IPv4 address in +dotted-quad format, or an IPv6 address surrounded by square brackets; +and where PORT is a decimal integer between 1 and 65535, inclusive. + +The ADDRPORT string SHOULD be sent in lower case, to avoid +fingerprinting. Implementations MUST accept strings in any case. + +The FLAGS value has one or more of the following bits set, where +"bit 1" is the LSB of the 32-bit value, and "bit 32" is the MSB. +(Remember that all values in Tor are big-endian (see 0.1.1 above), so +the MSB of a 4-byte value is the MSB of the first byte, and the LSB +of a 4-byte value is the LSB of its last byte.) + +```text + bit meaning + 1 -- IPv6 okay. We support learning about IPv6 addresses and + connecting to IPv6 addresses. + 2 -- IPv4 not okay. We don't want to learn about IPv4 addresses + or connect to them. + 3 -- IPv6 preferred. If there are both IPv4 and IPv6 addresses, + we want to connect to the IPv6 one. (By default, we connect + to the IPv4 address.) + 4..32 -- Reserved. Current clients MUST NOT set these. Servers + MUST ignore them. +``` + +Upon receiving this cell, the exit node resolves the address as +necessary, and opens a new TCP connection to the target port. If the +address cannot be resolved, or a connection can't be established, the +exit node replies with a RELAY_END cell. (See 6.3 below.) +Otherwise, the exit node replies with a RELAY_CONNECTED cell, whose +payload is in one of the following formats: + +```text + The IPv4 address to which the connection was made [4 octets] + A number of seconds (TTL) for which the address may be cached [4 octets] + + or + + Four zero-valued octets [4 octets] + An address type (6) [1 octet] + The IPv6 address to which the connection was made [16 octets] + A number of seconds (TTL) for which the address may be cached [4 octets] +``` + +[Tor exit nodes before 0.1.2.0 set the TTL field to a fixed value. Later +versions set the TTL to the last value seen from a DNS server, and expire +their own cached entries after a fixed interval. This prevents certain +attacks.] + +Once a connection has been established, the OP and exit node +package stream data in RELAY_DATA cells, and upon receiving such +cells, echo their contents to the corresponding TCP stream. + +If the exit node does not support optimistic data (i.e. its +version number is before 0.2.3.1-alpha), then the OP MUST wait +for a RELAY_CONNECTED cell before sending any data. If the exit +node supports optimistic data (i.e. its version number is +0.2.3.1-alpha or later), then the OP MAY send RELAY_DATA cells +immediately after sending the RELAY_BEGIN cell (and before +receiving either a RELAY_CONNECTED or RELAY_END cell). + +RELAY_DATA cells sent to unrecognized streams are dropped. If +the exit node supports optimistic data, then RELAY_DATA cells it +receives on streams which have seen RELAY_BEGIN but have not yet +been replied to with a RELAY_CONNECTED or RELAY_END are queued. +If the stream creation succeeds with a RELAY_CONNECTED, the queue +is processed immediately afterwards; if the stream creation fails +with a RELAY_END, the contents of the queue are deleted. + +Relay RELAY_DROP cells are long-range dummies; upon receiving such +a cell, the OR or OP must drop it. + + +### Opening a directory stream + +If a Tor relay is a directory server, it should respond to a +RELAY_BEGIN_DIR cell as if it had received a BEGIN cell requesting a +connection to its directory port. RELAY_BEGIN_DIR cells ignore exit +policy, since the stream is local to the Tor process. + +Directory servers may be: +* authoritative directories (RELAY_BEGIN_DIR, usually non-anonymous), +* bridge authoritative directories (RELAY_BEGIN_DIR, anonymous), +* directory mirrors (RELAY_BEGIN_DIR, usually non-anonymous), +* onion service directories (RELAY_BEGIN_DIR, anonymous). + +If the Tor relay is not running a directory service, it should respond +with a REASON_NOTDIRECTORY RELAY_END cell. + +Clients MUST generate an all-zero payload for RELAY_BEGIN_DIR cells, +and relays MUST ignore the payload. + +In response to a RELAY_BEGIN_DIR cell, relays respond either with a +RELAY_CONNECTED cell on success, or a RELAY_END cell on failure. They +MUST send a RELAY_CONNECTED cell all-zero payload, and clients MUST ignore +the payload. + +[RELAY_BEGIN_DIR was not supported before Tor 0.1.2.2-alpha; clients +SHOULD NOT send it to routers running earlier versions of Tor.] + diff --git a/spec/tor-spec/preliminaries.md b/spec/tor-spec/preliminaries.md new file mode 100644 index 0000000..5cb0410 --- /dev/null +++ b/spec/tor-spec/preliminaries.md @@ -0,0 +1,146 @@ + +# Preliminaries + +```text + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL + NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + RFC 2119. +``` + + +## Notation and encoding + +```text + KP -- a public key for an asymmetric cipher. + KS -- a private key for an asymmetric cipher. + K -- a key for a symmetric cipher. + N -- a "nonce", a random value, usually deterministically chosen + from other inputs using hashing. + + a|b -- concatenation of 'a' and 'b'. +``` + +[A0 B1 C2] -- a three-byte sequence, containing the bytes with +hexadecimal values A0, B1, and C2, in that order. + +H(m) -- a cryptographic hash of m. + +We use "byte" and "octet" interchangeably. Possibly we shouldn't. + +Some specs mention "base32". This means RFC4648, without "=" padding. + + +### Encoding integers + +Unless we explicitly say otherwise below, all numeric values in the +Tor protocol are encoded in network (big-endian) order. So a "32-bit +integer" means a big-endian 32-bit integer; a "2-byte" integer means +a big-endian 16-bit integer, and so forth. + + +## Security parameters + +Tor uses a stream cipher, a public-key cipher, the Diffie-Hellman +protocol, and a hash function. + +KEY_LEN -- the length of the stream cipher's key, in bytes. + +```text + KP_ENC_LEN -- the length of a public-key encrypted message, in bytes. + KP_PAD_LEN -- the number of bytes added in padding for public-key + encryption, in bytes. (The largest number of bytes that can be encrypted + in a single public-key operation is therefore KP_ENC_LEN-KP_PAD_LEN.) + + DH_LEN -- the number of bytes used to represent a member of the + Diffie-Hellman group. + DH_SEC_LEN -- the number of bytes used in a Diffie-Hellman private key (x). + + HASH_LEN -- the length of the hash function's output, in bytes. + + PAYLOAD_LEN -- The longest allowable cell payload, in bytes. (509) + + CELL_LEN(v) -- The length of a Tor cell, in bytes, for link protocol + version v. + CELL_LEN(v) = 512 if v is less than 4; + = 514 otherwise. +``` + + +## Ciphers + +These are the ciphers we use _unless otherwise specified_. Several of +them are deprecated for new use. + +For a stream cipher, unless otherwise specified, we use 128-bit AES in +counter mode, with an IV of all 0 bytes. (We also require AES256.) + +For a public-key cipher, unless otherwise specified, we use RSA with +1024-bit keys and a fixed exponent of 65537. We use OAEP-MGF1 +padding, with SHA-1 as its digest function. We leave the optional +"Label" parameter unset. (For OAEP padding, see +ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-1/pkcs-1v2-1.pdf) + +We also use the Curve25519 group and the Ed25519 signature format in +several places. + +For Diffie-Hellman, unless otherwise specified, we use a generator +(g) of 2. For the modulus (p), we use the 1024-bit safe prime from +rfc2409 section 6.2 whose hex representation is: + +```text + "FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024E08" + "8A67CC74020BBEA63B139B22514A08798E3404DDEF9519B3CD3A431B" + "302B0A6DF25F14374FE1356D6D51C245E485B576625E7EC6F44C42E9" + "A637ED6B0BFF5CB6F406B7EDEE386BFB5A899FA5AE9F24117C4B1FE6" + "49286651ECE65381FFFFFFFFFFFFFFFF" +``` + +As an optimization, implementations SHOULD choose DH private keys (x) of +320 bits. Implementations that do this MUST never use any DH key more +than once. +[May other implementations reuse their DH keys?? -RD] +[Probably not. Conceivably, you could get away with changing DH keys once +per second, but there are too many oddball attacks for me to be +comfortable that this is safe. -NM] + +For a hash function, unless otherwise specified, we use SHA-1. + +KEY_LEN=16. +DH_LEN=128; DH_SEC_LEN=40. +KP_ENC_LEN=128; KP_PAD_LEN=42. +HASH_LEN=20. + +We also use SHA256 and SHA3-256 in some places. + +When we refer to "the hash of a public key", unless otherwise +specified, we mean the SHA-1 hash of the DER encoding of an ASN.1 RSA +public key (as specified in PKCS.1). + +All "random" values MUST be generated with a cryptographically +strong pseudorandom number generator seeded from a strong entropy +source, unless otherwise noted. + + +## A bad hybrid encryption algorithm, for legacy purposes. + +Some specifications will refer to the "legacy hybrid encryption" of a +byte sequence M with a public key KP. It is computed as follows: + +```text + 1. If the length of M is no more than KP_ENC_LEN-KP_PAD_LEN, + pad and encrypt M with KP. + 2. Otherwise, generate a KEY_LEN byte random key K. + Let M1 = the first KP_ENC_LEN-KP_PAD_LEN-KEY_LEN bytes of M, + and let M2 = the rest of M. + Pad and encrypt K|M1 with KP. Encrypt M2 with our stream cipher, + using the key K. Concatenate these encrypted values. +``` + +Note that this "hybrid encryption" approach does not prevent +an attacker from adding or removing bytes to the end of M. It also +allows attackers to modify the bytes not covered by the OAEP -- +see Goldberg's PET2006 paper for details. Do not use it as the basis +for new protocols! Also note that as used in Tor's protocols, case 1 +never occurs. + diff --git a/spec/tor-spec/relay-cells.md b/spec/tor-spec/relay-cells.md new file mode 100644 index 0000000..2f89832 --- /dev/null +++ b/spec/tor-spec/relay-cells.md @@ -0,0 +1,159 @@ + +## Relay cells + +Within a circuit, the OP and the end node use the contents of +RELAY packets to tunnel end-to-end commands and TCP connections +("Streams") across circuits. End-to-end commands can be initiated +by either edge; streams are initiated by the OP. + +End nodes that accept streams may be: +* exit relays (RELAY_BEGIN, anonymous), +* directory servers (RELAY_BEGIN_DIR, anonymous or non-anonymous), +* onion services (RELAY_BEGIN, anonymous via a rendezvous point). + +The payload of each unencrypted RELAY cell consists of: + +```text + Relay command [1 byte] + 'Recognized' [2 bytes] + StreamID [2 bytes] + Digest [4 bytes] + Length [2 bytes] + Data [Length bytes] + Padding [PAYLOAD_LEN - 11 - Length bytes] + + The relay commands are: + + 1 -- RELAY_BEGIN [forward] + 2 -- RELAY_DATA [forward or backward] + 3 -- RELAY_END [forward or backward] + 4 -- RELAY_CONNECTED [backward] + 5 -- RELAY_SENDME [forward or backward] [sometimes control] + 6 -- RELAY_EXTEND [forward] [control] + 7 -- RELAY_EXTENDED [backward] [control] + 8 -- RELAY_TRUNCATE [forward] [control] + 9 -- RELAY_TRUNCATED [backward] [control] + 10 -- RELAY_DROP [forward or backward] [control] + 11 -- RELAY_RESOLVE [forward] + 12 -- RELAY_RESOLVED [backward] + 13 -- RELAY_BEGIN_DIR [forward] + 14 -- RELAY_EXTEND2 [forward] [control] + 15 -- RELAY_EXTENDED2 [backward] [control] + + 16..18 -- Reserved for UDP; Not yet in use, see prop339. + + 19..22 -- Reserved for Conflux, see prop329. + + 32..40 -- Used for hidden services; see rend-spec-{v2,v3}.txt. + + 41..42 -- Used for circuit padding; see Section 3 of padding-spec.txt. + + Used for flow control; see Section 4 of prop324. + 43 -- XON [forward or backward] + 44 -- XOFF [forward or backward] +``` + +Commands labelled as "forward" must only be sent by the originator +of the circuit. Commands labelled as "backward" must only be sent by +other nodes in the circuit back to the originator. Commands marked +as either can be sent either by the originator or other nodes. + +The 'recognized' field is used as a simple indication that the cell +is still encrypted. It is an optimization to avoid calculating +expensive digests for every cell. When sending cells, the unencrypted +'recognized' MUST be set to zero. + +When receiving and decrypting cells the 'recognized' will always be +zero if we're the endpoint that the cell is destined for. For cells +that we should relay, the 'recognized' field will usually be nonzero, +but will accidentally be zero with P=2^-16. + +When handling a relay cell, if the 'recognized' in field in a +decrypted relay payload is zero, the 'digest' field is computed as +the first four bytes of the running digest of all the bytes that have +been destined for this hop of the circuit or originated from this hop +of the circuit, seeded from Df or Db respectively (obtained in +section 5.2 above), and including this RELAY cell's entire payload +(taken with the digest field set to zero). Note that these digests +_do_ include the padding bytes at the end of the cell, not only those up +to "Len". If the digest is correct, the cell is considered "recognized" +for the purposes of decryption (see section 5.5 above). + +(The digest does not include any bytes from relay cells that do +not start or end at this hop of the circuit. That is, it does not +include forwarded data. Therefore if 'recognized' is zero but the +digest does not match, the running digest at that node should +not be updated, and the cell should be forwarded on.) + +All RELAY cells pertaining to the same tunneled stream have the same +stream ID. StreamIDs are chosen arbitrarily by the OP. No stream +may have a StreamID of zero. Rather, RELAY cells that affect the +entire circuit rather than a particular stream use a StreamID of zero +-- they are marked in the table above as "[control]" style +cells. (Sendme cells are marked as "sometimes control" because they +can include a StreamID or not depending on their purpose -- see +Section 7.) + +The 'Length' field of a relay cell contains the number of bytes in +the relay payload which contain real payload data. The remainder of +the unencrypted payload is padded with padding bytes. Implementations +handle padding bytes of unencrypted relay cells as they do padding +bytes for other cell types; see Section 3. + +The 'Padding' field is used to make relay cell contents unpredictable, to +avoid certain attacks (see proposal 289 for rationale). Implementations +SHOULD fill this field with four zero-valued bytes, followed by as many +random bytes as will fit. (If there are fewer than 4 bytes for padding, +then they should all be filled with zero. + +Implementations MUST NOT rely on the contents of the 'Padding' field. + +If the RELAY cell is recognized but the relay command is not +understood, the cell must be dropped and ignored. Its contents +still count with respect to the digests and flow control windows, though. + + +### Calculating the 'Digest' field + +The 'Digest' field itself serves the purpose to check if a cell has been +fully decrypted, that is, all onion layers have been removed. Having a +single field, namely 'Recognized' is not sufficient, as outlined above. + +When ENCRYPTING a RELAY cell, an implementation does the following: + +```text + # Encode the cell in binary (recognized and digest set to zero) + tmp = cmd + [0, 0] + stream_id + [0, 0, 0, 0] + length + data + padding + + # Update the digest with the encoded data + digest_state = hash_update(digest_state, tmp) + digest = hash_calculate(digest_state) + + # The encoded data is the same as above with the digest field not being + # zero anymore + encoded = cmd + [0, 0] + stream_id + digest[0..4] + length + data + + padding + + # Now we can encrypt the cell by adding the onion layers ... + + When DECRYPTING a RELAY cell, an implementation does the following: + + decrypted = decrypt(cell) + + # Replace the digest field in decrypted by zeros + tmp = decrypted[0..5] + [0, 0, 0, 0] + decrypted[9..] + + # Update the digest field with the decrypted data and its digest field + # set to zero + digest_state = hash_update(digest_state, tmp) + digest = hash_calculate(digest_state) + + if digest[0..4] == decrypted[5..9] + # The cell has been fully decrypted ... +``` + +The caveat itself is that only the binary data with the digest bytes set to +zero are being taken into account when calculating the running digest. The +final plain-text cells (with the digest field set to its actual value) are +not taken into the running digest. + diff --git a/spec/tor-spec/remote-hostname-lookup.md b/spec/tor-spec/remote-hostname-lookup.md new file mode 100644 index 0000000..3abb2cf --- /dev/null +++ b/spec/tor-spec/remote-hostname-lookup.md @@ -0,0 +1,43 @@ + +## Remote hostname lookup + +To find the address associated with a hostname, the OP sends a +RELAY_RESOLVE cell containing the hostname to be resolved with a NUL +terminating byte. (For a reverse lookup, the OP sends a RELAY_RESOLVE +cell containing an in-addr.arpa address.) The OR replies with a +RELAY_RESOLVED cell containing any number of answers. Each answer is +of the form: + +```text + Type (1 octet) + Length (1 octet) + Value (variable-width) + TTL (4 octets) + "Length" is the length of the Value field. + "Type" is one of: + + 0x00 -- Hostname + 0x04 -- IPv4 address + 0x06 -- IPv6 address + 0xF0 -- Error, transient + 0xF1 -- Error, nontransient + + If any answer has a type of 'Error', then no other answer may be + given. + + The 'Value' field encodes the answer: + IP addresses are given in network order. + Hostnames are given in standard DNS order ("www.example.com") + and not NUL-terminated. + The content of Errors is currently ignored. Relays currently + set it to the string "Error resolving hostname" with no + terminating NUL. Implementations MUST ignore this value. + + For backward compatibility, if there are any IPv4 answers, one of those + must be given as the first answer. + + The RELAY_RESOLVE cell must use a nonzero, distinct streamID; the + corresponding RELAY_RESOLVED cell must use the same streamID. No stream + is actually created by the OR when resolving the name. +``` + diff --git a/spec/tor-spec/routing-relay-cells.md b/spec/tor-spec/routing-relay-cells.md new file mode 100644 index 0000000..95888f7 --- /dev/null +++ b/spec/tor-spec/routing-relay-cells.md @@ -0,0 +1,86 @@ + +## Routing relay cells + + +### Circuit ID Checks + +When a node wants to send a RELAY or RELAY_EARLY cell, it checks the cell's +circID and determines whether the corresponding circuit along that +connection is still open. If not, the node drops the cell. + +When a node receives a RELAY or RELAY_EARLY cell, it checks the cell's +circID and determines whether it has a corresponding circuit along +that connection. If not, the node drops the cell. + + +### Forward Direction + +The forward direction is the direction that CREATE/CREATE2 cells +are sent. + + +#### Routing from the Origin + +When a relay cell is sent from an OP, the OP encrypts the payload +with the stream cipher as follows: + +```text + OP sends relay cell: + For I=N...1, where N is the destination node: + Encrypt with Kf_I. + Transmit the encrypted cell to node 1. +``` + + +#### Relaying Forward at Onion Routers + +When a forward relay cell is received by an OR, it decrypts the payload +with the stream cipher, as follows: + +```text + 'Forward' relay cell: + Use Kf as key; decrypt. +``` + +The OR then decides whether it recognizes the relay cell, by +inspecting the payload as described in section 6.1 below. If the OR +recognizes the cell, it processes the contents of the relay cell. +Otherwise, it passes the decrypted relay cell along the circuit if +the circuit continues. If the OR at the end of the circuit +encounters an unrecognized relay cell, an error has occurred: the OR +sends a DESTROY cell to tear down the circuit. + +For more information, see section 6 below. + + +### Backward Direction + +The backward direction is the opposite direction from +CREATE/CREATE2 cells. + + +#### Relaying Backward at Onion Routers + +When a backward relay cell is received by an OR, it encrypts the payload +with the stream cipher, as follows: + +```text + 'Backward' relay cell: + Use Kb as key; encrypt. +``` + + +### Routing to the Origin + +When a relay cell arrives at an OP, the OP decrypts the payload +with the stream cipher as follows: + +```text + OP receives relay cell from node 1: + For I=1...N, where N is the final node on the circuit: + Decrypt with Kb_I. + If the payload is recognized (see section 6.1), then: + The sending node is I. + Stop and process the payload. +``` + diff --git a/spec/tor-spec/setting-circuit-keys.md b/spec/tor-spec/setting-circuit-keys.md new file mode 100644 index 0000000..d566ff6 --- /dev/null +++ b/spec/tor-spec/setting-circuit-keys.md @@ -0,0 +1,60 @@ + +## Setting circuit keys + + +### KDF-TOR + +This key derivation function is used by the TAP and CREATE_FAST +handshakes, and in the current hidden service protocol. It shouldn't +be used for new functionality. + +If the TAP handshake is used to extend a circuit, both parties +base their key material on K0=g^xy, represented as a big-endian unsigned +integer. + +If CREATE_FAST is used, both parties base their key material on +K0=X|Y. + +From the base key material K0, they compute KEY_LEN*2+HASH_LEN*3 bytes of +derivative key data as + +K = H(K0 | [00]) | H(K0 | [01]) | H(K0 | [02]) | ... + +The first HASH_LEN bytes of K form KH; the next HASH_LEN form the forward +digest Df; the next HASH_LEN 41-60 form the backward digest Db; the next +KEY_LEN 61-76 form Kf, and the final KEY_LEN form Kb. Excess bytes from K +are discarded. + +KH is used in the handshake response to demonstrate knowledge of the +computed shared key. Df is used to seed the integrity-checking hash +for the stream of data going from the OP to the OR, and Db seeds the +integrity-checking hash for the data stream from the OR to the OP. Kf +is used to encrypt the stream of data going from the OP to the OR, and +Kb is used to encrypt the stream of data going from the OR to the OP. + + +### KDF-RFC5869 + +For newer KDF needs, Tor uses the key derivation function HKDF from +RFC5869, instantiated with SHA256. (This is due to a construction +from Krawczyk.) The generated key material is: + +K = K_1 | K_2 | K_3 | ... + +```text + Where H(x,t) is HMAC_SHA256 with value x and key t + and K_1 = H(m_expand | INT8(1) , KEY_SEED ) + and K_(i+1) = H(K_i | m_expand | INT8(i+1) , KEY_SEED ) + and m_expand is an arbitrarily chosen value, + and INT8(i) is a octet with the value "i". +``` + +In RFC5869's vocabulary, this is HKDF-SHA256 with info == m_expand, +salt == t_key, and IKM == secret_input. + +When used in the ntor handshake, the first HASH_LEN bytes form the +forward digest Df; the next HASH_LEN form the backward digest Db; the +next KEY_LEN form Kf, the next KEY_LEN form Kb, and the final +DIGEST_LEN bytes are taken as a nonce to use in the place of KH in the +hidden service protocol. Excess bytes from K are discarded. + diff --git a/spec/tor-spec/subprotocol-versioning.md b/spec/tor-spec/subprotocol-versioning.md new file mode 100644 index 0000000..0fd5388 --- /dev/null +++ b/spec/tor-spec/subprotocol-versioning.md @@ -0,0 +1,296 @@ + +# Subprotocol versioning + +This section specifies the Tor subprotocol versioning. They are broken down +into different types with their current version numbers. Any new version +number should be added to this section. + +The dir-spec.txt details how those versions are encoded. See the +"proto"/"pr" line in a descriptor and the "recommended-relay-protocols", +"required-relay-protocols", "recommended-client-protocols" and +"required-client-protocols" lines in the vote/consensus format. + +Here are the rules a relay and client should follow when encountering a +protocol list in the consensus: + +```text + - When a relay lacks a protocol listed in recommended-relay-protocols, + it should warn its operator that the relay is obsolete. + + - When a relay lacks a protocol listed in required-relay-protocols, it + should warn its operator as above. If the consensus is newer than the + date when the software was released or scheduled for release, it must + not attempt to join the network. + + - When a client lacks a protocol listed in recommended-client-protocols, + it should warn the user that the client is obsolete. + + - When a client lacks a protocol listed in required-client-protocols, + it should warn the user as above. If the consensus is newer than the + date when the software was released, it must not connect to the + network. This implements a "safe forward shutdown" mechanism for + zombie clients. + + - If a client or relay has a cached consensus telling it that a given + protocol is required, and it does not implement that protocol, it + SHOULD NOT try to fetch a newer consensus. +``` + +Software release dates SHOULD be automatically updated as part of the +release process, to prevent forgetting to move them forward. Software +release dates MAY be manually adjusted by maintainers if necessary. + +Starting in version 0.2.9.4-alpha, the initial required protocols for +clients that we will Recommend and Require are: + +```text + Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1 Link=4 + LinkAuth=1 Microdesc=1-2 Relay=2 + + For relays we will Require: + + Cons=1 Desc=1 DirCache=1 HSDir=1 HSIntro=3 HSRend=1 Link=3-4 + LinkAuth=1 Microdesc=1 Relay=1-2 +``` + +For relays, we will additionally Recommend all protocols which we +recommend for clients. + + +## "Link" + +The "link" protocols are those used by clients and relays to initiate and +receive OR connections and to handle cells on OR connections. The "link" +protocol versions correspond 1:1 to those versions. + +Two Tor instances can make a connection to each other only if they have at +least one link protocol in common. + +The current "link" versions are: "1" through "5". See section 4.1 for more +information. All current Tor versions support "1-3"; versions from +0.2.4.11-alpha and on support "1-4"; versions from 0.3.1.1-alpha and on +support "1-5". Eventually we will drop "1" and "2". + + +## "LinkAuth" + +LinkAuth protocols correspond to varieties of Authenticate cells used for +the v3+ link protocols. + +Current versions are: + +"1" is the RSA link authentication described in section 4.4.1 above. + +"2" is unused, and reserved by proposal 244. + +"3" is the ed25519 link authentication described in 4.4.2 above. + + +## "Relay" + +The "relay" protocols are those used to handle CREATE/CREATE2 +cells, and those that handle the various RELAY cell types received +after a CREATE/CREATE2 cell. (Except, relay cells used to manage +introduction and rendezvous points are managed with the "HSIntro" +and "HSRend" protocols respectively.) + +Current versions are: + +```text + "1" -- supports the TAP key exchange, with all features in Tor 0.2.3. + Support for CREATE and CREATED and CREATE_FAST and CREATED_FAST + and EXTEND and EXTENDED. + + "2" -- supports the ntor key exchange, and all features in Tor + 0.2.4.19. Includes support for CREATE2 and CREATED2 and + EXTEND2 and EXTENDED2. + + Relay=2 has limited IPv6 support: + * Clients might not include IPv6 ORPorts in EXTEND2 cells. + * Relays (and bridges) might not initiate IPv6 connections in + response to EXTEND2 cells containing IPv6 ORPorts, even if they + are configured with an IPv6 ORPort. + + However, relays support accepting inbound connections to their IPv6 + ORPorts. And they might extend circuits via authenticated IPv6 + connections to other relays. + + "3" -- relays support extending over IPv6 connections in response to an + EXTEND2 cell containing an IPv6 ORPort. + + Bridges might not extend over IPv6, because they try to imitate + client behaviour. + + A successful IPv6 extend requires: + * Relay subprotocol version 3 (or later) on the extending relay, + * an IPv6 ORPort on the extending relay, + * an IPv6 ORPort for the accepting relay in the EXTEND2 cell, and + * an IPv6 ORPort on the accepting relay. + (Because different tor instances can have different views of the + network, these checks should be done when the path is selected. + Extending relays should only check local IPv6 information, before + attempting the extend.) + + When relays receive an EXTEND2 cell containing both an IPv4 and an + IPv6 ORPort, and there is no existing authenticated connection with + the target relay, the extending relay may choose between IPv4 and + IPv6 at random. The extending relay might not try the other address, + if the first connection fails. + + As is the case with other subprotocol versions, tor advertises, + recommends, or requires support for this protocol version, regardless + of its current configuration. + + In particular: + * relays without an IPv6 ORPort, and + * tor instances that are not relays, + have the following behaviour, regardless of their configuration: + * advertise support for "Relay=3" in their descriptor + (if they are a relay, bridge, or directory authority), and + * react to consensuses recommending or requiring support for + "Relay=3". + + This subprotocol version is described in proposal 311, and + implemented in Tor 0.4.5.1-alpha. + + "4" -- support the ntorv3 (version 3) key exchange and all features in + 0.4.7.3-alpha. This adds a new CREATE2 cell type. See proposal 332 + and section 5.1.4.1 above for more details. +``` + + +## "HSIntro" + +The "HSIntro" protocol handles introduction points. + +```text + "3" -- supports authentication as of proposal 121 in Tor + 0.2.1.6-alpha. + + "4" -- support ed25519 authentication keys which is defined by the HS v3 + protocol as part of proposal 224 in Tor 0.3.0.4-alpha. + + "5" -- support ESTABLISH_INTRO cell DoS parameters extension for onion + service version 3 only in Tor 0.4.2.1-alpha. +``` + + +## "HSRend" + +The "HSRend" protocol handles rendezvous points. + +"1" -- supports all features in Tor 0.0.6. + +```text + "2" -- supports RENDEZVOUS2 cells of arbitrary length as long as they + have 20 bytes of cookie in Tor 0.2.9.1-alpha. +``` + + +## "HSDir" + +The "HSDir" protocols are the set of hidden service document types that can +be uploaded to, understood by, and downloaded from a tor relay, and the set +of URLs available to fetch them. + +"1" -- supports all features in Tor 0.2.0.10-alpha. + +```text + "2" -- support ed25519 blinded keys request which is defined by the HS v3 + protocol as part of proposal 224 in Tor 0.3.0.4-alpha. +``` + + +## "DirCache" + +The "DirCache" protocols are the set of documents available for download +from a directory cache via BEGIN_DIR, and the set of URLs available to +fetch them. (This excludes URLs for hidden service objects.) + +"1" -- supports all features in Tor 0.2.4.19. + +"2" -- adds support for consensus diffs in Tor 0.3.1.1-alpha. + + +## "Desc" + +Describes features present or absent in descriptors. + +Most features in descriptors don't require a "Desc" update -- only those +that need to someday be required. For example, someday clients will need +to understand ed25519 identities. + +"1" -- supports all features in Tor 0.2.4.19. + +"2" -- cross-signing with onion-keys, signing with ed25519 +identities. + + +## "Microdesc" + +Describes features present or absent in microdescriptors. + +Most features in descriptors don't require a "MicroDesc" update -- only +those that need to someday be required. These correspond more or less with +consensus methods. + +"1" -- consensus methods 9 through 20. + +"2" -- consensus method 21 (adds ed25519 keys to microdescs). + + +## "Cons" + +Describes features present or absent in consensus documents. + +Most features in consensus documents don't require a "Cons" update -- only +those that need to someday be required. + +These correspond more or less with consensus methods. + +"1" -- consensus methods 9 through 20. + +"2" -- consensus method 21 (adds ed25519 keys to microdescs). + + +## "Padding" + +Describes the padding capabilities of the relay. + +```text + "1" -- [DEFUNCT] Relay supports circuit-level padding. This version MUST NOT + be used as it was also enabled in relays that don't actually support + circuit-level padding. Advertised by Tor versions from + tor-0.4.0.1-alpha and only up to and including tor-0.4.1.4-rc. + + "2" -- Relay supports the HS circuit setup padding machines (proposal 302). + Advertised by Tor versions from tor-0.4.1.5 and onwards. +``` + + +## "FlowCtrl" + +Describes the flow control protocol at the circuit and stream level. If +there is no FlowCtrl advertised, tor supports the unauthenticated flow +control features (version 0). + +```text + "1" -- supports authenticated circuit level SENDMEs as of proposal 289 in + Tor 0.4.1.1-alpha. + + "2" -- supports congestion control by the Exits which implies a new SENDME + format and algorithm. See proposal 324 for more details. Advertised + in tor 0.4.7.3-alpha. +``` + + +## "Datagram" + +Describes the UDP protocol capabilities of a relay. + +```text + "1" -- [RESERVED] supports UDP by an Exit as in the relay command + CONNECT_UDP, CONNECTED_UDP and DATAGRAM. See proposal + 339 for more details. (Not yet advertised, reserved) +``` + diff --git a/spec/tor-spec/system-overview.md b/spec/tor-spec/system-overview.md new file mode 100644 index 0000000..4897d95 --- /dev/null +++ b/spec/tor-spec/system-overview.md @@ -0,0 +1,73 @@ + +# System overview + +Tor is a distributed overlay network designed to anonymize +low-latency TCP-based applications such as web browsing, secure shell, +and instant messaging. Clients choose a path through the network and +build a ``circuit'', in which each node (or ``onion router'' or ``OR'') +in the path knows its predecessor and successor, but no other nodes in +the circuit. Traffic flowing down the circuit is sent in fixed-size +``cells'', which are unwrapped by a symmetric key at each node (like +the layers of an onion) and relayed downstream. + + +## Keys and names + +Every Tor relay has multiple public/private keypairs: + +These are 1024-bit RSA keys: + +```text + - A long-term signing-only "Identity key" used to sign documents and + certificates, and used to establish relay identity. + KP_relayid_rsa, KS_relayid_rsa. + - A medium-term TAP "Onion key" used to decrypt onion skins when accepting + circuit extend attempts. (See 5.1.) Old keys MUST be accepted for a + while after they are no longer advertised. Because of this, + relays MUST retain old keys for a while after they're rotated. (See + "onion key lifetime parameters" in dir-spec.txt.) + KP_onion_tap, KS_onion_tap. + - A short-term "Connection key" used to negotiate TLS connections. + Tor implementations MAY rotate this key as often as they like, and + SHOULD rotate this key at least once a day. + KP_conn_tls, KS_conn_tls. + + This is Curve25519 key: + + - A medium-term ntor "Onion key" used to handle onion key handshakes when + accepting incoming circuit extend requests. As with TAP onion keys, + old ntor keys MUST be accepted for at least one week after they are no + longer advertised. Because of this, relays MUST retain old keys for a + while after they're rotated. (See "onion key lifetime parameters" in + dir-spec.txt.) + KP_ntor, KS_ntor. + + These are Ed25519 keys: + + - A long-term "master identity" key. This key never + changes; it is used only to sign the "signing" key below. It may be + kept offline. + KP_relayid_ed, KS_relayid_ed. + - A medium-term "signing" key. This key is signed by the master identity + key, and must be kept online. A new one should be generated + periodically. It signs nearly everything else. + KP_relaysign_ed, KS_relaysign_ed. + - A short-term "link authentication" key, used to authenticate + the link handshake: see section 4 below. This key is signed + by the "signing" key, and should be regenerated frequently. + KP_link_ed, KS_link_ed. +``` + +KP_relayid_* together identify a router uniquely. Once a router +has used a KP_relayid_ed (an Ed25519 master identity key) +together with a given KP_relayid_rsa (RSA identity key), neither of +those keys may ever be used with a different key. + +We write KP_relayid to refer to a key which is either +KP_relayid_rsa or KP_relayid_ed. + +The same key or keypair should never be used for separate roles within +the Tor protocol suite, unless specifically stated. For example, +a relay's identity keys K_relayid should not also be used as the +identity keypair for a hidden service K_hs_id (see rend-spec-v3.txt). + diff --git a/spec/tor-spec/tearing-down-circuits.md b/spec/tor-spec/tearing-down-circuits.md new file mode 100644 index 0000000..52f1cc8 --- /dev/null +++ b/spec/tor-spec/tearing-down-circuits.md @@ -0,0 +1,107 @@ + +## Tearing down circuits + +Circuits are torn down when an unrecoverable error occurs along +the circuit, or when all streams on a circuit are closed and the +circuit's intended lifetime is over. + +ORs SHOULD also tear down circuits which attempt to create: + +* streams with RELAY_BEGIN, or +* rendezvous points with ESTABLISH_RENDEZVOUS, +ending at the first hop. Letting Tor be used as a single hop proxy makes +exit and rendezvous nodes a more attractive target for compromise. + +ORs MAY use multiple methods to check if they are the first hop: + +```text + * If an OR sees a circuit created with CREATE_FAST, the OR is sure to be + the first hop of a circuit. + * If an OR is the responder, and the initiator: + * did not authenticate the link, or + * authenticated with a key that is not in the consensus, + then the OR is probably the first hop of a circuit (or the second hop of + a circuit via a bridge relay). + + Circuits may be torn down either completely or hop-by-hop. +``` + +To tear down a circuit completely, an OR or OP sends a DESTROY +cell to the adjacent nodes on that circuit, using the appropriate +direction's circID. + +Upon receiving an outgoing DESTROY cell, an OR frees resources +associated with the corresponding circuit. If it's not the end of +the circuit, it sends a DESTROY cell for that circuit to the next OR +in the circuit. If the node is the end of the circuit, then it tears +down any associated edge connections (see section 6.1). + +After a DESTROY cell has been processed, an OR ignores all data or +destroy cells for the corresponding circuit. + +To tear down part of a circuit, the OP may send a RELAY_TRUNCATE cell +signaling a given OR (Stream ID zero). That OR sends a DESTROY +cell to the next node in the circuit, and replies to the OP with a +RELAY_TRUNCATED cell. + +[Note: If an OR receives a TRUNCATE cell and it has any RELAY cells +still queued on the circuit for the next node it will drop them +without sending them. This is not considered conformant behavior, +but it probably won't get fixed until a later version of Tor. Thus, +clients SHOULD NOT send a TRUNCATE cell to a node running any current +version of Tor if a) they have sent relay cells through that node, +and b) they aren't sure whether those cells have been sent on yet.] + +```text + When an unrecoverable error occurs along one a circuit, the nodes + must report it as follows: + * If possible, send a DESTROY cell to ORs _away_ from the client. + * If possible, send *either* a DESTROY cell towards the client, or + a RELAY_TRUNCATED cell towards the client. +``` + +Current versions of Tor do not reuse truncated RELAY_TRUNCATED +circuits: An OP, upon receiving a RELAY_TRUNCATED, will send +forward a DESTROY cell in order to entirely tear down the circuit. +Because of this, we recommend that relays should send DESTROY +towards the client, not RELAY_TRUNCATED. + +```text + NOTE: + In tor versions before 0.4.5.13, 0.4.6.11 and 0.4.7.9, relays would + handle an inbound DESTROY by sending the client a RELAY_TRUNCATED + message. Beginning with those versions, relays now propagate + DESTROY cells in either direction, in order to tell every + intermediary ORs to stop queuing data on the circuit. The earlier + behavior created queuing pressure on the intermediary ORs. +``` + +The payload of a DESTROY and RELAY_TRUNCATED cell contains a single +octet, describing the reason that the circuit was +closed. RELAY_TRUNCATED cells, and DESTROY cells sent _towards the +client, should contain the actual reason from the list of error codes +below. Reasons in DESTROY cell SHOULD NOT be propagated downward or +upward, due to potential side channel risk: An OR receiving a DESTROY +command should use the DESTROYED reason for its next cell. An OP +should always use the NONE reason for its own DESTROY cells. + +The error codes are: + +```text + 0 -- NONE (No reason given.) + 1 -- PROTOCOL (Tor protocol violation.) + 2 -- INTERNAL (Internal error.) + 3 -- REQUESTED (A client sent a TRUNCATE command.) + 4 -- HIBERNATING (Not currently operating; trying to save bandwidth.) + 5 -- RESOURCELIMIT (Out of memory, sockets, or circuit IDs.) + 6 -- CONNECTFAILED (Unable to reach relay.) + 7 -- OR_IDENTITY (Connected to relay, but its OR identity was not + as expected.) + 8 -- CHANNEL_CLOSED (The OR connection that was carrying this circuit + died.) + 9 -- FINISHED (The circuit has expired for being dirty or old.) + 10 -- TIMEOUT (Circuit construction took too long) + 11 -- DESTROYED (The circuit was destroyed w/o client TRUNCATE) + 12 -- NOSUCHSERVICE (Request for unknown hidden service) +``` + diff --git a/spec/version-spec.md b/spec/version-spec.md new file mode 100644 index 0000000..ac9432b --- /dev/null +++ b/spec/version-spec.md @@ -0,0 +1,93 @@ +HOW TOR VERSION NUMBERS WORK + +Table of Contents + +```text + 1. The Old Way + 2. The New Way + 3. Version status. +``` + + +# The Old Way + +Before 0.1.0, versions were of the format: + +MAJOR.MINOR.MICRO(status(PATCHLEVEL))?(-cvs)? + +where MAJOR, MINOR, MICRO, and PATCHLEVEL are numbers, status is one +of "pre" (for an alpha release), "rc" (for a release candidate), or +"." for a release. As a special case, "a.b.c" was equivalent to +"a.b.c.0". We compare the elements in order (major, minor, micro, +status, patchlevel, cvs), with "cvs" preceding non-cvs. + +We would start each development branch with a final version in mind: +say, "0.0.8". Our first pre-release would be "0.0.8pre1", followed by +(for example) "0.0.8pre2-cvs", "0.0.8pre2", "0.0.8pre3-cvs", +"0.0.8rc1", "0.0.8rc2-cvs", and "0.0.8rc2". Finally, we'd release +0.0.8. The stable CVS branch would then be versioned "0.0.8.1-cvs", +and any eventual bugfix release would be "0.0.8.1". + + +# The New Way + +Starting at 0.1.0.1-rc, versions are of the format: + +MAJOR.MINOR.MICRO[.PATCHLEVEL][-STATUS_TAG][ (EXTRA_INFO)]* + +The stuff in parentheses is optional. As before, MAJOR, MINOR, MICRO, +and PATCHLEVEL are numbers, with an absent number equivalent to 0. +All versions should be distinguishable purely by those four +numbers. + +The STATUS_TAG is purely informational, and lets you know how +stable we think the release is: "alpha" is pretty unstable; "rc" is a +release candidate; and no tag at all means that we have a final +release. If the tag ends with "-cvs" or "-dev", you're looking at a +development snapshot that came after a given release. If we *do* +encounter two versions that differ only by status tag, we compare them +lexically. The STATUS_TAG can't contain whitespace. + +The EXTRA_INFO is also purely informational, often containing information +about the SCM commit this version came from. It is surrounded by parentheses +and can't contain whitespace. Unlike the STATUS_TAG this never impacts the way +that versions should be compared. EXTRA_INFO may appear any number of +times. Tools should generally not parse EXTRA_INFO entries. + +Now, we start each development branch with (say) 0.1.1.1-alpha. The +patchlevel increments consistently as the status tag changes, for +example, as in: 0.1.1.2-alpha, 0.1.1.3-alpha, 0.1.1.4-rc, 0.1.1.5-rc. +Eventually, we release 0.1.1.6. The next patch release is 0.1.1.7. + +Between these releases, CVS is versioned with a -cvs tag: after +0.1.1.1-alpha comes 0.1.1.1-alpha-cvs, and so on. But starting with +0.1.2.1-alpha-dev, we switched to SVN and started using the "-dev" +suffix instead of the "-cvs" suffix. + + +# Version status. + +```text + Sometimes we need to determine whether a Tor version is obsolete, + experimental, or neither, based on a list of recommended versions. The + logic is as follows: + + * If a version is listed on the recommended list, then it is + "recommended". + + * If a version is newer than every recommended version, that version + is "experimental" or "new". + + * If a version is older than every recommended version, it is + "obsolete" or "old". + + * The first three components (major,minor,micro) of a version number + are its "release series". If a version has other recommended + versions with the same release series, and the version is newer + than all such recommended versions, but it is not newer than + _every_ recommended version, then the version is "new in series". + + * Finally, if none of the above conditions hold, then the version is + "un-recommended." +``` + -- cgit v1.2.3-54-g00ecf