aboutsummaryrefslogtreecommitdiff
path: root/dir-list-spec.txt
diff options
context:
space:
mode:
authorNick Mathewson <nickm@torproject.org>2023-10-12 12:27:58 -0400
committerNick Mathewson <nickm@torproject.org>2023-10-12 12:27:58 -0400
commite4e0d93d56ee8c1aec4c2efaa7046b651f0fe55c (patch)
tree15a085da265ae3b2b70f29a70f910a5371059a78 /dir-list-spec.txt
parentb719a373934d3e79ef833446c6aeeb19be485510 (diff)
downloadtorspec-e4e0d93d56ee8c1aec4c2efaa7046b651f0fe55c.tar.gz
torspec-e4e0d93d56ee8c1aec4c2efaa7046b651f0fe55c.zip
Move all text-only specifications into the OLD_TXT directory.
Diffstat (limited to 'dir-list-spec.txt')
-rw-r--r--dir-list-spec.txt529
1 files changed, 0 insertions, 529 deletions
diff --git a/dir-list-spec.txt b/dir-list-spec.txt
deleted file mode 100644
index 65af536..0000000
--- a/dir-list-spec.txt
+++ /dev/null
@@ -1,529 +0,0 @@
-
- Tor Directory List Format
- Tim Wilson-Brown (teor)
-
-Table of Contents
-
- 1. Scope and Preliminaries
- 1.1. Format Overview
- 1.2. Acknowledgements
- 1.3. Format Versions
- 1.4. Future Plans
- 2. Format Details
- 2.1. Nonterminals
- 2.2. List Header
- 2.2.1. List Header Format
- 2.3. List Generation
- 2.3.1. List Generation Format
- 2.4. Directory Entry
- 2.4.1. Directory Entry Format
- 3. Usage Considerations
- 3.1. Caching
- 3.2. Retrieving Directory Information
- 3.3. Fallback Reliability
- A.1. Sample Data
- A.1.1. Sample Fallback List Header
- A.1.2. Sample Fallback List Generation
- A.1.3. Sample Fallback Entries
-
-1. Scope and Preliminaries
-
- This document describes the format of Tor's directory lists, which are
- compiled and hard-coded into the tor binary. There is currently one
- list: the fallback directory mirrors. This list is also parsed by other
- libraries, like stem and metrics-lib. Alternate Tor implementations can
- use this list to bootstrap from the latest public Tor directory
- information.
-
- The FallbackDir feature was introduced by proposal 210, and was first
- supported by Tor in Tor version 0.2.4.7-alpha. The first hard-coded
- list was shipped in 0.2.8.1-alpha.
-
- The hard-coded fallback directory list is located in the tor source
- repository at:
-
- src/app/config/fallback_dirs.inc
-
- In Tor 0.3.4 and earlier, the list is located at:
-
- src/or/fallback_dirs.inc
-
- This document describes version 2.0.0 and later of the directory list
- format.
-
- Legacy, semi-structured versions of the fallback list were released with
- Tor 0.2.8.1-alpha through Tor 0.3.1.9. We call this format version 1.
- Stem and Relay Search have parsers for this legacy format.
-
-1.1. Format Overview
-
- A directory list is a C code fragment containing an array of C string
- constants. Each double-quoted C string constant is a valid torrc
- FallbackDir entry. Each entry contains various data fields.
-
- Directory lists do not include the C array's declaration, or the array's
- terminating NULL. Entries in directory lists do not include the
- FallbackDir torrc option. These are handled by the including C code.
-
- Directory lists also include C-style comments and whitespace. The
- presence of whitespace may be significant, but the amount of whitespace
- is never significant. The type of whitespace is not significant to the
- C compiler or Tor C string parser. However, other parsers MAY rely on
- the distinction between newlines and spaces. (And that the only
- whitespace characters in the list are newlines and spaces.)
-
- The directory entry C string constants are split over multiple lines for
- readability. Structured C-style comments are used to provide additional
- data fields. This information is not used by Tor, but may be of interest
- to other libraries.
-
- The order of directory entries and data fields is not significant,
- except where noted below.
-
-1.2. Acknowledgements
-
- The original fallback directory script and format was created by
- weasel. The current script uses code written by gsathya & karsten.
-
- This specification was revised after feedback from:
-
- Damian Johnson ("atagar")
- Iain R. Learmonth ("irl")
-
-1.3. Format Versions
-
- The directory list format uses semantic versioning: https://semver.org
-
- In particular:
- * major versions are used for incompatible changes, like
- removing non-optional fields
- * minor versions are used for compatible changes, like adding
- fields
- * patch versions are for bug fixes, like fixing an
- incorrectly-formatted Summary item
-
- 1.0.0 - The legacy fallback directory list format
-
- 2.0.0 - Adds name and extrainfo structured comments, and section separator
- comments to make the list easier to parses. Also adds a source list
- comment to the header.
-
- 3.0.0 - Modifies the format of the source list comment.
-
-1.4. Future Plans
-
- Tor also has an auth_dirs.inc file, but it is not yet in this format.
- Tor uses slightly different formats for authorities and fallback
- directory mirrors, so we will need to make some changes to tor so that
- it parses this format. (We will also need to add authority-specific
- information to this format.) See #24818 for details.
-
- We want to add a torrc option so operators can opt-in their relays as
- fallback directory mirrors. This gives us a signed opt-in confirmation.
- (We can also continue to accept whitelist entries, and do other checks.)
- We need to write a short proposal, and make some changes to tor and the
- fallback update script. See #24839 for details.
-
-2. Format Details
-
- Directory lists contain the following sections:
-
- - List Header (exactly once)
- - List Generation (exactly once, may be empty)
- - Directory Entry (zero or more times)
-
- Each section (or entry) ends with a separator.
-
-2.1. Nonterminals
-
- The following nonterminals are defined in the Onionoo details document
- specification:
-
- dir_address
- fingerprint
- nickname
-
- See https://metrics.torproject.org/onionoo.html#details
-
- The following nonterminals are defined in the "Tor directory protocol"
- specification in dir-spec.txt:
-
- Keyword
- ArgumentChar
- NL (newline)
- SP (space)
- bool (must not be confused with Onionoo's JSON "boolean")
-
- We derive the following nonterminals from Onionoo and dir-spec.txt:
-
- ipv4_or_port ::= port from an IPv4 or_addresses item
-
- The ipv4_or_port is the port part of an IPv4 address from the
- Onionoo or_addresses list.
-
- ipv6_or_address ::= an IPv6 or_addresses item
-
- The ipv6_or_address is an IPv6 address and port from the Onionoo
- or_addresses list. The address MAY be in the canonical RFC 5952
- IPv6 address format.
-
- A key-value pair:
-
- value ::= Zero or more ArgumentChar, excluding the following strings:
- * a double quotation mark (DQUOTE), and
- * the C comment terminators ("/*" and "*/").
-
- Note that the C++ comment ("//") and equals sign ("=") are
- not excluded, because they are reserved for future use in
- base64 values.
-
- key_value ::= Keyword "=" value
-
- We also define these additional nonterminals:
-
- number ::= An optional negative sign ("-"), followed by one or more
- numeric characters ([0-9]), with an optional decimal part
- (".", followed by one or more numeric characters).
-
- separator ::= "/*" SP+ "=====" SP+ "*/"
-
-2.2. List Header
-
- The list header consists of a number of key-value pairs, embedded in
- C-style comments.
-
-2.2.1. List Header Format
-
- "/*" SP+ "type=" Keyword SP+ "*/" SP* NL
-
- [At start, exactly once.]
-
- The type of directory entries in the list. Parsers SHOULD exit with
- an error if this is not the first line of the list, or if the value
- is anything other than "fallback".
-
- "/*" SP+ "version=" version_number SP+ "*/" SP* NL
-
- [In second position, exactly once.]
-
- The version of the directory list format.
-
- version_number is a semantic version, see the "Format Versions"
- section for details.
-
- Version 1.0.0 represents the undocumented, legacy fallback list
- format(s). Version 2.0.0 and later are documented by this
- specification.
-
- "/*" SP+ "timestamp=" number SP+ "*/" SP* NL
-
- [Exactly once.]
-
- A positive integer that indicates when this directory list was
- generated. This timestamp is guaranteed to increase for every
- version 2.0.0 and later directory list.
-
- The current timestamp format is YYYYMMDDHHMMSS, as an integer.
-
- "/*" SP+ "source=" Keyword ("," Keyword)* SP+ "*/" SP* NL
-
- [Zero or one time.]
-
- A list of the sources of the directory entries in the list.
-
- As of version 3.0.0, the possible sources are:
- * "offer-list" - the fallback_offer_list file in the fallback-scripts
- repository.
- * "descriptor" - one or more signed descriptors, each containing an
- "offer-fallback-dir" line. This feature will be
- implemented in ticket #24839.
- * "fallback" - a fallback_dirs.inc file from a tor repository.
- Used in check_existing mode.
-
- Before #24839 is implemented, the default is "offer-list". During the
- transition to signed offers, it will be "descriptor,offer-list".
- Afterwards, it will be "descriptor".
-
- In version 2.0.0, only one source name was allowed after "source=",
- and the deprecated "whitelist" source name was used instead of
- "offer-list".
-
- This line was added in version 2.0.0 of this specification. The format
- of this line was modified in version 3.0.0 of this specification.
-
- "/*" SP+ key_value SP+ "*/" SP* NL
-
- [Zero or more times.]
-
- Future releases may include additional header fields. Parsers MUST NOT
- rely on the order of these additional fields. Additional header fields
- will be accompanied by a minor version increment.
-
- separator SP* NL
-
- The list header ends with the section separator.
-
-2.3. List Generation
-
- The list generation information consists of human-readable prose
- describing the content and origin of this directory list. It is contained
- in zero or more C-style comments, and may contain multi-line comments and
- uncommented C code.
-
- In particular, this section may contain C-style comments that contain
- an equals ("=") character. It may also be entirely empty.
-
- Future releases may arbitrarily change the content of this section.
- Parsers MUST NOT rely on a version increment when the format changes.
-
-2.3.1. List Generation Format
-
- In general, parsers MUST NOT rely on the format of this section.
-
- Parsers MAY rely on the following details:
-
- The list generation section MUST NOT be a valid directory entry.
-
- The list generation summary MUST end with a section separator:
-
- separator SP* NL
-
- There MUST NOT be any section separators in the list generation
- section, other than the terminating section separator.
-
-2.4. Directory Entry
-
- A directory entry consists of a C string constant, and one or more
- C-style comments. The C string constant is a valid argument to the
- DirAuthority or FallbackDir torrc option. The section also contains
- additional key-value fields in C-style comments.
-
- The list of fallback entries does not include the directory
- authorities: they are in a separate list. (The Tor implementation combines
- these lists after parsing them, and applies the DirAuthorityFallbackRate
- to their weights.)
-
-2.4.1. Directory Entry Format
-
- If a directory entry does not conform to this format, the entry SHOULD
- be ignored by parsers.
-
- DQUOTE dir_address SP+ "orport=" ipv4_or_port SP+
- "id=" fingerprint DQUOTE SP* NL
-
- [At start, exactly once, on a single line.]
-
- This line consists of the following fields:
-
- dir_address
-
- An IPv4 address and DirPort for this directory, as defined by
- Onionoo. In this format version, all IPv4 addresses and DirPorts
- are guaranteed to be non-zero. (For IPv4 addresses, this means
- that they are not equal to "0.0.0.0".)
-
- ipv4_or_port
-
- An IPv4 ORPort for this directory, derived from Onionoo. In this
- format version, all IPv4 ORPorts are guaranteed to be non-zero.
-
- fingerprint
-
- The relay fingerprint of this directory, as defined by Onionoo.
- All relay fingerprints are guaranteed to have one or more non-zero
- digits.
-
- Note:
-
- Each double-quoted C string line that occurs after the first line,
- starts with space inside the quotes. This is a requirement of the
- Tor implementation.
-
- DQUOTE SP+ "ipv6=" ipv6_or_address DQUOTE SP* NL
-
- [Zero or one time.]
-
- The IPv6 address and ORPort for this directory, as defined by
- Onionoo. If present, IPv6 addresses and ORPorts are guaranteed to be
- non-zero. (For IPv6 addresses, this means that they are not equal to
- "[::]".)
-
- DQUOTE SP+ "weight=" number DQUOTE SP* NL
-
- [Zero or one time.]
-
- A non-negative, real-numbered weight for this directory.
- The default fallback weight is 1.0, and the default
- DirAuthorityFallbackRate is 1.0 in legacy Tor versions, and 0.1 in
- recent Tor versions.
-
- weight was removed in version 2.0.0, but is documented because it
- may be of interest to libraries implementing Tor's fallback
- behaviour.
-
- DQUOTE SP+ key_value DQUOTE SP* NL
-
- [Zero or more times.]
-
- Future releases may include additional data fields in double-quoted
- C string constants. Parsers MUST NOT rely on the order of these
- additional fields. Additional data fields will be accompanied by a
- minor version increment.
-
- "/*" SP+ "nickname=" nickname* SP+ "*/" SP* NL
-
- [Exactly once.]
-
- The nickname for this directory, as defined by Onionoo. An
- empty nickname indicates that the nickname is unknown.
-
- The first fallback list in the 2.0.0 format had nickname lines, but
- they were all empty.
-
- "/*" SP+ "extrainfo=" bool SP+ "*/" SP* NL
-
- [Exactly once.]
-
- An integer flag that indicates whether this directory caches
- extra-info documents. Set to 1 if the directory claimed that it
- cached extra-info documents in its descriptor when the list was
- created. 0 indicates that it did not, or its descriptor was not
- available.
-
- The first fallback list in the 2.0.0 format had extrainfo lines, but
- they were all zero.
-
- "/*" SP+ key_value SP+ "*/" SP* NL
-
- [Zero or more times.]
-
- Future releases may include additional data fields in C-style
- comments. Parsers MUST NOT rely on the order of these additional
- fields. Additional data fields will be accompanied by a minor version
- increment.
-
- separator SP* NL
-
- [Exactly once.]
-
- Each directory entry ends with the section separator.
-
- "," SP* NL
-
- [Exactly once.]
-
- The comma terminates the C string constant. (Multiple C string
- constants separated by whitespace or comments are coalesced by
- the C compiler.)
-
-3. Usage Considerations
-
- This section contains recommended library behaviours. It does not affect
- the format of directory lists.
-
-3.1. Caching
-
- The fallback list typically changes once every 6-12 months. The data in
- the list represents the state of the fallback directory entries when the
- list was created. Fallbacks can and do change their details over time.
-
- Libraries SHOULD parse and cache the most recent version of these lists
- during their build or release processes. Libraries MUST NOT retrieve the
- lists by default every time they are deployed or executed.
-
- The latest fallback list can be retrieved from:
-
- https://gitweb.torproject.org/tor.git/plain/src/or/fallback_dirs.inc
-
- Libraries MUST NOT rely on the availability of the server that hosts
- these lists.
-
- The list can also be retrieved using:
-
- git clone https://git.torproject.org/tor.git
-
- If you just want the latest list, you may wish to perform a shallow
- clone.
-
-3.2. Retrieving Directory Information
-
- Some libraries retrieve directory documents directly from the Tor
- Directory Authorities. The directory authorities are designed to support
- Tor relay and client bootstrap, and MAY choose to rate-limit library
- access. Libraries MAY provide a user-agent in their requests, if they
- are not intended to support anonymous operation. (User agents are a
- fingerprinting vector.)
-
- Libraries SHOULD consider the potential load on the authorities, and
- whether other sources can meet their needs.
-
- Libraries that require high-uptime availability of Tor directory
- information should investigate the following options:
-
- * OnionOO: https://metrics.torproject.org/onionoo.html
- * Third-party OnionOO mirrors are also available
- * CollecTor: https://collector.torproject.org/
- * Fallback Directory Mirrors
-
- Onionoo and CollecTor are typically updated every hour on a regular
- schedule. Fallbacks update their own directory information at random
- intervals, see dir-spec for details.
-
-3.3. Fallback Reliability
-
- The fallback list is typically regenerated when the fallback failure
- rate exceeds 25%. Libraries SHOULD NOT rely on any particular fallback
- being available, or some proportion of fallbacks being available.
-
- Libraries that use fallbacks MAY wish to query an authority after a
- few fallback queries fail. For example, Tor clients try 3-4 fallbacks
- before trying an authority.
-
-A.1. Sample Data
-
- A sample version 2.0.0 fallback list is available here:
-
- https://trac.torproject.org/projects/tor/raw-attachment/ticket/22759/fallback_dirs_new_format_version.4.inc
-
- A sample transitional version 2.0.0 fallback list is available here:
-
- https://raw.githubusercontent.com/teor2345/tor/fallback-format-2-v4/src/or/fallback_dirs.inc
-
-A.1.1. Sample Fallback List Header
-
-/* type=fallback */
-/* version=2.0.0 */
-/* ===== */
-
-A.1.2. Sample Fallback List Generation
-
-/* Whitelist & blacklist excluded 1326 of 1513 candidates. */
-/* Checked IPv4 DirPorts served a consensus within 15.0s. */
-/*
-Final Count: 151 (Eligible 187, Target 392 (1963 * 0.20), Max 200)
-Excluded: 36 (Same Operator 27, Failed/Skipped Download 9, Excess 0)
-Bandwidth Range: 1.3 - 40.0 MByte/s
-*/
-/*
-Onionoo Source: details Date: 2017-05-16 07:00:00 Version: 4.0
-URL: https:onionoo.torproject.orgdetails?fields=fingerprint%2Cnickname%2Ccontact%2Clast_changed_address_or_port%2Cconsensus_weight%2Cadvertised_bandwidth%2Cor_addresses%2Cdir_address%2Crecommended_version%2Cflags%2Ceffective_family%2Cplatform&flag=V2Dir&type=relay&last_seen_days=-0&first_seen_days=30-
-*/
-/*
-Onionoo Source: uptime Date: 2017-05-16 07:00:00 Version: 4.0
-URL: https:onionoo.torproject.orguptime?first_seen_days=30-&flag=V2Dir&type=relay&last_seen_days=-0
-*/
-/* ===== */
-
-A.1.3. Sample Fallback Entries
-
-"176.10.104.240:80 orport=443 id=0111BA9B604669E636FFD5B503F382A4B7AD6E80"
-/* nickname=foo */
-/* extrainfo=1 */
-/* ===== */
-,
-"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB"
-" ipv6=[2a01:4f8:162:51e2::2]:9001"
-/* nickname= */
-/* extrainfo=0 */
-/* ===== */
-,