From e4e0d93d56ee8c1aec4c2efaa7046b651f0fe55c Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Thu, 12 Oct 2023 12:27:58 -0400 Subject: Move all text-only specifications into the OLD_TXT directory. --- dir-list-spec.txt | 529 ------------------------------------------------------ 1 file changed, 529 deletions(-) delete mode 100644 dir-list-spec.txt (limited to 'dir-list-spec.txt') diff --git a/dir-list-spec.txt b/dir-list-spec.txt deleted file mode 100644 index 65af536..0000000 --- a/dir-list-spec.txt +++ /dev/null @@ -1,529 +0,0 @@ - - Tor Directory List Format - Tim Wilson-Brown (teor) - -Table of Contents - - 1. Scope and Preliminaries - 1.1. Format Overview - 1.2. Acknowledgements - 1.3. Format Versions - 1.4. Future Plans - 2. Format Details - 2.1. Nonterminals - 2.2. List Header - 2.2.1. List Header Format - 2.3. List Generation - 2.3.1. List Generation Format - 2.4. Directory Entry - 2.4.1. Directory Entry Format - 3. Usage Considerations - 3.1. Caching - 3.2. Retrieving Directory Information - 3.3. Fallback Reliability - A.1. Sample Data - A.1.1. Sample Fallback List Header - A.1.2. Sample Fallback List Generation - A.1.3. Sample Fallback Entries - -1. Scope and Preliminaries - - This document describes the format of Tor's directory lists, which are - compiled and hard-coded into the tor binary. There is currently one - list: the fallback directory mirrors. This list is also parsed by other - libraries, like stem and metrics-lib. Alternate Tor implementations can - use this list to bootstrap from the latest public Tor directory - information. - - The FallbackDir feature was introduced by proposal 210, and was first - supported by Tor in Tor version 0.2.4.7-alpha. The first hard-coded - list was shipped in 0.2.8.1-alpha. - - The hard-coded fallback directory list is located in the tor source - repository at: - - src/app/config/fallback_dirs.inc - - In Tor 0.3.4 and earlier, the list is located at: - - src/or/fallback_dirs.inc - - This document describes version 2.0.0 and later of the directory list - format. - - Legacy, semi-structured versions of the fallback list were released with - Tor 0.2.8.1-alpha through Tor 0.3.1.9. We call this format version 1. - Stem and Relay Search have parsers for this legacy format. - -1.1. Format Overview - - A directory list is a C code fragment containing an array of C string - constants. Each double-quoted C string constant is a valid torrc - FallbackDir entry. Each entry contains various data fields. - - Directory lists do not include the C array's declaration, or the array's - terminating NULL. Entries in directory lists do not include the - FallbackDir torrc option. These are handled by the including C code. - - Directory lists also include C-style comments and whitespace. The - presence of whitespace may be significant, but the amount of whitespace - is never significant. The type of whitespace is not significant to the - C compiler or Tor C string parser. However, other parsers MAY rely on - the distinction between newlines and spaces. (And that the only - whitespace characters in the list are newlines and spaces.) - - The directory entry C string constants are split over multiple lines for - readability. Structured C-style comments are used to provide additional - data fields. This information is not used by Tor, but may be of interest - to other libraries. - - The order of directory entries and data fields is not significant, - except where noted below. - -1.2. Acknowledgements - - The original fallback directory script and format was created by - weasel. The current script uses code written by gsathya & karsten. - - This specification was revised after feedback from: - - Damian Johnson ("atagar") - Iain R. Learmonth ("irl") - -1.3. Format Versions - - The directory list format uses semantic versioning: https://semver.org - - In particular: - * major versions are used for incompatible changes, like - removing non-optional fields - * minor versions are used for compatible changes, like adding - fields - * patch versions are for bug fixes, like fixing an - incorrectly-formatted Summary item - - 1.0.0 - The legacy fallback directory list format - - 2.0.0 - Adds name and extrainfo structured comments, and section separator - comments to make the list easier to parses. Also adds a source list - comment to the header. - - 3.0.0 - Modifies the format of the source list comment. - -1.4. Future Plans - - Tor also has an auth_dirs.inc file, but it is not yet in this format. - Tor uses slightly different formats for authorities and fallback - directory mirrors, so we will need to make some changes to tor so that - it parses this format. (We will also need to add authority-specific - information to this format.) See #24818 for details. - - We want to add a torrc option so operators can opt-in their relays as - fallback directory mirrors. This gives us a signed opt-in confirmation. - (We can also continue to accept whitelist entries, and do other checks.) - We need to write a short proposal, and make some changes to tor and the - fallback update script. See #24839 for details. - -2. Format Details - - Directory lists contain the following sections: - - - List Header (exactly once) - - List Generation (exactly once, may be empty) - - Directory Entry (zero or more times) - - Each section (or entry) ends with a separator. - -2.1. Nonterminals - - The following nonterminals are defined in the Onionoo details document - specification: - - dir_address - fingerprint - nickname - - See https://metrics.torproject.org/onionoo.html#details - - The following nonterminals are defined in the "Tor directory protocol" - specification in dir-spec.txt: - - Keyword - ArgumentChar - NL (newline) - SP (space) - bool (must not be confused with Onionoo's JSON "boolean") - - We derive the following nonterminals from Onionoo and dir-spec.txt: - - ipv4_or_port ::= port from an IPv4 or_addresses item - - The ipv4_or_port is the port part of an IPv4 address from the - Onionoo or_addresses list. - - ipv6_or_address ::= an IPv6 or_addresses item - - The ipv6_or_address is an IPv6 address and port from the Onionoo - or_addresses list. The address MAY be in the canonical RFC 5952 - IPv6 address format. - - A key-value pair: - - value ::= Zero or more ArgumentChar, excluding the following strings: - * a double quotation mark (DQUOTE), and - * the C comment terminators ("/*" and "*/"). - - Note that the C++ comment ("//") and equals sign ("=") are - not excluded, because they are reserved for future use in - base64 values. - - key_value ::= Keyword "=" value - - We also define these additional nonterminals: - - number ::= An optional negative sign ("-"), followed by one or more - numeric characters ([0-9]), with an optional decimal part - (".", followed by one or more numeric characters). - - separator ::= "/*" SP+ "=====" SP+ "*/" - -2.2. List Header - - The list header consists of a number of key-value pairs, embedded in - C-style comments. - -2.2.1. List Header Format - - "/*" SP+ "type=" Keyword SP+ "*/" SP* NL - - [At start, exactly once.] - - The type of directory entries in the list. Parsers SHOULD exit with - an error if this is not the first line of the list, or if the value - is anything other than "fallback". - - "/*" SP+ "version=" version_number SP+ "*/" SP* NL - - [In second position, exactly once.] - - The version of the directory list format. - - version_number is a semantic version, see the "Format Versions" - section for details. - - Version 1.0.0 represents the undocumented, legacy fallback list - format(s). Version 2.0.0 and later are documented by this - specification. - - "/*" SP+ "timestamp=" number SP+ "*/" SP* NL - - [Exactly once.] - - A positive integer that indicates when this directory list was - generated. This timestamp is guaranteed to increase for every - version 2.0.0 and later directory list. - - The current timestamp format is YYYYMMDDHHMMSS, as an integer. - - "/*" SP+ "source=" Keyword ("," Keyword)* SP+ "*/" SP* NL - - [Zero or one time.] - - A list of the sources of the directory entries in the list. - - As of version 3.0.0, the possible sources are: - * "offer-list" - the fallback_offer_list file in the fallback-scripts - repository. - * "descriptor" - one or more signed descriptors, each containing an - "offer-fallback-dir" line. This feature will be - implemented in ticket #24839. - * "fallback" - a fallback_dirs.inc file from a tor repository. - Used in check_existing mode. - - Before #24839 is implemented, the default is "offer-list". During the - transition to signed offers, it will be "descriptor,offer-list". - Afterwards, it will be "descriptor". - - In version 2.0.0, only one source name was allowed after "source=", - and the deprecated "whitelist" source name was used instead of - "offer-list". - - This line was added in version 2.0.0 of this specification. The format - of this line was modified in version 3.0.0 of this specification. - - "/*" SP+ key_value SP+ "*/" SP* NL - - [Zero or more times.] - - Future releases may include additional header fields. Parsers MUST NOT - rely on the order of these additional fields. Additional header fields - will be accompanied by a minor version increment. - - separator SP* NL - - The list header ends with the section separator. - -2.3. List Generation - - The list generation information consists of human-readable prose - describing the content and origin of this directory list. It is contained - in zero or more C-style comments, and may contain multi-line comments and - uncommented C code. - - In particular, this section may contain C-style comments that contain - an equals ("=") character. It may also be entirely empty. - - Future releases may arbitrarily change the content of this section. - Parsers MUST NOT rely on a version increment when the format changes. - -2.3.1. List Generation Format - - In general, parsers MUST NOT rely on the format of this section. - - Parsers MAY rely on the following details: - - The list generation section MUST NOT be a valid directory entry. - - The list generation summary MUST end with a section separator: - - separator SP* NL - - There MUST NOT be any section separators in the list generation - section, other than the terminating section separator. - -2.4. Directory Entry - - A directory entry consists of a C string constant, and one or more - C-style comments. The C string constant is a valid argument to the - DirAuthority or FallbackDir torrc option. The section also contains - additional key-value fields in C-style comments. - - The list of fallback entries does not include the directory - authorities: they are in a separate list. (The Tor implementation combines - these lists after parsing them, and applies the DirAuthorityFallbackRate - to their weights.) - -2.4.1. Directory Entry Format - - If a directory entry does not conform to this format, the entry SHOULD - be ignored by parsers. - - DQUOTE dir_address SP+ "orport=" ipv4_or_port SP+ - "id=" fingerprint DQUOTE SP* NL - - [At start, exactly once, on a single line.] - - This line consists of the following fields: - - dir_address - - An IPv4 address and DirPort for this directory, as defined by - Onionoo. In this format version, all IPv4 addresses and DirPorts - are guaranteed to be non-zero. (For IPv4 addresses, this means - that they are not equal to "0.0.0.0".) - - ipv4_or_port - - An IPv4 ORPort for this directory, derived from Onionoo. In this - format version, all IPv4 ORPorts are guaranteed to be non-zero. - - fingerprint - - The relay fingerprint of this directory, as defined by Onionoo. - All relay fingerprints are guaranteed to have one or more non-zero - digits. - - Note: - - Each double-quoted C string line that occurs after the first line, - starts with space inside the quotes. This is a requirement of the - Tor implementation. - - DQUOTE SP+ "ipv6=" ipv6_or_address DQUOTE SP* NL - - [Zero or one time.] - - The IPv6 address and ORPort for this directory, as defined by - Onionoo. If present, IPv6 addresses and ORPorts are guaranteed to be - non-zero. (For IPv6 addresses, this means that they are not equal to - "[::]".) - - DQUOTE SP+ "weight=" number DQUOTE SP* NL - - [Zero or one time.] - - A non-negative, real-numbered weight for this directory. - The default fallback weight is 1.0, and the default - DirAuthorityFallbackRate is 1.0 in legacy Tor versions, and 0.1 in - recent Tor versions. - - weight was removed in version 2.0.0, but is documented because it - may be of interest to libraries implementing Tor's fallback - behaviour. - - DQUOTE SP+ key_value DQUOTE SP* NL - - [Zero or more times.] - - Future releases may include additional data fields in double-quoted - C string constants. Parsers MUST NOT rely on the order of these - additional fields. Additional data fields will be accompanied by a - minor version increment. - - "/*" SP+ "nickname=" nickname* SP+ "*/" SP* NL - - [Exactly once.] - - The nickname for this directory, as defined by Onionoo. An - empty nickname indicates that the nickname is unknown. - - The first fallback list in the 2.0.0 format had nickname lines, but - they were all empty. - - "/*" SP+ "extrainfo=" bool SP+ "*/" SP* NL - - [Exactly once.] - - An integer flag that indicates whether this directory caches - extra-info documents. Set to 1 if the directory claimed that it - cached extra-info documents in its descriptor when the list was - created. 0 indicates that it did not, or its descriptor was not - available. - - The first fallback list in the 2.0.0 format had extrainfo lines, but - they were all zero. - - "/*" SP+ key_value SP+ "*/" SP* NL - - [Zero or more times.] - - Future releases may include additional data fields in C-style - comments. Parsers MUST NOT rely on the order of these additional - fields. Additional data fields will be accompanied by a minor version - increment. - - separator SP* NL - - [Exactly once.] - - Each directory entry ends with the section separator. - - "," SP* NL - - [Exactly once.] - - The comma terminates the C string constant. (Multiple C string - constants separated by whitespace or comments are coalesced by - the C compiler.) - -3. Usage Considerations - - This section contains recommended library behaviours. It does not affect - the format of directory lists. - -3.1. Caching - - The fallback list typically changes once every 6-12 months. The data in - the list represents the state of the fallback directory entries when the - list was created. Fallbacks can and do change their details over time. - - Libraries SHOULD parse and cache the most recent version of these lists - during their build or release processes. Libraries MUST NOT retrieve the - lists by default every time they are deployed or executed. - - The latest fallback list can be retrieved from: - - https://gitweb.torproject.org/tor.git/plain/src/or/fallback_dirs.inc - - Libraries MUST NOT rely on the availability of the server that hosts - these lists. - - The list can also be retrieved using: - - git clone https://git.torproject.org/tor.git - - If you just want the latest list, you may wish to perform a shallow - clone. - -3.2. Retrieving Directory Information - - Some libraries retrieve directory documents directly from the Tor - Directory Authorities. The directory authorities are designed to support - Tor relay and client bootstrap, and MAY choose to rate-limit library - access. Libraries MAY provide a user-agent in their requests, if they - are not intended to support anonymous operation. (User agents are a - fingerprinting vector.) - - Libraries SHOULD consider the potential load on the authorities, and - whether other sources can meet their needs. - - Libraries that require high-uptime availability of Tor directory - information should investigate the following options: - - * OnionOO: https://metrics.torproject.org/onionoo.html - * Third-party OnionOO mirrors are also available - * CollecTor: https://collector.torproject.org/ - * Fallback Directory Mirrors - - Onionoo and CollecTor are typically updated every hour on a regular - schedule. Fallbacks update their own directory information at random - intervals, see dir-spec for details. - -3.3. Fallback Reliability - - The fallback list is typically regenerated when the fallback failure - rate exceeds 25%. Libraries SHOULD NOT rely on any particular fallback - being available, or some proportion of fallbacks being available. - - Libraries that use fallbacks MAY wish to query an authority after a - few fallback queries fail. For example, Tor clients try 3-4 fallbacks - before trying an authority. - -A.1. Sample Data - - A sample version 2.0.0 fallback list is available here: - - https://trac.torproject.org/projects/tor/raw-attachment/ticket/22759/fallback_dirs_new_format_version.4.inc - - A sample transitional version 2.0.0 fallback list is available here: - - https://raw.githubusercontent.com/teor2345/tor/fallback-format-2-v4/src/or/fallback_dirs.inc - -A.1.1. Sample Fallback List Header - -/* type=fallback */ -/* version=2.0.0 */ -/* ===== */ - -A.1.2. Sample Fallback List Generation - -/* Whitelist & blacklist excluded 1326 of 1513 candidates. */ -/* Checked IPv4 DirPorts served a consensus within 15.0s. */ -/* -Final Count: 151 (Eligible 187, Target 392 (1963 * 0.20), Max 200) -Excluded: 36 (Same Operator 27, Failed/Skipped Download 9, Excess 0) -Bandwidth Range: 1.3 - 40.0 MByte/s -*/ -/* -Onionoo Source: details Date: 2017-05-16 07:00:00 Version: 4.0 -URL: https:onionoo.torproject.orgdetails?fields=fingerprint%2Cnickname%2Ccontact%2Clast_changed_address_or_port%2Cconsensus_weight%2Cadvertised_bandwidth%2Cor_addresses%2Cdir_address%2Crecommended_version%2Cflags%2Ceffective_family%2Cplatform&flag=V2Dir&type=relay&last_seen_days=-0&first_seen_days=30- -*/ -/* -Onionoo Source: uptime Date: 2017-05-16 07:00:00 Version: 4.0 -URL: https:onionoo.torproject.orguptime?first_seen_days=30-&flag=V2Dir&type=relay&last_seen_days=-0 -*/ -/* ===== */ - -A.1.3. Sample Fallback Entries - -"176.10.104.240:80 orport=443 id=0111BA9B604669E636FFD5B503F382A4B7AD6E80" -/* nickname=foo */ -/* extrainfo=1 */ -/* ===== */ -, -"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB" -" ipv6=[2a01:4f8:162:51e2::2]:9001" -/* nickname= */ -/* extrainfo=0 */ -/* ===== */ -, -- cgit v1.2.3-54-g00ecf