diff options
Diffstat (limited to 'doc')
26 files changed, 1168 insertions, 755 deletions
diff --git a/doc/HACKING/CircuitPaddingDevelopment.md b/doc/HACKING/CircuitPaddingDevelopment.md index a4e65697b8..95ffbae4dd 100644 --- a/doc/HACKING/CircuitPaddingDevelopment.md +++ b/doc/HACKING/CircuitPaddingDevelopment.md @@ -150,13 +150,13 @@ might be recognized on the wire. The problem of differentiating Tor traffic from non-Tor traffic based on TCP/TLS packet sizes, initial handshake patterns, and DPI characteristics is the domain of [pluggable -transports](https://trac.torproject.org/projects/tor/wiki/doc/AChildsGardenOfPluggableTransports), +transports](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/AChildsGardenOfPluggableTransports), which may optionally be used in conjunction with this framework (or without it). This document focuses primarily on the circuit padding framework's cover traffic features, and will only briefly touch on the potential obfuscation and -application layer coupling points of the framework. Explicit layer coupling +application layer coupling points of the framework. Explicit layer coupling points can be created by adding either new [machine application events](#62-machine-application-events) or new [internal machine events](#63-internal-machine-events) to the circuit padding framework, so that @@ -178,7 +178,7 @@ Most importantly: this framing allows cover traffic defenses to be modeled as an optimization problem search space, expressed as fields of a C structure (which is simultaneously a compact opaque bitstring as well as a symbolic vector in an abstract feature space). This kind of space is particularly well -suited to search by gradient descent, GAs, and GANs. +suited to search by gradient descent, GAs, and GANs. When performing this optimization search, each padding machine should have a fitness function, which will allow two padding machines to be compared for @@ -186,7 +186,7 @@ relative effectiveness. Optimization searches work best if this fitness can be represented as a single number, for example the total amount by which it reduces the [Balanced Accuracy](https://en.wikipedia.org/wiki/Precision_and_recall#Imbalanced_Data) -of an adversary's classifier, divided by an amount of traffic overhead. +of an adversary's classifier, divided by an amount of traffic overhead. Before you begin the optimization phase for your defense, you should also carefully consider the [features and @@ -1024,11 +1024,11 @@ would gladly accept patches that do so. The following list gives an overview of these improvements, but as this document ages, it may become stale. The canonical list of improvements that -researchers may find useful is tagged in our bugtracker with -[circpad-researchers](https://trac.torproject.org/projects/tor/query?keywords=~circpad-researchers), +researchers may find useful is labeled in our bugtracker with +[Padding Research](https://gitlab.torproject.org/tpo/core/tor/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Padding%20Research), and the list of improvements that are known to be necessary for some research -areas are tagged with -[circpad-researchers-want](https://trac.torproject.org/projects/tor/query?keywords=~circpad-researchers-want). +areas are labeled with +[Padding Research Requires](https://gitlab.torproject.org/tpo/core/tor/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Padding%20Research%20Requires). Please consult those lists for the latest status of these issues. Note that not all fixes will be backported to all Tor versions, so be mindful of which diff --git a/doc/HACKING/CircuitPaddingQuickStart.md b/doc/HACKING/CircuitPaddingQuickStart.md index 2780b5c6ea..25bf05048c 100644 --- a/doc/HACKING/CircuitPaddingQuickStart.md +++ b/doc/HACKING/CircuitPaddingQuickStart.md @@ -7,7 +7,7 @@ circuit padding framework from scratch. Notes were taken as part of porting [Adaptive Padding Early (APE)](https://www.cs.kau.se/pulls/hot/thebasketcase-ape/) from basket2 to the circuit padding framework. The goal is just to document the process and provide -useful pointers along the way, not create a useful machine. +useful pointers along the way, not create a useful machine. The quick and dirty plan is to: 1. clone and compile tor @@ -18,20 +18,20 @@ The quick and dirty plan is to: ## Clone and compile tor -```bash -git clone https://git.torproject.org/tor.git -cd tor -git checkout tor-0.4.1.5 +```console +$ git clone https://git.torproject.org/tor.git +$ cd tor +$ git checkout tor-0.4.1.5 ``` Above we use the tag for tor-0.4.1.5 where the circuit padding framework was released. Note that this version of the framework is missing many features and fixes that have since been merged to origin/master. If you need the newest framework features, you should use that master instead. -```bash -sh autogen.sh -./configure -make +```console +$ sh autogen.sh +$ ./configure +$ make ``` When you run `./configure` you'll be told of missing dependencies and packages to install on debian-based distributions. Important: if you plan to run `tor` on @@ -73,7 +73,7 @@ We have one more step left before we move on the machine: configure TB to always use our middle relay. Edit `Browser/TorBrowser/Data/Tor/torrc` and set `MiddleNodes <fingerprint>`, where `<fingerprint>` is the fingerprint of the relay. Start TB, visit a website, and manually confirm that the middle is used -by looking at the circuit display. +by looking at the circuit display. ## Add a bare-bones APE padding machine @@ -89,10 +89,10 @@ might be broken in the future, just search for the headings): [circuitpadding_machines.h](https://gitweb.torproject.org/tor.git/tree/src/core/or/circuitpadding_machines.h). Please consult the above links for details. Moving forward, the focus is to -describe what was done, not necessarily explaining all the details why. +describe what was done, not necessarily explaining all the details why. Since we plan to make changes to tor, create a new branch `git checkout -b -circuit-padding-ape-machine tor-0.4.1.5`. +circuit-padding-ape-machine tor-0.4.1.5`. We start with declaring two functions, one for the machine at the client and one at the relay, in `circuitpadding_machines.h`: @@ -107,8 +107,8 @@ The definitions go into `circuitpadding_machines.c`: ```c /**************** Adaptive Padding Early (APE) machine ****************/ -/** - * Create a relay-side padding machine based on the APE design. +/** + * Create a relay-side padding machine based on the APE design. */ void circpad_machine_relay_wf_ape(smartlist_t *machines_sl) @@ -137,14 +137,14 @@ circpad_machine_relay_wf_ape(smartlist_t *machines_sl) // register the machine relay_machine->machine_num = smartlist_len(machines_sl); circpad_register_padding_machine(relay_machine, machines_sl); - + log_info(LD_CIRC, "Registered relay WF APE padding machine (%u)", relay_machine->machine_num); } -/** - * Create a client-side padding machine based on the APE design. +/** + * Create a client-side padding machine based on the APE design. */ void circpad_machine_client_wf_ape(smartlist_t *machines_sl) @@ -186,18 +186,18 @@ We also have to modify `circpad_machines_init()` in `circuitpadding.c` to register our machines: ```c - /* Register machines for the APE WF defense */ - circpad_machine_client_wf_ape(origin_padding_machines); - circpad_machine_relay_wf_ape(relay_padding_machines); +/* Register machines for the APE WF defense */ +circpad_machine_client_wf_ape(origin_padding_machines); +circpad_machine_relay_wf_ape(relay_padding_machines); ``` -We run `make` to get a new `tor` binary and copy it to our local TB. +We run `make` to get a new `tor` binary and copy it to our local TB. ## Run the machine To be able to view circuit info events in the console as we launch TB, we add `Log -[circ]info notice stdout` to `torrc` of TB. +[circ]info notice stdout` to `torrc` of TB. Running TB to visit example.com we first find in the log: @@ -224,13 +224,13 @@ Aug 30 18:36:55.000 [info] circpad_handle_padding_negotiated(): Middle node did We see that our middle support padding (since we upgraded to tor-0.4.1.5), that we attempt to negotiate, our machine starts on the client, transitions to the end state, and is freed. The last line shows that the middle doesn't have a -padding machine that can run. +padding machine that can run. Next, we follow the same steps as earlier and replace the modified `tor` at our middle relay. We don't update the logging there to avoid logging on the info level on the live network. Looking at the client log again we see that negotiation works as before except for the last line: it's missing, so the -machine is running at the middle as well. +machine is running at the middle as well. ## Implementing the APE state machine @@ -239,14 +239,14 @@ more machines (for the receive portion of WTFP-PAD, beyond AP), and pick reasonable parameters for the distributions (I completely winged it now, as when implementing APE). The [circuit-padding-ape-machine branch](https://github.com/pylls/tor/tree/circuit-padding-ape-machine) contains -the commits for the full machines with plenty of comments. +the commits for the full machines with plenty of comments. Some comments on the process: -- `tor-0.4.1.5` does not support two machines on the same circuit, the following - fix has to be made: https://trac.torproject.org/projects/tor/ticket/31111 . +- `tor-0.4.1.5` did not support two machines on the same circuit, the following + fix had to be made: https://bugs.torproject.org/tpo/core/tor/31111 . The good news is that everything else seems to work after the small change in - the fix. + the fix. - APE randomizes its distributions. Currently, this can only be done during start of `tor`. This makes sense in the censorship circumvention setting (`obfs4`), less so for WF defenses: further randomizing each circuit is likely diff --git a/doc/HACKING/CodeStructure.md b/doc/HACKING/CodeStructure.md deleted file mode 100644 index d387018f9b..0000000000 --- a/doc/HACKING/CodeStructure.md +++ /dev/null @@ -1,123 +0,0 @@ -# Code Structure - -TODO: revise this to talk about how things are, rather than how things -have changed. - -For quite a while now, the program *tor* has been built from source -code in just two directories: **src/common** and **src/or**. - -This has become more-or-less untenable, for a few reasons -- most -notably of which is that it has led our code to become more -spaghetti-ish than I can endorse with a clean conscience. - -So to fix that, we've gone and done a huge code movement in our git -master branch, which will land in a release once Tor `0.3.5.1-alpha` is -out. - -Here's what we did: - - * **src/common** has been turned into a set of static libraries. These -all live in the **src/lib/*** directories. The dependencies between -these libraries should have no cycles. The libraries are: - - - **arch** -- Headers to handle architectural differences - - **cc** -- headers to handle differences among compilers - - **compress** -- wraps zlib, zstd, lzma - - **container** -- high-level container types - - **crypt_ops** -- Cryptographic operations. Planning to split this into -a higher and lower level library - - **ctime** -- Operations that need to run in constant-time. (Properly, -data-invariant time) - - **defs** -- miscelaneous definitions needed throughout Tor. - - **encoding** -- transforming one data type into another, and various -data types into strings. - - **err** -- lowest-level error handling, in cases where we can't use -the logs because something that the logging system needs has broken. - - **evloop** -- Generic event-loop handling logic - - **fdio** -- Low-level IO wrapper functions for file descriptors. - - **fs** -- Operations on the filesystem - - **intmath** -- low-level integer math and misc bit-twiddling hacks - - **lock** -- low-level locking code - - **log** -- Tor's logging module. This library sits roughly halfway up -the library dependency diagram, since everything it depends on has to -be carefully crafted to *not* log. - - **malloc** -- Low-level wrappers for the platform memory allocation functions. - - **math** -- Higher-level mathematical functions, and floating-point math - - **memarea** -- An arena allocator - - **meminfo** -- Functions for querying the current process's memory -status and resources - - **net** -- Networking compatibility and convenience code - - **osinfo** -- Querying information about the operating system - - **process** -- Launching and querying the status of other processes - - **sandbox** -- Backend for the linux seccomp2 sandbox - - **smartlist_core** -- The lowest-level of the smartlist_t data type. -Separated from the rest of the containers library because the logging -subsystem depends on it. - - **string** -- Compatibility and convenience functions for manipulating -C strings. - - **term** -- Terminal-related functions (currently limited to a getpass -function). - - **testsupport** -- Macros for mocking, unit tests, etc. - - **thread** -- Higher-level thread compatibility code - - **time** -- Higher-level time management code, including format -conversions and monotonic time - - **tls** -- Our wrapper around our TLS library - - **trace** -- Formerly src/trace -- a generic event tracing API - - **wallclock** -- Low-level time code, used by the log module. - - * To ensure that the dependency graph in **src/common** remains under -control, there is a tool that you can run called `make -check-includes`. It verifies that each module in Tor only includes -the headers that it is permitted to include, using a per-directory -*.may_include* file. - - * The **src/or/or.h** header has been split into numerous smaller -headers. Notably, many important structures are now declared in a -header called *foo_st.h*, where "foo" is the name of the structure. - - * The **src/or** directory, which had most of Tor's code, had been split -up into several directories. This is still a work in progress: This -code has not itself been refactored, and its dependency graph is still -a tangled web. I hope we'll be working on that over the coming -releases, but it will take a while to do. - - - The new top-level source directories are: - - **src/core** -- Code necessary to actually perform or use onion routing. - - **src/feature** -- Code used only by some onion routing -configurations, or only for a special purpose. - - **src/app** -- Top-level code to run, invoke, and configure the -lower-level code - - - The new second-level source directories are: - - **src/core/crypto** -- High-level cryptographic protocols used in Tor - - **src/core/mainloop** -- Tor's event loop, connection-handling, and -traffic-routing code. - - **src/core/or** -- Parts related to handling onion routing itself - - **src/core/proto** -- support for encoding and decoding different -wire protocols - - **src/feature/api** -- Support for making Tor embeddable - - **src/feature/client** -- Functionality which only Tor clients need - - **src/feature/control** -- Controller implementation - - **src/feature/dirauth** -- Directory authority - - **src/feature/dircache** -- Directory cache - - **src/feature/dirclient** -- Directory client - - **src/feature/dircommon** -- Shared code between the other directory modules - - **src/feature/hibernate** -- Hibernating when Tor is out of bandwidth -or shutting down - - **src/feature/hs** -- v3 onion service implementation - - **src/feature/hs_common** -- shared code between both onion service -implementations - - **src/feature/nodelist** -- storing and accessing the list of relays on -the network. - - **src/feature/relay** -- code that only relay servers and exit servers need. - - **src/feature/rend** -- v2 onion service implementation - - **src/feature/stats** -- statistics and history - - **src/app/config** -- configuration and state for Tor - - **src/app/main** -- Top-level functions to invoke the rest or Tor. - - * The `tor` executable is now built in **src/app/tor** rather than **src/or/tor**. - - * There are more static libraries than before that you need to build -into your application if you want to embed Tor. Rather than -maintaining this list yourself, I recommend that you run `make -show-libs` to have Tor emit a list of what you need to link. diff --git a/doc/HACKING/CodingStandards.md b/doc/HACKING/CodingStandards.md index 150acf1852..cd3417d0b5 100644 --- a/doc/HACKING/CodingStandards.md +++ b/doc/HACKING/CodingStandards.md @@ -118,27 +118,36 @@ instance of the feature (--reverse). For example, for #30224, we wanted to know when the bridge-distribution-request feature was introduced into Tor: - $ git log -S bridge-distribution-request --reverse - commit ebab521525 - Author: Roger Dingledine <arma@torproject.org> - Date: Sun Nov 13 02:39:16 2016 -0500 - Add new BridgeDistribution config option +```console +$ git log -S bridge-distribution-request --reverse commit ebab521525 +Author: Roger Dingledine <arma@torproject.org> +Date: Sun Nov 13 02:39:16 2016 -0500 - $ git describe --contains ebab521525 - tor-0.3.2.3-alpha~15^2~4 + Add new BridgeDistribution config option + +$ git describe --contains ebab521525 +tor-0.3.2.3-alpha~15^2~4 +``` If you need to know all the Tor versions that contain a commit, use: - $ git tag --contains 9f2efd02a1 | sort -V - tor-0.2.5.16 - tor-0.2.8.17 - tor-0.2.9.14 - tor-0.2.9.15 - ... - tor-0.3.0.13 - tor-0.3.1.9 - tor-0.3.1.10 - ... + +```console +$ git tag --contains 9f2efd02a1 | sort -V +tor-0.2.5.16 +tor-0.2.8.17 +tor-0.2.9.14 +tor-0.2.9.15 +... +tor-0.3.0.13 +tor-0.3.1.9 +tor-0.3.1.10 +... +``` + +If a bug was introduced before the oldest currently supported release series +of Tor, and it's hard to track down where it was introduced, you may say +"bugfix on all supported versions of Tor." If at all possible, try to create the changes file in the same commit where you are making the change. Please give it a distinctive name that no other @@ -180,6 +189,14 @@ What needs a changes file? What does not need a changes file? * Bugfixes for code that hasn't shipped in any released version of Tor + * Any change to a file that is not distributed in the tarball. This + includes: + * Any change to our CI configuration that does not affect the distributed + source. + * Any change to developer-only tools, unless those tools are distributed + in the tarball. + * Non-functional code movement. + * Identifier re-namings, comment edits, spelling fixes, and so on. Why use changes files instead of Git commit messages? @@ -438,8 +455,10 @@ use `tor_assert_nonfatal()` in place of `tor_assert()`. If you'd like to write a conditional that incorporates a nonfatal assertion, use the `BUG()` macro, as in: - if (BUG(ptr == NULL)) - return -1; +```c +if (BUG(ptr == NULL)) + return -1; +``` ## Allocator conventions @@ -451,33 +470,39 @@ Also, a type named `abc_t` should be freed by a function named `abc_free_()`. Don't call this `abc_free_()` function directly -- instead, wrap it in a macro called `abc_free()`, using the `FREE_AND_NULL` macro: - void abc_free_(abc_t *obj); - #define abc_free(obj) FREE_AND_NULL(abc_t, abc_free_, (obj)) +```c +void abc_free_(abc_t *obj); +#define abc_free(obj) FREE_AND_NULL(abc_t, abc_free_, (obj)) +``` This macro will free the underlying `abc_t` object, and will also set the object pointer to NULL. You should define all `abc_free_()` functions to accept NULL inputs: - void - abc_free_(abc_t *obj) - { - if (!obj) - return; - tor_free(obj->name); - thing_free(obj->thing); - tor_free(obj); - } +```c +void +abc_free_(abc_t *obj) +{ + if (!obj) + return; + tor_free(obj->name); + thing_free(obj->thing); + tor_free(obj); +} +``` If you need a free function that takes a `void *` argument (for example, to use it as a function callback), define it with a name like `abc_free_void()`: - static void - abc_free_void_(void *obj) - { - abc_free_(obj); - } +```c +static void +abc_free_void_(void *obj) +{ + abc_free_(obj); +} +``` When deallocating, don't say e.g. `if (x) tor_free(x)`. The convention is to have deallocators do nothing when NULL pointer is passed. @@ -488,24 +513,28 @@ Say what functions do as a series of one or more imperative sentences, as though you were telling somebody how to be the function. In other words, DO NOT say: - /** The strtol function parses a number. - * - * nptr -- the string to parse. It can include whitespace. - * endptr -- a string pointer to hold the first thing that is not part - * of the number, if present. - * base -- the numeric base. - * returns: the resulting number. - */ - long strtol(const char *nptr, char **nptr, int base); +```c +/** The strtol function parses a number. + * + * nptr -- the string to parse. It can include whitespace. + * endptr -- a string pointer to hold the first thing that is not part + * of the number, if present. + * base -- the numeric base. + * returns: the resulting number. + */ +long strtol(const char *nptr, char **nptr, int base); +``` Instead, please DO say: - /** Parse a number in radix <b>base</b> from the string <b>nptr</b>, - * and return the result. Skip all leading whitespace. If - * <b>endptr</b> is not NULL, set *<b>endptr</b> to the first character - * after the number parsed. - **/ - long strtol(const char *nptr, char **nptr, int base); +```c +/** Parse a number in radix <b>base</b> from the string <b>nptr</b>, + * and return the result. Skip all leading whitespace. If + * <b>endptr</b> is not NULL, set *<b>endptr</b> to the first character + * after the number parsed. + **/ +long strtol(const char *nptr, char **nptr, int base); +``` Doxygen comments are the contract in our abstraction-by-contract world: if the functions that call your function rely on it doing something, then your diff --git a/doc/HACKING/CodingStandardsRust.md b/doc/HACKING/CodingStandardsRust.md index 36a0dcda2a..c821465173 100644 --- a/doc/HACKING/CodingStandardsRust.md +++ b/doc/HACKING/CodingStandardsRust.md @@ -22,20 +22,26 @@ For example, in a hypothetical `tor_addition` Rust module: In `src/rust/tor_addition/addition.rs`: - pub fn get_sum(a: i32, b: i32) -> i32 { - a + b - } +```rust +pub fn get_sum(a: i32, b: i32) -> i32 { + a + b +} +``` In `src/rust/tor_addition/lib.rs`: - pub use addition::*; +```rust +pub use addition::*; +``` In `src/rust/tor_addition/ffi.rs`: - #[no_mangle] - pub extern "C" fn tor_get_sum(a: c_int, b: c_int) -> c_int { - get_sum(a, b) - } +```rust +#[no_mangle] +pub extern "C" fn tor_get_sum(a: c_int, b: c_int) -> c_int { + get_sum(a, b) +} +``` If your Rust code must call out to parts of Tor's C code, you must declare the functions you are calling in the `external` crate, located @@ -129,16 +135,18 @@ crate. Unittests SHOULD go into their own module inside the module they are testing, e.g. in `src/rust/tor_addition/addition.rs` you should put: - #[cfg(test)] - mod test { - use super::*; +```rust +#[cfg(test)] +mod test { + use super::*; - #[test] - fn addition_with_zero() { - let sum: i32 = get_sum(5i32, 0i32); - assert_eq!(sum, 5); - } +#[test] + fn addition_with_zero() { + let sum: i32 = get_sum(5i32, 0i32); + assert_eq!(sum, 5); } +} +``` ## Benchmarking @@ -151,13 +159,17 @@ benchmarks in the following manner. If you wish to benchmark some of your Rust code, you MUST put the following in the `[features]` section of your crate's `Cargo.toml`: - [features] - bench = [] +```toml +[features] +bench = [] +``` Next, in your crate's `lib.rs` you MUST put: - #[cfg(all(test, feature = "bench"))] - extern crate test; +```rust +#[cfg(all(test, feature = "bench"))] +extern crate test; +``` This ensures that the external crate `test`, which contains utilities for basic benchmarks, is only used when running benchmarks via `cargo @@ -166,16 +178,18 @@ bench --features bench`. Finally, to write your benchmark code, in `src/rust/tor_addition/addition.rs` you SHOULD put: - #[cfg(all(test, features = "bench"))] - mod bench { - use test::Bencher; - use super::*; +```rust +#[cfg(all(test, features = "bench"))] +mod bench { + use test::Bencher; + use super::*; - #[bench] - fn addition_small_integers(b: &mut Bencher) { - b.iter(| | get_sum(5i32, 0i32)); - } +#[bench] + fn addition_small_integers(b: &mut Bencher) { + b.iter(| | get_sum(5i32, 0i32)); } +} +``` ## Fuzzing @@ -247,39 +261,47 @@ Here are some additional bits of advice and rules: potential error with the eel operator, `?` or another non panicking way. For example, consider a function which parses a string into an integer: - fn parse_port_number(config_string: &str) -> u16 { - u16::from_str_radix(config_string, 10).unwrap() - } + ```rust + fn parse_port_number(config_string: &str) -> u16 { + u16::from_str_radix(config_string, 10).unwrap() + } + ``` There are numerous ways this can fail, and the `unwrap()` will cause the whole program to byte the dust! Instead, either you SHOULD use `ok()` (or another equivalent function which will return an `Option` or a `Result`) and change the return type to be compatible: - fn parse_port_number(config_string: &str) -> Option<u16> { - u16::from_str_radix(config_string, 10).ok() - } + ```rust + fn parse_port_number(config_string: &str) -> Option<u16> { + u16::from_str_radix(config_string, 10).ok() + } + ``` or you SHOULD use `or()` (or another similar method): - fn parse_port_number(config_string: &str) -> Option<u16> { - u16::from_str_radix(config_string, 10).or(Err("Couldn't parse port into a u16") - } + ```rust + fn parse_port_number(config_string: &str) -> Option<u16> { + u16::from_str_radix(config_string, 10).or(Err("Couldn't parse port into a u16") + } + ``` Using methods like `or()` can be particularly handy when you must do something afterwards with the data, for example, if we wanted to guarantee that the port is high. Combining these methods with the eel operator (`?`) makes this even easier: - fn parse_port_number(config_string: &str) -> Result<u16, Err> { - let port = u16::from_str_radix(config_string, 10).or(Err("Couldn't parse port into a u16"))?; + ```rust + fn parse_port_number(config_string: &str) -> Result<u16, Err> { + let port = u16::from_str_radix(config_string, 10).or(Err("Couldn't parse port into a u16"))?; - if port > 1024 { - return Ok(port); - } else { - return Err("Low ports not allowed"); - } + if port > 1024 { + return Ok(port); + } else { + return Err("Low ports not allowed"); } + } + ``` 2. `unsafe` @@ -292,25 +314,29 @@ Here are some additional bits of advice and rules: When creating an FFI in Rust for C code to call, it is NOT REQUIRED to declare the entire function `unsafe`. For example, rather than doing: - #[no_mangle] - pub unsafe extern "C" fn increment_and_combine_numbers(mut numbers: [u8; 4]) -> u32 { - for number in &mut numbers { - *number += 1; - } - std::mem::transmute::<[u8; 4], u32>(numbers) + ```rust + #[no_mangle] + pub unsafe extern "C" fn increment_and_combine_numbers(mut numbers: [u8; 4]) -> u32 { + for number in &mut numbers { + *number += 1; } + std::mem::transmute::<[u8; 4], u32>(numbers) + } + ``` You SHOULD instead do: - #[no_mangle] - pub extern "C" fn increment_and_combine_numbers(mut numbers: [u8; 4]) -> u32 { - for index in 0..numbers.len() { - numbers[index] += 1; - } - unsafe { - std::mem::transmute::<[u8; 4], u32>(numbers) - } + ```rust + #[no_mangle] + pub extern "C" fn increment_and_combine_numbers(mut numbers: [u8; 4]) -> u32 { + for index in 0..numbers.len() { + numbers[index] += 1; } + unsafe { + std::mem::transmute::<[u8; 4], u32>(numbers) + } + } + ``` 3. Pass only C-compatible primitive types and bytes over the boundary @@ -385,45 +411,51 @@ Here are some additional bits of advice and rules: rather than using an untyped mapping between strings and integers like so: - use std::collections::HashMap; + ```rust + use std::collections::HashMap; - pub fn get_elements_with_over_9000_points(map: &HashMap<String, usize>) -> Vec<String> { - ... - } + pub fn get_elements_with_over_9000_points(map: &HashMap<String, usize>) -> Vec<String> { + ... + } + ``` It would be safer to define a new type, such that some other usage of `HashMap<String, usize>` cannot be confused for this type: - pub struct DragonBallZPowers(pub HashMap<String, usize>); + ```rust + pub struct DragonBallZPowers(pub HashMap<String, usize>); - impl DragonBallZPowers { - pub fn over_nine_thousand<'a>(&'a self) -> Vec<&'a String> { - let mut powerful_enough: Vec<&'a String> = Vec::with_capacity(5); + impl DragonBallZPowers { + pub fn over_nine_thousand<'a>(&'a self) -> Vec<&'a String> { + let mut powerful_enough: Vec<&'a String> = Vec::with_capacity(5); - for (character, power) in &self.0 { - if *power > 9000 { - powerful_enough.push(character); - } - } - powerful_enough - } - } + for (character, power) in &self.0 { + if *power > 9000 { + powerful_enough.push(character); + } + } + powerful_enough + } + } + ``` Note the following code, which uses Rust's type aliasing, is valid but it does NOT meet the desired type safety goals: - pub type Power = usize; + ```rust + pub type Power = usize; - pub fn over_nine_thousand(power: &Power) -> bool { - if *power > 9000 { - return true; - } - false + pub fn over_nine_thousand(power: &Power) -> bool { + if *power > 9000 { + return true; } + false + } - // We can still do the following: - let his_power: usize = 9001; - over_nine_thousand(&his_power); + // We can still do the following: + let his_power: usize = 9001; + over_nine_thousand(&his_power); + ``` 7. Unsafe mucking around with lifetimes @@ -431,15 +463,17 @@ Here are some additional bits of advice and rules: family of types, individual lifetimes can be treated as types. For example, one can arbitrarily extend and shorten lifetime using `std::mem::transmute`: - struct R<'a>(&'a i32); + ```rust + struct R<'a>(&'a i32); - unsafe fn extend_lifetime<'b>(r: R<'b>) -> R<'static> { - std::mem::transmute::<R<'b>, R<'static>>(r) - } + unsafe fn extend_lifetime<'b>(r: R<'b>) -> R<'static> { + std::mem::transmute::<R<'b>, R<'static>>(r) + } - unsafe fn shorten_invariant_lifetime<'b, 'c>(r: &'b mut R<'static>) -> &'b mut R<'c> { - std::mem::transmute::<&'b mut R<'static>, &'b mut R<'c>>(r) - } + unsafe fn shorten_invariant_lifetime<'b, 'c>(r: &'b mut R<'static>) -> &'b mut R<'c> { + std::mem::transmute::<&'b mut R<'static>, &'b mut R<'c>>(r) + } + ``` Calling `extend_lifetime()` would cause an `R` passed into it to live forever for the life of the program (the `'static` lifetime). Similarly, @@ -460,12 +494,14 @@ Here are some additional bits of advice and rules: For example, `std::mem::transmute` can be abused in ways where casting with `as` would be both simpler and safer: - // Don't do this - let ptr = &0; - let ptr_num_transmute = unsafe { std::mem::transmute::<&i32, usize>(ptr)}; + ```rust + // Don't do this + let ptr = &0; + let ptr_num_transmute = unsafe { std::mem::transmute::<&i32, usize>(ptr)}; - // Use an `as` cast instead - let ptr_num_cast = ptr as *const i32 as usize; + // Use an `as` cast instead + let ptr_num_cast = ptr as *const i32 as usize; + ``` In fact, using `std::mem::transmute` for *any* reason is a code smell and as such SHOULD be avoided. @@ -475,8 +511,10 @@ Here are some additional bits of advice and rules: This is generally fine to do, but it has some behaviours which you should be aware of. Casting down chops off the high bits, e.g.: - let x: u32 = 4294967295; - println!("{}", x as u16); // prints 65535 + ```rust + let x: u32 = 4294967295; + println!("{}", x as u16); // prints 65535 + ``` Some cases which you MUST NOT do include: @@ -487,24 +525,28 @@ Here are some additional bits of advice and rules: * Casting between integers and floats when the thing being cast cannot fit into the type it is being casted into, e.g.: - println!("{}", 42949.0f32 as u8); // prints 197 in debug mode and 0 in release - println!("{}", 1.04E+17 as u8); // prints 0 in both modes - println!("{}", (0.0/0.0) as i64); // prints whatever the heck LLVM wants + ```rust + println!("{}", 42949.0f32 as u8); // prints 197 in debug mode and 0 in release + println!("{}", 1.04E+17 as u8); // prints 0 in both modes + println!("{}", (0.0/0.0) as i64); // prints whatever the heck LLVM wants + ``` Because this behaviour is undefined, it can even produce segfaults in safe Rust code. For example, the following program built in release mode segfaults: - #[inline(never)] - pub fn trigger_ub(sl: &[u8; 666]) -> &[u8] { - // Note that the float is out of the range of `usize`, invoking UB when casting. - let idx = 1e99999f64 as usize; - &sl[idx..] // The bound check is elided due to `idx` being of an undefined value. - } - - fn main() { - println!("{}", trigger_ub(&[1; 666])[999999]); // ~ out of bound - } + ```rust + #[inline(never)] + pub fn trigger_ub(sl: &[u8; 666]) -> &[u8] { + // Note that the float is out of the range of `usize`, invoking UB when casting. + let idx = 1e99999f64 as usize; + &sl[idx..] // The bound check is elided due to `idx` being of an undefined value. + } + + fn main() { + println!("{}", trigger_ub(&[1; 666])[999999]); // ~ out of bound + } + ``` And in debug mode panics with: diff --git a/doc/HACKING/Fuzzing.md b/doc/HACKING/Fuzzing.md index 487716bb6d..1a9185aebf 100644 --- a/doc/HACKING/Fuzzing.md +++ b/doc/HACKING/Fuzzing.md @@ -6,7 +6,10 @@ Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place where you checked it out. To run the fuzzing test cases in a deterministic fashion, use: - make test-fuzz-corpora + +```console +$ make test-fuzz-corpora +``` This won't actually fuzz Tor! It will just run all the fuzz binaries on our existing set of testcases for the fuzzer. @@ -58,11 +61,13 @@ machine you care about, anyway. To Build: Get AFL from http://lcamtuf.coredump.cx/afl/ and unpack it - cd afl - make - cd ../tor - PATH=$PATH:../afl/ CC="../afl/afl-gcc" ./configure --enable-expensive-hardening - AFL_HARDEN=1 make clean fuzzers + ```console + $ cd afl + $ make + $ cd ../tor + $ PATH=$PATH:../afl/ CC="../afl/afl-gcc" ./configure --enable-expensive-hardening + $ AFL_HARDEN=1 make clean fuzzers + ``` To Find The ASAN Memory Limit: (64-bit only) @@ -75,10 +80,12 @@ Read afl/docs/notes_for_asan.txt for more details. Download recidivm from https://jwilk.net/software/recidivm Download the signature Check the signature - tar xvzf recidivm*.tar.gz - cd recidivm* - make - /path/to/recidivm -v src/test/fuzz/fuzz-http + ```console + $ tar xvzf recidivm*.tar.gz + $ cd recidivm* + $ make + $ /path/to/recidivm -v src/test/fuzz/fuzz-http + ``` Use the final "ok" figure as the input to -m when calling afl-fuzz (Normally, recidivm would output a figure automatically, but in some cases, the fuzzing harness will hang when the memory limit is too small.) @@ -88,9 +95,11 @@ don't care about memory limits. To Run: - mkdir -p src/test/fuzz/fuzz_http_findings - ../afl/afl-fuzz -i ${TOR_FUZZ_CORPORA}/http -o src/test/fuzz/fuzz_http_findings -m <asan-memory-limit> -- src/test/fuzz/fuzz-http +```console +$ mkdir -p src/test/fuzz/fuzz_http_findings +$ ../afl/afl-fuzz -i ${TOR_FUZZ_CORPORA}/http -o src/test/fuzz/fuzz_http_findings -m <asan-memory-limit> -- src/test/fuzz/fuzz-http +``` AFL has a multi-core mode, check the documentation for details. You might find the included fuzz-multi.sh script useful for this. @@ -109,7 +118,10 @@ valid inputs may take a second or so, particularly with the fuzzer and sanitizers enabled. To see what fuzz-http is doing with a test case, call it like this: - src/test/fuzz/fuzz-http --debug < /path/to/test.case + +```console +$ src/test/fuzz/fuzz-http --debug < /path/to/test.case +``` (Logging is disabled while fuzzing to increase fuzzing speed.) @@ -118,4 +130,4 @@ To see what fuzz-http is doing with a test case, call it like this: Please report any issues discovered using the process in Tor's security issue policy: -https://trac.torproject.org/projects/tor/wiki/org/meetings/2016SummerDevMeeting/Notes/SecurityIssuePolicy +https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/SecurityPolicy diff --git a/doc/HACKING/GettingStarted.md b/doc/HACKING/GettingStarted.md index 633a7f0417..6d61be9881 100644 --- a/doc/HACKING/GettingStarted.md +++ b/doc/HACKING/GettingStarted.md @@ -37,13 +37,17 @@ Once you've reached this point, here's what you need to know. We keep our source under version control in Git. To get the latest version, run: - git clone https://git.torproject.org/git/tor + ```console + $ git clone https://git.torproject.org/git/tor + ``` This will give you a checkout of the master branch. If you're going to fix a bug that appears in a stable version, check out the appropriate "maint" branch, as in: - git checkout maint-0.4.3 + ```console + $ git checkout maint-0.4.3 + ``` 2. Find your way around the source. @@ -67,10 +71,11 @@ Once you've reached this point, here's what you need to know. Many people have gotten started by looking for an area where they personally felt Tor was underperforming, and investigating ways to fix it. If you're looking for ideas, you can head to - [trac](https://trac.torproject.org) our bug tracking tool and look for - tickets that have received the "easy" tag: these are ones that developers + [gitlab](https://gitlab.torproject.org) our bug tracking tool and look for + tickets that have received the "First Contribution" label: these are ones + that developers think would be pretty simple for a new person to work on. For a bigger - challenge, you might want to look for tickets with the "lorax" + challenge, you might want to look for tickets with the "Project Ideas" keyword: these are tickets that the developers think might be a good idea to build, but which we have no time to work on any time soon. @@ -136,10 +141,10 @@ Once you've reached this point, here's what you need to know. 8. Submitting your patch We review patches through tickets on our bugtracker at - [trac](https://trac.torproject.org). You can either upload your patches there, or + [gitlab](https://gitlab.torproject.org). You can either upload your patches there, or put them at a public git repository somewhere we can fetch them (like gitlab, github or bitbucket) and then paste a link on the appropriate - trac ticket. + ticket. Once your patches are available, write a short explanation of what you've done on trac, and then change the status of the ticket to diff --git a/doc/HACKING/GettingStartedRust.md b/doc/HACKING/GettingStartedRust.md index af80018f4e..adacf8afc2 100644 --- a/doc/HACKING/GettingStartedRust.md +++ b/doc/HACKING/GettingStartedRust.md @@ -54,7 +54,9 @@ fetching dependencies from Cargo or specifying a local directory. **Fetch dependencies from Cargo** - ./configure --enable-rust --enable-cargo-online-mode +```console +$ ./configure --enable-rust --enable-cargo-online-mode +``` **Using a local dependency cache** @@ -66,13 +68,17 @@ We vendor our Rust dependencies in a separate repo using [cargo-vendor](https://github.com/alexcrichton/cargo-vendor). To use them, do: - git submodule init - git submodule update +```console +$ git submodule init +$ git submodule update +``` To specify the local directory containing the dependencies, (assuming you are in the top level of the repository) configure tor with: - TOR_RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust +```console +$ TOR_RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust +``` (Note that `TOR_RUST_DEPENDENCIES` must be the full path to the directory; it cannot be relative.) @@ -80,7 +86,9 @@ cannot be relative.) Assuming you used the above `git submodule` commands and you're in the topmost directory of the repository, this would be: - TOR_RUST_DEPENDENCIES=`pwd`/src/ext/rust/crates ./configure --enable-rust +```console +$ TOR_RUST_DEPENDENCIES=`pwd`/src/ext/rust/crates ./configure --enable-rust +``` ## Identifying which modules to rewrite @@ -102,10 +110,12 @@ areas of responsibility. A good first step is to build a module-level callgraph to understand how interconnected your target module is. - git clone https://git.torproject.org/user/nickm/calltool.git - cd tor - CFLAGS=0 ./configure - ../calltool/src/main.py module_callgraph +```console +$ git clone https://git.torproject.org/user/nickm/calltool.git +$ cd tor +$ CFLAGS=0 ./configure +$ ../calltool/src/main.py module_callgraph +``` The output will tell you each module name, along with a set of every module that the module calls. Modules which call fewer other modules are better targets. @@ -156,15 +166,21 @@ run on your crate. Configure Tor's build system to build with Rust enabled: - ./configure --enable-fatal-warnings --enable-rust --enable-cargo-online-mode +```console +$ ./configure --enable-fatal-warnings --enable-rust --enable-cargo-online-mode +``` Tor's test should be run by doing: - make check +```console +$ make check +``` Tor's integration tests should also pass: - make test-stem +```console +$ make test-stem +``` ## Submitting a patch diff --git a/doc/HACKING/HelpfulTools.md b/doc/HACKING/HelpfulTools.md index 15bd153318..0ce59576f0 100644 --- a/doc/HACKING/HelpfulTools.md +++ b/doc/HACKING/HelpfulTools.md @@ -43,7 +43,9 @@ Builds should show up on the web at jenkins.torproject.org and on IRC at ## Valgrind - valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/app/tor +```console +$ valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/app/tor +``` (Note that if you get a zillion openssl warnings, you will also need to pass `--undef-value-errors=no` to valgrind, or rebuild your openssl @@ -77,10 +79,12 @@ we wish to permit are also documented in the blacklist file. Lcov is a utility that generates pretty HTML reports of test code coverage. To generate such a report: - ./configure --enable-coverage - make - make coverage-html - $BROWSER ./coverage_html/index.html +```console +$ ./configure --enable-coverage +$ make +$ make coverage-html +$ $BROWSER ./coverage_html/index.html +``` This will run the tor unit test suite `./src/test/test` and generate the HTML coverage code report under the directory `./coverage_html/`. To change the @@ -93,36 +97,48 @@ investigated (as of July 2014). To quickly run all the tests distributed with Tor: - make check +```console +$ make check +``` To run the fast unit tests only: - make test +```console +$ make test +``` To selectively run just some tests (the following can be combined arbitrarily): - ./src/test/test <name_of_test> [<name of test 2>] ... - ./src/test/test <prefix_of_name_of_test>.. [<prefix_of_name_of_test2>..] ... - ./src/test/test :<name_of_excluded_test> [:<name_of_excluded_test2]... +```console +$ ./src/test/test <name_of_test> [<name of test 2>] ... +$ ./src/test/test <prefix_of_name_of_test>.. [<prefix_of_name_of_test2>..] ... +$ ./src/test/test :<name_of_excluded_test> [:<name_of_excluded_test2]... +``` To run all tests, including those based on Stem or Chutney: - make test-full +```console +$ make test-full +``` To run all tests, including those based on Stem or Chutney that require a working connection to the internet: - make test-full-online +```console +$ make test-full-online +``` ## Running gcov for unit test coverage - ./configure --enable-coverage - make - make check - # or--- make test-full ? make test-full-online? - mkdir coverage-output - ./scripts/test/coverage coverage-output +```console +$ ./configure --enable-coverage +$ make +$ make check +$ # or--- make test-full ? make test-full-online? +$ mkdir coverage-output +$ ./scripts/test/coverage coverage-output +``` (On OSX, you'll need to start with `--enable-coverage CC=clang`.) @@ -145,7 +161,9 @@ you can run `make reset-gcov` to clear the intermediary gcov output. If you have two different `coverage-output` directories, and you want to see a meaningful diff between them, you can run: - ./scripts/test/cov-diff coverage-output1 coverage-output2 | less +```console +$ ./scripts/test/cov-diff coverage-output1 coverage-output2 | less +``` In this diff, any lines that were visited at least once will have coverage "1", and line numbers are deleted. This lets you inspect what you (probably) really @@ -313,12 +331,16 @@ that you're using the emacs-specific version of `etags` (bundled under the If you're using vim or emacs, you can also use Universal Ctags to build a tag file using the syntax: - ctags -R -D 'MOCK_IMPL(r,h,a)=r h a' . +```console +$ ctags -R -D 'MOCK_IMPL(r,h,a)=r h a' . +``` If you're using an older version of Universal Ctags, you can use the following instead: - ctags -R --mline-regex-c='/MOCK_IMPL\([^,]+,\W*([a-zA-Z0-9_]+)\W*,/\1/f/{mgroup=1}' . +```console +ctags -R --mline-regex-c='/MOCK_IMPL\([^,]+,\W*([a-zA-Z0-9_]+)\W*,/\1/f/{mgroup=1}' . +``` A vim-compatible tag file will be generated by default. If you use emacs, add the `-e` flag to generate an emacs-compatible tag file. @@ -330,50 +352,58 @@ source code. Here's how to use it: 1. Begin every file that should be documented with - /** - * \file filename.c - * \brief Short description of the file. - */ +``` + /** + * \file filename.c + * \brief Short description of the file. + */ +``` - (Doxygen will recognize any comment beginning with /** as special.) + (Doxygen will recognize any comment beginning with /** as special.) 2. Before any function, structure, #define, or variable you want to document, add a comment of the form: - /** Describe the function's actions in imperative sentences. - * - * Use blank lines for paragraph breaks - * - and - * - hyphens - * - for - * - lists. - * - * Write <b>argument_names</b> in boldface. - * - * \code - * place_example_code(); - * between_code_and_endcode_commands(); - * \endcode - */ +``` +/** Describe the function's actions in imperative sentences. + * + * Use blank lines for paragraph breaks + * - and + * - hyphens + * - for + * - lists. + * + * Write <b>argument_names</b> in boldface. + * + * \code + * place_example_code(); + * between_code_and_endcode_commands(); + * \endcode + */ +``` 3. Make sure to escape the characters `<`, `>`, `\`, `%` and `#` as `\<`, `\>`, `\\`, `\%` and `\#`. 4. To document structure members, you can use two forms: - struct foo { - /** You can put the comment before an element; */ - int a; - int b; /**< Or use the less-than symbol to put the comment - * after the element. */ - }; +```c +struct foo { + /** You can put the comment before an element; */ + int a; + int b; /**< Or use the less-than symbol to put the comment + * after the element. */ +}; +``` 5. To generate documentation from the Tor source code, type: - $ doxygen -g +```console +$ doxygen -g +``` - to generate a file called `Doxyfile`. Edit that file and run - `doxygen` to generate the API documentation. + to generate a file called `Doxyfile`. Edit that file and run + `doxygen` to generate the API documentation. 6. See the Doxygen manual for more information; this summary just scratches the surface. diff --git a/doc/HACKING/Module.md b/doc/HACKING/Module.md index f8a9773d47..b9d3a654eb 100644 --- a/doc/HACKING/Module.md +++ b/doc/HACKING/Module.md @@ -70,7 +70,7 @@ There are couples of "rules" you want to follow: base. Every entry point should have a second definition if the module is disabled. For instance: - ``` + ```c #ifdef HAVE_MODULE_DIRAUTH int sr_init(int save_to_disk); @@ -109,7 +109,9 @@ There are couples of "rules" you want to follow: * When you include headers from the module, **always** use the full module path in your statement. Example: - `#include "feature/dirauth/dirvote.h"` +```c +#include "feature/dirauth/dirvote.h"` +``` The main reason is that we do **not** add the module include path by default so it needs to be specified. But also, it helps our human brain understand diff --git a/doc/HACKING/README.1st.md b/doc/HACKING/README.1st.md index 2278a61d6c..4bc3298c67 100644 --- a/doc/HACKING/README.1st.md +++ b/doc/HACKING/README.1st.md @@ -32,7 +32,9 @@ For an explanation of how to change Tor's design to work differently, look at For the latest version of the code, get a copy of git, and - git clone https://git.torproject.org/git/tor +```console +$ git clone https://git.torproject.org/git/tor +``` ## Stay in touch diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index 2464d8afb4..24b66a069a 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -9,8 +9,20 @@ new Tor release: version number in their approved versions list. Give them a few days to do this if you can. -2. If this is going to be an important security release, give the packagers - some advance warning: See this list of packagers in IV.3 below. +2. If this is going to be an important security release, give these packagers + some advance warning: + + - {weasel,sysrqb,mikeperry} at torproject dot org + - {blueness} at gentoo dot org + - {paul} at invizbox dot io + - {vincent} at invizbox dot com + - {lfleischer} at archlinux dot org + - {Nathan} at freitas dot net + - {mike} at tig dot as + - {tails-rm} at boum dot org + - {simon} at sdeziel.info + - {yuri} at freebsd.org + - {mh+tor} at scrit.ch 3. Given the release date for Tor, ask the TB team about the likely release date of a TB that contains it. See note below in "commit, upload, @@ -36,19 +48,6 @@ new Tor release: * On OSS-Fuzz -3. Run checks that aren't covered above, including: - - * clang scan-build. (See the script in ./scripts/test/scan_build.sh) - - * make test-network and make test-network-all (with - --enable-fragile-hardening) - - * Running Tor yourself and making sure that it actually works for you. - - * Running Tor under valgrind. (Our 'fragile hardening' doesn't cover - libevent and openssl, so using valgrind will sometimes find extra - memory leaks.) - ## II. Write a changelog 1a. (Alpha release variant) @@ -57,11 +56,12 @@ new Tor release: of them and reordering to focus on what users and funders would find interesting and understandable. - To do this, run - `./scripts/maint/sortChanges.py changes/* > changelog.in` - to combine headings and sort the entries. Copy the changelog.in file - into the ChangeLog. Run 'format_changelog.py' (see below) to clean - up the line breaks. + To do this, run `./scripts/maint/sortChanges.py changes/* > changelog.in` + to combine headings and sort the entries. Copy the changelog.in file into + the ChangeLog. Run `format_changelog.py --inplace` (see below) to clean up + the line breaks. + + Remove the `changes/*` files that you just merged into the ChangeLog. After that, it's time to hand-edit and fix the issues that lintChanges can't find: @@ -141,10 +141,11 @@ new Tor release: places, and commit. Then merge `maint-0.?.x` into `release-0.?.x`. When you merge the maint branch forward to the next maint branch, or into - master, merge it with "-s ours" to avoid a needless version bump. + master, merge it with "-s ours" to avoid conflict with the version + bump. 2. Make distcheck, put the tarball up in somewhere (how about your - homedir on your homedir on people.torproject.org?) , and tell `#tor-dev` + homedir on people.torproject.org?) , and tell `#tor-dev` about it. If you want, wait until at least one person has built it @@ -152,7 +153,6 @@ new Tor release: CI has successfully caught these kinds of errors for the last several years.) - 3. Make sure that the new version is recommended in the latest consensus. (Otherwise, users will get confused when it complains to them about its status.) @@ -164,9 +164,11 @@ new Tor release: 1. Sign the tarball, then sign and push the git tag: - gpg -ba <the_tarball> - git tag -s tor-0.4.x.y-<status> - git push origin tag tor-0.4.x.y-<status> +```console +$ gpg -ba <the_tarball> +$ git tag -s tor-0.4.x.y-<status> +$ git push origin tag tor-0.4.x.y-<status> +``` (You must do this before you update the website: the website scripts rely on finding the version by tag.) @@ -178,7 +180,6 @@ new Tor release: `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run "static-update-component dist.torproject.org" on dist-master. - In the webwml.git repository, `include/versions.wmi` and `Makefile`. In the project/web/tpo.git repository, update `databags/versions.ini` to note the new version. Push these changes to master. @@ -189,20 +190,8 @@ new Tor release: (NOTE: It will take a while for the website update scripts to update the website.) -3. Email the packagers (cc'ing tor-team) that a new tarball is up. - The current list of packagers is: - - - {weasel,sysrqb,mikeperry} at torproject dot org - - {blueness} at gentoo dot org - - {paul} at invizbox dot io - - {vincent} at invizbox dot com - - {lfleischer} at archlinux dot org - - {Nathan} at freitas dot net - - {mike} at tig dot as - - {tails-rm} at boum dot org - - {simon} at sdeziel.info - - {yuri} at freebsd.org - - {mh+tor} at scrit.ch +3. Email the tor-packagers@lists.torproject.org mailing list to tell them + about the new release. Also, email tor-packagers@lists.torproject.org. @@ -210,22 +199,15 @@ new Tor release: Include a link to the changelog. -4. Add the version number to Trac. To do this, go to Trac, log in, - select "Admin" near the top of the screen, then select "Versions" from - the menu on the left. At the right, there will be an "Add version" - box. By convention, we enter the version in the form "Tor: - 0.4.0.1-alpha" (or whatever the version is), and we select the date as - the date in the ChangeLog. - -5. Wait for the download page to be updated. (If you don't do this before you +4. Wait for the download page to be updated. (If you don't do this before you announce, people will be confused.) -6. Mail the release blurb and ChangeLog to tor-talk (development release) or +5. Mail the release blurb and ChangeLog to tor-talk (development release) or tor-announce (stable). Post the changelog on the blog as well. You can generate a blog-formatted version of the changelog with - `./scripts/maint/format_changelog.py --B` + `./scripts/maint/format_changelog.py -B` When you post, include an estimate of when the next TorBrowser releases will come out that include this Tor release. This will @@ -233,13 +215,13 @@ new Tor release: can vary. For templates to use when announcing, see: - https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates + https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/AnnouncementTemplates ## V. Aftermath and cleanup 1. If it's a stable release, bump the version number in the - `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours` - merge to avoid taking that change into master. + `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours` + merge to avoid taking that change into master. 2. If there is a new `maint-x.y.z` branch, create a Travis CI cron job that builds the release every week. (It's ok to skip the weekly build if the diff --git a/doc/HACKING/Tracing.md b/doc/HACKING/Tracing.md deleted file mode 100644 index e1e97abe6d..0000000000 --- a/doc/HACKING/Tracing.md +++ /dev/null @@ -1,91 +0,0 @@ -# Tracing - -This document describes how the event tracing subsystem works in tor so -developers can add events to the code base but also hook them to an event -tracing framework. - -## Basics - -Event tracing is separated in two concepts, trace events and a tracer. The -tracing subsystem can be found in `src/trace`. The `events.h` header file is -the main file that maps the different tracers to trace events. - -### Events - -A trace event is basically a function from which we can pass any data that -we want to collect. In addition, we specify a context for the event such as -a subsystem and an event name. - -A trace event in tor has the following standard format: - - tor_trace(subsystem, event\_name, args...) - -The `subsystem` parameter is the name of the subsytem the trace event is in. -For example that could be "scheduler" or "vote" or "hs". The idea is to add -some context to the event so when we collect them we know where it's coming -from. The `event_name` is the name of the event which helps a lot with -adding some semantic to the event. Finally, `args` is any number of -arguments we want to collect. - -Here is an example of a possible tracepoint in main(): - - tor_trace(main, init_phase, argc) - -The above is a tracepoint in the `main` subsystem with `init_phase` as the -event name and the `int argc` is passed to the event as well. - -How `argc` is collected or used has nothing to do with the instrumentation -(adding trace events to the code). It is the work of the tracer so this is why -the trace events and collection framework (tracer) are decoupled. You _can_ -have trace events without a tracer. - -### Tracer - -In `src/trace/events.h`, we map the `tor_trace()` function to the right -tracer. A tracer support is only enabled at compile time. For instance, the -file `src/trace/debug.h` contains the mapping of the generic tracing function -`tor_trace()` to the `log_debug()` function. More specialized function can be -mapped depending on the tracepoint. - -## Build System - -This section describes how it is integrated into the build system of tor. - -By default, every tracing events are disabled in tor that is `tor_trace()` -is a NOP. - -To enable a tracer, there is a configure option on the form of: - - --enable-tracing-<tracer> - -We have an option that will send every trace events to a `log_debug()` (as -mentionned above) which will print you the subsystem and name of the event but -not the arguments for technical reasons. This is useful if you want to quickly -see if your trace event is being hit or well written. To do so, use this -configure option: - - --enable-tracing-debug - -## Instrument Tor - -This is pretty easy. Let's say you want to add a trace event in -`src/feature/rend/rendcache.c`, you only have to add this include statement: - - #include "trace/events.h" - -Once done, you can add as many as you want `tor_trace()` that you need. -Please use the right subsystem (here it would be `hs`) and a unique name that -tells what the event is for. For example: - - tor_trace(hs, store_desc_as_client, desc, desc_id); - -If you look in `src/trace/events.h`, you'll see that if tracing is enabled it -will be mapped to a function called: - - tor_trace_hs_store_desc_as_client(desc, desc_id) - -And the point of all this is for that function to be defined in a new file -that you might want to add named `src/trace/hs.{c|h}` which would defined how -to collect the data for the `tor_trace_hs_store_desc_as_client()` function -like for instance sending it to a `log_debug()` or do more complex operations -or use a userspace tracer like LTTng (https://lttng.org). diff --git a/doc/HACKING/WritingTests.md b/doc/HACKING/WritingTests.md index d212020525..e1497a77c2 100644 --- a/doc/HACKING/WritingTests.md +++ b/doc/HACKING/WritingTests.md @@ -107,7 +107,9 @@ covered or uncovered. To count new or modified uncovered lines in D2, you can run: - ./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' | wc -l +```console +$ ./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' | wc -l +``` ## Marking lines as unreachable by tests @@ -163,28 +165,30 @@ I use the term "unit test" and "regression tests" very sloppily here. Here's an example of a test function for a simple function in util.c: - static void - test_util_writepid(void *arg) - { - (void) arg; +```c +static void +test_util_writepid(void *arg) +{ + (void) arg; - char *contents = NULL; - const char *fname = get_fname("tmp_pid"); - unsigned long pid; - char c; + char *contents = NULL; + const char *fname = get_fname("tmp_pid"); + unsigned long pid; + char c; - write_pidfile(fname); + write_pidfile(fname); - contents = read_file_to_str(fname, 0, NULL); - tt_assert(contents); + contents = read_file_to_str(fname, 0, NULL); + tt_assert(contents); - int n = sscanf(contents, "%lu\n%c", &pid, &c); - tt_int_op(n, OP_EQ, 1); - tt_int_op(pid, OP_EQ, getpid()); + int n = sscanf(contents, "%lu\n%c", &pid, &c); + tt_int_op(n, OP_EQ, 1); + tt_int_op(pid, OP_EQ, getpid()); - done: - tor_free(contents); - } +done: + tor_free(contents); +} +``` This should look pretty familiar to you if you've read the tinytest manual. One thing to note here is that we use the testing-specific @@ -214,10 +218,12 @@ macro-protected declaration of the function in the module's header. For example, `crypto_curve25519.h` contains: - #ifdef CRYPTO_CURVE25519_PRIVATE - STATIC int curve25519_impl(uint8_t *output, const uint8_t *secret, - const uint8_t *basepoint); - #endif +```c +#ifdef CRYPTO_CURVE25519_PRIVATE +STATIC int curve25519_impl(uint8_t *output, const uint8_t *secret, + const uint8_t *basepoint); +#endif +``` The `crypto_curve25519.c` file and the `test_crypto.c` file both define `CRYPTO_CURVE25519_PRIVATE`, so they can see this declaration. @@ -231,28 +237,29 @@ the test _really tests_ the code. For example, here is a _bad_ test for the unlink() function (which is supposed to remove a file). - static void - test_unlink_badly(void *arg) - { - (void) arg; - int r; +```c +static void +test_unlink_badly(void *arg) +{ + (void) arg; + int r; - const char *fname = get_fname("tmpfile"); + const char *fname = get_fname("tmpfile"); - /* If the file isn't there, unlink returns -1 and sets ENOENT */ - r = unlink(fname); - tt_int_op(n, OP_EQ, -1); - tt_int_op(errno, OP_EQ, ENOENT); + /* If the file isn't there, unlink returns -1 and sets ENOENT */ + r = unlink(fname); + tt_int_op(n, OP_EQ, -1); + tt_int_op(errno, OP_EQ, ENOENT); - /* If the file DOES exist, unlink returns 0. */ - write_str_to_file(fname, "hello world", 0); - r = unlink(fnme); - tt_int_op(r, OP_EQ, 0); - - done: - tor_free(contents); - } + /* If the file DOES exist, unlink returns 0. */ + write_str_to_file(fname, "hello world", 0); + r = unlink(fnme); + tt_int_op(r, OP_EQ, 0); +done: + tor_free(contents); +} +``` This test might get very high coverage on unlink(). So why is it a bad test? Because it doesn't check that unlink() *actually removes the @@ -273,20 +280,25 @@ To write tests for this case, you can replace the underlying functions with testing stubs while your unit test is running. You need to declare the underlying function as 'mockable', as follows: - MOCK_DECL(returntype, functionname, (argument list)); +```c +MOCK_DECL(returntype, functionname, (argument list)); +``` and then later implement it as: - MOCK_IMPL(returntype, functionname, (argument list)) - { - /* implementation here */ - } +```c +MOCK_IMPL(returntype, functionname, (argument list)) +{ + /* implementation here */ +} +``` For example, if you had a 'connect to remote server' function, you could declare it as: - - MOCK_DECL(int, connect_to_remote, (const char *name, status_t *status)); +```c +MOCK_DECL(int, connect_to_remote, (const char *name, status_t *status)); +``` When you declare a function this way, it will be declared as normal in regular builds, but when the module is built for testing, it is declared @@ -295,11 +307,15 @@ as a function pointer initialized to the actual implementation. In your tests, if you want to override the function with a temporary replacement, you say: - MOCK(functionname, replacement_function_name); +```c +MOCK(functionname, replacement_function_name); +``` And later, you can restore the original function with: - UNMOCK(functionname); +```c +UNMOCK(functionname); +``` For more information, see the definitions of this mocking logic in `testsupport.h`. @@ -324,11 +340,13 @@ cases and failure csaes. For example, consider testing this function: - /** Remove all elements E from sl such that E==element. Preserve - * the order of any elements before E, but elements after E can be - * rearranged. - */ - void smartlist_remove(smartlist_t *sl, const void *element); +```c +/** Remove all elements E from sl such that E==element. Preserve + * the order of any elements before E, but elements after E can be + * rearranged. + */ +void smartlist_remove(smartlist_t *sl, const void *element); +``` In order to test it well, you should write tests for at least all of the following cases. (These would be black-box tests, since we're only looking @@ -355,19 +373,21 @@ When you consider edge cases, you might try: Now let's look at the implementation: - void - smartlist_remove(smartlist_t *sl, const void *element) - { - int i; - if (element == NULL) +```c +void +smartlist_remove(smartlist_t *sl, const void *element) +{ + int i; + if (element == NULL) return; - for (i=0; i < sl->num_used; i++) + for (i=0; i < sl->num_used; i++) if (sl->list[i] == element) { - sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */ - i--; /* so we process the new i'th element */ - sl->list[sl->num_used] = NULL; + sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */ + i--; /* so we process the new i'th element */ + sl->list[sl->num_used] = NULL; } - } +} +``` Based on the implementation, we now see three more edge cases to test: @@ -484,3 +504,15 @@ targets in `Makefile.am`. (Adding new kinds of program to chutney will still require hacking the code.) + +## Other integration tests + +It's fine to write tests that use a POSIX shell to invoke Tor or test other +aspects of the system. When you do this, have a look at our existing tests +of this kind in `src/test/` to make sure that you haven't forgotten anything +important. For example: it can be tricky to make sure you're invoking Tor at +the right path in various build scenarios. + +We use a POSIX shell whenever possible here, and we use the shellcheck tool +to make sure that our scripts portable. We should only require bash for +scripts that are developer-only. diff --git a/doc/HACKING/android/Simpleperf.md b/doc/HACKING/android/Simpleperf.md index c7e63a7c86..ed640f912e 100644 --- a/doc/HACKING/android/Simpleperf.md +++ b/doc/HACKING/android/Simpleperf.md @@ -29,7 +29,9 @@ the Android Software Development Kit (SDK) and Native Development Kit 3. Install the Android Package you generated in step 1: +```bash $ adb install /path/to/your/app-fullperm-debug.apk +``` 4. Check on your device that the newly installed Orbot actually works and behaves in the way you expect it to. @@ -76,10 +78,12 @@ was spend on the call. To access binaries, `torrc` files, and other useful information on the device do the following: +```console $ adb shell (device):/ $ run-as org.torproject.android (device):/data/data/org.torproject.android $ ls app_bin app_data cache databases files lib shared_prefs +``` Descriptors, control authentication cookie, state, and other files can be found in the `app_data` directory. The `torrc` can be found in the `app_bin/` @@ -88,10 +92,14 @@ was spend on the call. - You can enable logging in Tor via the syslog (or android) log mechanism with: +```console $ adb shell (device):/ $ run-as org.torproject.android (device):/data/data/org.torproject.android $ echo -e "\nLog info syslog" >> app_bin/torrc +``` Start Tor the normal way via Orbot and collect the logs from your computer using +```console $ adb logcat +``` diff --git a/doc/HACKING/tracing/EventsCircuit.md b/doc/HACKING/tracing/EventsCircuit.md new file mode 100644 index 0000000000..42abdda856 --- /dev/null +++ b/doc/HACKING/tracing/EventsCircuit.md @@ -0,0 +1,139 @@ +# Circuit Subsystem Trace Events + +The circuit subsystem emits a series of tracing events related to a circuit +object life cycle and its state change. + +This document describes each event as in what data they record and what they +represent. + +## Background + +There are two types of circuits: origin and OR (onion router). Both of them +are derived from a base object called a general circuit. + +- Origin circuits are the ones initiated by tor itself so client or onion + service circuits for instance. + +- OR circuits are the ones going through us that we have not initiated and + thus only seen by relays. + +Many operations are done on the base (general) circuit, and some are specific +to an origin or OR. The following section describes each of them by circuit +type. + +## Trace Events + +For the LTTng tracer, the subsystem name of these events is: `tor_circuit`. + +Also, unless specified otherwise, every event emits a common set of parameters +thus they should always be expected in the following order: + +- `circ_id`: For an origin circuit, this is the global circuit identifier used + in a cell. For an OR circuit, the value is 0. + +- `purpose`: Purpose of the circuit as in what it is used for. Note that this + can change during the lifetime of a circuit. See `CIRCUIT_PURPOSE_*` in + `core/or/circuitlist.h` for an exhaustive list of the possible values. + +- `state`: State of a circuit. This changes during the lifetime of a circuit. + See `CIRCUIT_STATE_*` in `core/or/circuitlist.h` for an exhaustive list of + the possible values. + +Now, the tracing events. + +### General Circuit (`circuit_t`) + +The following events are triggered for the base circuit object and thus apply +to all types of circuits. + + * `free`: A circuit object is freed that is memory is released and not + usable anymore. After this event, no more events will be emitted for the + specific circuit object. + + * `mark_for_close`: A circuit object is marked for close that is scheduled + to be closed in a later mainloop periodic event. + + Extra parameters: + + - `end_reason`: Reason why the circuit is closed. Tor often changes that + reason to something generic sometimes in order to avoid leaking internal + reasons to the end point. Thus, this value can be different from + orig_close_reason. + + - `orig_close_reason`: Original reason why the circuit is closed. That + value never changes and contains the internal reason why we close it. It + is **never** this reason that is sent back on the circuit. + + * `change_purpose`: Purpose change. + + Extra parameters: + + (`purpose` parameter is not present) + + - `old_purpose`: Previous purpose that is no longer. + + - `new_purpose`: New purpose assigned to the circuit. + + * `change_state`: State change. + + Extra parameters: + + (`state` parameter is not present) + + - `old_state`: Previous state that is no longer. + + - `new_state`: New state assigned to the circuit. + +### Origin Circuit (`origin_circuit_t`) + +The following events are triggered only for origin circuits. + + * `new_origin`: New origin circuit has been created meaning it has been + newly allocated, initialized and added to the global list. + + * `establish`: Circuit is being established. This is the initial first step + where the path was selected and a connection to the first hop has been + launched. + + * `cannibalized`: Circuit has been cannibalized. This happens when we have + an already opened unused circuit (preemptive circuits) and it was picked. + + * `first_onion_skin`: First onion skin was sent that is the handshake with + the first hop. + + Extra parameters: + + - `fingerprint`: Identity digest (RSA) of the first hop. + + * `intermediate_onion_skin`: An intermediate onion skin was sent which can + be why any hops after the first one. There is thus `N - 1` of these events + where `N` is the total number of hops in the path. + + Extra parameters: + + - `fingerprint`: Identity digest (RSA) of the next hop. + + * `opened`: Circuit just became opened which means that all hops down the + path have negotiated the handshake between them and us and the circuit is + now ready to send cells. + + * `timeout`: Circuit has timed out that is we waited too long for the + circuit to be built. + + * `idle_timeout`: Circuit has timed out due to idleness. This is controlled + by the MaxCircuitDirtiness parameter which is 10 min by default. + +For the common use case of a 3-hop circuit, the following events should be +seen in this order: + + `new_origin` -> `establish` -> `first_onion_skin` -> + `intermediate_onion_skin` -> `intermediate_onion_skin` -> `opened` + +### OR Circuit (`or_circuit_t`) + +The following events are triggered only for OR circuits. For each of them, the +`circ_id` parameter is not present since it would always be 0. The `purpose` +and `state` remain. + + * `new_or`: New OR circuit has been created meaning it has been newly + allocated, initialized and added to the global list. diff --git a/doc/HACKING/tracing/README.md b/doc/HACKING/tracing/README.md new file mode 100644 index 0000000000..d9fb2e5341 --- /dev/null +++ b/doc/HACKING/tracing/README.md @@ -0,0 +1,163 @@ +# Tracing + +This document describes how the event tracing subsystem works in tor so +developers can add events to the code base but also hook them to an event +tracing framework (i.e. tracer). + +## WARNING ## + +Tracing the tor daemon **always** generates sensitive data if used in +production (on the public network). + +It **is** ethical for researchers to use tracing for their own tor client (for +example: building paths, timings, or performance). + +It is **NOT** ethical to archive, publish or keep data containing other users' +activity such as relay data or anything that handles users' traffic. This +of course includes any logs below notice level. + +Publishing analysis of tracing data containing user traffic is **NOT** safe +either. + +In other words, tracing data that contains other users's activity is **NOT** +safe to publish in any form. + +## Basics ### + +Tracing is separated in two different concepts. The tracing API and the +tracing probes. + +The API is in `src/lib/trace/` which defines how to call tracepoints in the +tor code. Every C files should include `src/lib/trace/events.h` if they want +to call a tracepoint. + +The probes are what actually record the tracepoint data. Because they often +need to access specific subsystem objects, the probes are within each +subsystem. They are defined in the `trace-probes-<subsystem>.c` files. + +### Events + +A trace event is basically a function from which we can pass any data that we +want to collect. In addition, we specify a context for the event such as the +subsystem and an event name. + +A trace event in tor has the following standard format: + +```c +tor_trace(subsystem, event_name, args...); +``` + +The `subsystem` parameter is the name of the subsytem the trace event is in. +For example that could be "scheduler" or "vote" or "hs". The idea is to add +some context to the event so when we collect them we know where it's coming +from. + +The `event_name` is the name of the event which adds better semantic to the +event. + +The `args` can be any number of arguments we want to collect. + +Here is an example of a possible tracepoint in main(): + +```c +tor_trace(main, init_phase, argc); +``` + +The above is a tracepoint in the `main` subsystem with `init_phase` as the +event name and the `int argc` is passed to the event as one argument. + +How `argc` is collected or used has nothing to do with the instrumentation +(adding trace events to the code). It is the work of the tracer so this is why +the trace events and collection framework (tracer) are decoupled. You _can_ +have trace events without a tracer. + +### Instrumentation ### + +In `src/lib/trace/events.h`, we map the high level `tor_trace()` macro to one +or many enabled instrumentation. + +Currently, we have 3 types of possible instrumentation: + +1. Debug + + This will map every tracepoint to `log_debug()`. However, none of the + arguments will be passed on because we don't know their type nor the string + format of the debug log. The output is standardized like this: + +``` +[debug] __FUNC__: Tracepoint <event_name> from subsystem <subsystem> hit. +``` + +2. USDT + + User Statically-Defined Tracing (USDT) is a kind of probe which can be + handled by a variety of tracers such as SystemTap, DTrace, perf, eBPF and + ftrace. + + For each tracer, one will need to define the ABI in order for the tracer to + be able to extract the data from the tracepoint objects. For instance, the + tracer needs to know how to print the circuit state of a `circuit_t` + object. + +3. LTTng-UST + + LTTng Userspace is a tracer that has it own type of instrumentation. The + probe definitions are created within the C code and is strongly typed. + + For more information, see https://lttng.org/docs. + +## Build System + +This section describes how the instrumentation is integrated into the build +system of tor. + +By default, every tracing events are disabled in tor that is `tor_trace()` is +a NOP thus has no execution cost time. + +To enable a specific instrumentation, there are configure options: + +1. Debug: `--enable-tracing-instrumentation-debug` + +2. USDT: `--enable-tracing-instrumentation-usdt` + +3. LTTng: `--enable-tracing-instrumentation-lttng` + +They can all be used together or independently. If one of them is set, +`HAVE_TRACING` define is set. And for each instrumentation, a +`USE_TRACING_INSTRUMENTATION_<type>` is set. + +## Adding a Tracepoint ## + +This is pretty easy. Let's say you want to add a trace event in +`src/feature/rend/rendcache.c`, you first need to include this file: + +```c +#include "lib/trace/events.h" +``` + +Then, the `tor_trace()` macro can be used with the specific format detailled +before in a previous section. As an example: + +```c +tor_trace(hs, store_desc_as_client, desc, desc_id); +``` + +For `Debug` instrumentation, you have nothing else to do. + +For `USDT`, instrumentation, you will need to define the probes in a way the +specific tracer can understand. For instance, SystemTap requires you to define +a `tapset` for each tracepoints. + +For `LTTng`, you will need to define the probes in the +`trace-probes-<subsystem>.{c|h}` file. See the `trace-probes-circuit.{c|h}` +file as an example and https://lttng.org/docs/v2.11/#doc-instrumenting. + +## Performance ## + +A word about performance when a tracepoint is enabled. One of the goal of a +tracepoint (USDT, LTTng-UST, ...) is that they can be enabled or disabled. By +default, they are disabled which means the tracer will not record the data but +it has to do a check thus the cost is basically the one of a `branch`. + +If enabled, then the performance depends on the tracer. In the case of +LTTng-UST, the event costs around 110nsec. diff --git a/doc/TUNING b/doc/TUNING index 24552a38cb..7da37e1adc 100644 --- a/doc/TUNING +++ b/doc/TUNING @@ -1,13 +1,13 @@ -Most operating systems limit an amount of TCP sockets that can be used +Most operating systems limit an amount of TCP sockets that can be used simultaneously. It is possible for a busy Tor relay to run into these -limits, thus being unable to fully utilize the bandwidth resources it +limits, thus being unable to fully utilize the bandwidth resources it has at its disposal. Following system-specific tips might be helpful to alleviate the aforementioned problem. Linux ----- -Use 'ulimit -n' to raise an allowed number of file descriptors to be +Use 'ulimit -n' to raise an allowed number of file descriptors to be opened on your host at the same time. FreeBSD @@ -17,7 +17,7 @@ Tune the followind sysctl(8) variables: * kern.maxfiles - maximum allowed file descriptors (for entire system) * kern.maxfilesperproc - maximum file descriptors one process is allowed to use - * kern.ipc.maxsockets - overall maximum numbers of sockets for entire + * kern.ipc.maxsockets - overall maximum numbers of sockets for entire system * kern.ipc.somaxconn - size of listen queue for incoming TCP connections for entire system @@ -31,8 +31,8 @@ Mac OS X Since Mac OS X is BSD-based system, most of the above hold for OS X as well. However, launchd(8) is known to modify kern.maxfiles and kern.maxfilesperproc -when it launches tor service (see launchd.plist(5) manpage). Also, -kern.ipc.maxsockets is determined dynamically by the system and thus is +when it launches tor service (see launchd.plist(5) manpage). Also, +kern.ipc.maxsockets is determined dynamically by the system and thus is read-only on OS X. OpenBSD @@ -79,8 +79,7 @@ Disclaimer Do note that this document is a draft and above information may be technically incorrect and/or incomplete. If so, please open a ticket -on https://trac.torproject.org or post to tor-relays mailing list. +on https://gitlab.torproject.org or post to tor-relays mailing list. Are you running a busy Tor relay? Let us know how you are solving the out-of-sockets problem on your system. - diff --git a/doc/include.am b/doc/include.am index 8651f845eb..7a8a64ed16 100644 --- a/doc/include.am +++ b/doc/include.am @@ -12,7 +12,7 @@ # part of the source distribution, so that people without asciidoc can # just use the .1 and .html files. -all_mans = doc/tor doc/tor-gencert doc/tor-resolve doc/torify doc/tor-print-ed-signing-cert +all_mans = doc/man/tor doc/man/tor-gencert doc/man/tor-resolve doc/man/torify doc/man/tor-print-ed-signing-cert if USE_ASCIIDOC txt_in = $(all_mans:=.1.txt) @@ -52,7 +52,6 @@ EXTRA_DIST+= doc/asciidoc-helper.sh \ doc/HACKING/README.1st.md \ doc/HACKING/CodingStandards.md \ doc/HACKING/CodingStandardsRust.md \ - doc/HACKING/CodeStructure.md \ doc/HACKING/Fuzzing.md \ doc/HACKING/GettingStarted.md \ doc/HACKING/GettingStartedRust.md \ @@ -60,8 +59,9 @@ EXTRA_DIST+= doc/asciidoc-helper.sh \ doc/HACKING/HowToReview.md \ doc/HACKING/Module.md \ doc/HACKING/ReleasingTor.md \ - doc/HACKING/Tracing.md \ doc/HACKING/WritingTests.md + doc/HACKING/tracing/Tracing.md \ + doc/HACKING/tracing/EventsCircuit.md docdir = @docdir@ @@ -77,17 +77,17 @@ $(html_in) : $(man_in) : $(AM_V_GEN)$(top_srcdir)/doc/asciidoc-helper.sh man @A2X@ $(top_srcdir)/$@ -doc/tor.1.in: doc/tor.1.txt -doc/torify.1.in: doc/torify.1.txt -doc/tor-gencert.1.in: doc/tor-gencert.1.txt -doc/tor-resolve.1.in: doc/tor-resolve.1.txt -doc/tor-print-ed-signing-cert.1.in: doc/tor-print-ed-signing-cert.1.txt +doc/man/tor.1.in: doc/man/tor.1.txt +doc/man/torify.1.in: doc/man/torify.1.txt +doc/man/tor-gencert.1.in: doc/man/tor-gencert.1.txt +doc/man/tor-resolve.1.in: doc/man/tor-resolve.1.txt +doc/man/tor-print-ed-signing-cert.1.in: doc/man/tor-print-ed-signing-cert.1.txt -doc/tor.html.in: doc/tor.1.txt -doc/torify.html.in: doc/torify.1.txt -doc/tor-gencert.html.in: doc/tor-gencert.1.txt -doc/tor-resolve.html.in: doc/tor-resolve.1.txt -doc/tor-print-ed-signing-cert.html.in: doc/tor-print-ed-signing-cert.1.txt +doc/man/tor.html.in: doc/man/tor.1.txt +doc/man/torify.html.in: doc/man/torify.1.txt +doc/man/tor-gencert.html.in: doc/man/tor-gencert.1.txt +doc/man/tor-resolve.html.in: doc/man/tor-resolve.1.txt +doc/man/tor-print-ed-signing-cert.html.in: doc/man/tor-print-ed-signing-cert.1.txt # use config.status to swap all machine-specific magic strings # in the asciidoc with their replacements. @@ -98,17 +98,17 @@ $(asciidoc_product) : fi $(AM_V_at)$(top_builddir)/config.status -q --file=$@; -doc/tor.html: doc/tor.html.in -doc/tor-gencert.html: doc/tor-gencert.html.in -doc/tor-resolve.html: doc/tor-resolve.html.in -doc/tor-print-ed-signing-cert.html: doc/tor-print-ed-signing-cert.html.in -doc/torify.html: doc/torify.html.in - -doc/tor.1: doc/tor.1.in -doc/tor-gencert.1: doc/tor-gencert.1.in -doc/tor-resolve.1: doc/tor-resolve.1.in -doc/tor-print-ed-signing-cert.1: doc/tor-print-ed-signing-cert.1.in -doc/torify.1: doc/torify.1.in +doc/man/tor.html: doc/man/tor.html.in +doc/man/tor-gencert.html: doc/man/tor-gencert.html.in +doc/man/tor-resolve.html: doc/man/tor-resolve.html.in +doc/man/tor-print-ed-signing-cert.html: doc/man/tor-print-ed-signing-cert.html.in +doc/man/torify.html: doc/man/torify.html.in + +doc/man/tor.1: doc/man/tor.1.in +doc/man/tor-gencert.1: doc/man/tor-gencert.1.in +doc/man/tor-resolve.1: doc/man/tor-resolve.1.in +doc/man/tor-print-ed-signing-cert.1: doc/man/tor-print-ed-signing-cert.1.in +doc/man/torify.1: doc/man/torify.1.in CLEANFILES+= $(asciidoc_product) DISTCLEANFILES+= $(html_in) $(man_in) diff --git a/doc/tor-gencert.1.txt b/doc/man/tor-gencert.1.txt index 26f68b29c0..9262c4a39a 100644 --- a/doc/tor-gencert.1.txt +++ b/doc/man/tor-gencert.1.txt @@ -6,7 +6,6 @@ :man manual: Tor Manual tor-gencert(1) ============== -Nick Mathewson NAME ---- @@ -14,7 +13,7 @@ tor-gencert - Generate certs and keys for Tor directory authorities SYNOPSIS -------- -**tor-gencert** [-h|--help] [-v] [-r|--reuse] [--create-identity-key] [-i __id_file__] [-c +**tor-gencert** [-h|--help] [-v] [-r|--reuse] [--create-identity-key] [-i __id_file__] [-c __cert_file__] [-m __num__] [-a __address__:__port__] DESCRIPTION @@ -86,7 +85,3 @@ SEE ALSO **tor**(1) + See also the "dir-spec.txt" file, distributed with Tor. - -AUTHORS -------- - Roger Dingledine <arma@mit.edu>, Nick Mathewson <nickm@alum.mit.edu>. diff --git a/doc/tor-print-ed-signing-cert.1.txt b/doc/man/tor-print-ed-signing-cert.1.txt index 71c8b67ec4..2ce10d7cbd 100644 --- a/doc/tor-print-ed-signing-cert.1.txt +++ b/doc/man/tor-print-ed-signing-cert.1.txt @@ -32,7 +32,3 @@ SEE ALSO **tor**(1) + https://spec.torproject.org/cert-spec - -AUTHORS -------- -Roger Dingledine <arma@mit.edu>, Nick Mathewson <nickm@alum.mit.edu>. diff --git a/doc/tor-resolve.1.txt b/doc/man/tor-resolve.1.txt index 17a77e482f..c944d75650 100644 --- a/doc/tor-resolve.1.txt +++ b/doc/man/tor-resolve.1.txt @@ -48,7 +48,3 @@ SEE ALSO **tor**(1), **torify**(1). + For protocol details, see: https://spec.torproject.org/socks-extensions - -AUTHORS -------- -Roger Dingledine <arma@mit.edu>, Nick Mathewson <nickm@alum.mit.edu>. diff --git a/doc/tor.1.txt b/doc/man/tor.1.txt index 3dc8c7a034..5f94f414b8 100644 --- a/doc/tor.1.txt +++ b/doc/man/tor.1.txt @@ -97,11 +97,10 @@ The following options in this section are only recognized on the [[opt-verify-config]] **`--verify-config`**:: Verify whether the configuration file is valid. -[[opt-dump-config]] **`--dump-config`** **`short`**|**`full`**|**`non-builtin`**:: - Write a complete list of Tor's configured options to standard output. +[[opt-dump-config]] **`--dump-config`** **`short`**|**`full`**:: + Write a list of Tor's configured options to standard output. When the `short` flag is selected, only write the options that - are different from their default values. When `non-builtin` is selected, - write options that are not zero or the empty string. + are different from their default values When `full` is selected, write every option. [[opt-serviceinstall]] **`--service install`** [**`--options`** __command-line options__]:: @@ -151,7 +150,7 @@ The following options in this section are only recognized on the + Use the **`--newpass`** option with **`--keygen`** only when you need to add, change, or remove a passphrase on an existing ed25519 - master identity key. You will be prompted for the old passphase + master identity key. You will be prompted for the old passphrase (if any), and the new passphrase (if any). + [NOTE] @@ -175,16 +174,22 @@ The following options in this section are only recognized on the If the file descriptor is not specified, the passphrase is read from the terminal by default. -[[opt-key-expiration]] **`--key-expiration`** [__purpose__]:: +[[opt-key-expiration]] **`--key-expiration`** [__purpose__] [**`--format`** **`iso8601`**|**`timestamp`**]:: The __purpose__ specifies which type of key certificate to determine the expiration of. The only currently recognised __purpose__ is "sign". + + Running **`tor --key-expiration sign`** will attempt to find your signing key certificate and will output, both in the logs as well - as to stdout, the signing key certificate's expiration time in - ISO-8601 format. For example, the output sent to stdout will be - of the form: "signing-cert-expiry: 2017-07-25 08:30:15 UTC" + as to stdout. The optional **`--format`** argument lets you specify + the time format. Currently, **`iso8601`** and **`timestamp`** are + supported. If **`--format`** is not specified, the signing key + certificate's expiration time will be in ISO-8601 format. For example, + the output sent to stdout will be of the form: + "signing-cert-expiry: 2017-07-25 08:30:15 UTC". If **`--format`** **`timestamp`** + is specified, the signing key certificate's expiration time will be in + Unix timestamp format. For example, the output sent to stdout will be of the form: + "signing-cert-expiry: 1500971415". [[opt-dbg]] **--dbg-**...:: Tor may support other options beginning with the string "dbg". These @@ -206,14 +211,22 @@ backslash character (\) before the end of the line. Comments can be used in such multiline entries, but they must start at the beginning of a line. Configuration options can be imported from files or folders using the %include -option with the value being a path. If the path is a file, the options from the -file will be parsed as if they were written where the %include option is. If +option with the value being a path. This path can have wildcards. Wildcards are +expanded first, then sorted using lexical order. Then, for each matching file or +folder, the following rules are followed: if the path is a file, the options from +the file will be parsed as if they were written where the %include option is. If the path is a folder, all files on that folder will be parsed following lexical -order. Files starting with a dot are ignored. Files on subfolders are ignored. +order. Files starting with a dot are ignored. Files in subfolders are ignored. The %include option can be used recursively. New configuration files or directories cannot be added to already running Tor instance if **Sandbox** is enabled. +The supported wildcards are * meaning any number of characters including none +and ? meaning exactly one character. These characters can be escaped by preceding +them with a backslash, except on Windows. Files starting with a dot are not matched +when expanding wildcards unless the starting dot is explicitly in the pattern, except +on Windows. + By default, an option on the command line overrides an option found in the configuration file, and an option in a configuration file overrides one in the defaults file. @@ -263,11 +276,6 @@ forward slash (/) in the configuration file and on the command line. AlternateBridgeAuthority replaces the default bridge authority, but leaves the directory authorities alone. -[[AndroidIdentityTag]] **AndroidIdentityTag** __tag__:: - When logging to Android's logging subsystem, adds a tag to the log identity - such that log entries are marked with "Tor-__tag__". Can not be changed while - tor is running. (Default: none) - [[AvoidDiskWrites]] **AvoidDiskWrites** **0**|**1**:: If non-zero, try to write to disk less frequently than we would otherwise. This is useful when running on flash memory or other media that support @@ -608,7 +616,7 @@ forward slash (/) in the configuration file and on the command line. **DirCache** fetches and serves all documents except extrainfo descriptors, **DownloadExtraInfo*** fetches extrainfo documents, and serves them if **DirCache** is on, and **UseMicrodescriptors** changes the - flavour of consensues and descriptors that is fetched and used for + flavor of consensuses and descriptors that is fetched and used for building circuits. (Default: 0) [[HardwareAccel]] **HardwareAccel** **0**|**1**:: @@ -675,7 +683,7 @@ forward slash (/) in the configuration file and on the command line. Signal-safe logs are always sent to stderr or stdout. They are also sent to a limited number of log files that are configured to log messages at error severity from the bug or general domains. They are never sent as syslogs, - android logs, control port log events, or to any API-based log + control port log events, or to any API-based log destinations. [[Log2]] **Log** __minSeverity__[-__maxSeverity__] **file** __FILENAME__:: @@ -784,6 +792,17 @@ forward slash (/) in the configuration file and on the command line. field, Tor will use it for separate rate limiting for each connection from a non-relay. (Default: 0) +[[OutboundBindAddressPT]] **OutboundBindAddressPT** __IP__:: + Request that pluggable transports makes all outbound connections + originate from the IP address specified. Because outgoing connections + are handled by the pluggable transport itself, it is not possible for + Tor to enforce whether the pluggable transport honors this option. This + option overrides **OutboundBindAddress** for the same IP version. This + option may be used twice, once with an IPv4 address and once with an + IPv6 address. IPv6 addresses should be wrapped in square brackets. This + setting will be ignored for connections to the loopback addresses + (127.0.0.0/8 and ::1). + [[PidFile]] **PidFile** __FILE__:: On startup, write our PID to FILE. On clean shutdown, remove FILE. Can not be changed while tor is running. @@ -834,10 +853,11 @@ forward slash (/) in the configuration file and on the command line. [[Sandbox]] **Sandbox** **0**|**1**:: If set to 1, Tor will run securely through the use of a syscall sandbox. - Otherwise the sandbox will be disabled. The option is currently an - experimental feature. It only works on Linux-based operating systems, - and only when Tor has been built with the libseccomp library. This option - can not be changed while tor is running. + + Otherwise the sandbox will be disabled. The option only works on + Linux-based operating systems, and only when Tor has been built with the + libseccomp library. Note that this option may be incompatible with some + versions of libc, and some kernel versions. This option can not be + changed while tor is running. + + When the **Sandbox** is 1, the following options can not be changed when tor is running: @@ -1289,15 +1309,6 @@ The following options are useful only for clients (that is, if Every NUM seconds consider whether to build a new circuit. (Default: 30 seconds) -[[OptimisticData]] **OptimisticData** **0**|**1**|**auto**:: - When this option is set, and Tor is using an exit node that supports - the feature, it will try optimistically to send data to the exit node - without waiting for the exit node to report whether the connection - succeeded. This can save a round-trip time for protocols like HTTP - where the client talks first. If OptimisticData is set to **auto**, - Tor will look at the UseOptimisticData parameter in the networkstatus. - (Default: auto) - // These are out of order because they logically belong together [[PathBiasCircThreshold]] **PathBiasCircThreshold** __NUM__ + @@ -1808,7 +1819,7 @@ The following options control when Tor enters and leaves dormant mode: this option only with great caution: it has the potential to create spurious traffic on the network. This option should only be used if Tor is started by an affirmative user activity (like - clicking on an applcation or running a command), and not if Tor + clicking on an application or running a command), and not if Tor is launched for some other reason (for example, by a startup process, or by an application that launches itself on every login.) @@ -1852,6 +1863,9 @@ different from other Tor clients: option overrides this option; if you have configured bridges and UseBridges is 1, the Bridges are used as your entry nodes. + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + The ExcludeNodes option overrides this option: any node listed in both EntryNodes and ExcludeNodes is treated as excluded. See <<ExcludeNodes,ExcludeNodes>> for more information on how to specify nodes. @@ -1864,6 +1878,9 @@ different from other Tor clients: (Example: ExcludeNodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, \{cc}, 255.254.0.0/8) + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + By default, this option is treated as a preference that Tor is allowed to override in order to keep working. For example, if you try to connect to a hidden service, @@ -1889,13 +1906,19 @@ different from other Tor clients: list too. See <<ExcludeNodes,ExcludeNodes>> for more information on how to specify nodes. See also the caveats on the <<ExitNodes,ExitNodes>> option below. - + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + [[ExitNodes]] **ExitNodes** __node__,__node__,__...__:: A list of identity fingerprints, country codes, and address patterns of nodes to use as exit node---that is, a node that delivers traffic for you *outside* the Tor network. See <<ExcludeNodes,ExcludeNodes>> for more information on how to specify nodes. + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + Note that if you list too few nodes here, or if you exclude too many exit nodes with ExcludeExitNodes, you can degrade functionality. For example, if none of the exits you list allows traffic on port 80 or 443, you won't @@ -1934,6 +1957,9 @@ different from other Tor clients: (Example: HSLayer2Nodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, \{cc}, 255.254.0.0/8) + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + When this is set, the resulting hidden service paths will look like: + @@ -1990,6 +2016,9 @@ different from other Tor clients: (Example: HSLayer3Nodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, \{cc}, 255.254.0.0/8) + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + When this is set by itself, the resulting hidden service paths will look like: + C - G - M - L3 - Rend + @@ -2036,12 +2065,15 @@ different from other Tor clients: A list of identity fingerprints and country codes of nodes to use for "middle" hops in your normal circuits. Normal circuits include all circuits except for direct connections - to directory servers. Middle hops are all hops other than exit and entry. + -+ + to directory servers. Middle hops are all hops other than exit and entry. + + + This option can appear multiple times: the values from multiple lines are + spliced together. + + + This is an **experimental** feature that is meant to be used by researchers and developers to test new features in the Tor network safely. Using it - without care will strongly influence your anonymity. This feature might get - removed in the future. + without care will strongly influence your anonymity. Other tor features may + not work with MiddleNodes. This feature might get removed in the future. + The HSLayer2Node and HSLayer3Node options override this option for onion service circuits, if they are set. The vanguards addon will read this @@ -2139,12 +2171,22 @@ is non-zero): binds to. To bind to a different address, use the ORPort and OutboundBindAddress options. +[[AddressDisableIPv6]] **AddressDisableIPv6** **0**|**1**:: + By default, Tor will attempt to find the IPv6 of the relay if there is no + IPv4Only ORPort. If set, this option disables IPv6 auto discovery. This + disables IPv6 address resolution, IPv6 ORPorts, and IPv6 reachability + checks. Also, the relay won't publish an IPv6 ORPort in its + descriptor. (Default: 0) + [[AssumeReachable]] **AssumeReachable** **0**|**1**:: This option is used when bootstrapping a new Tor network. If set to 1, don't do self-reachability testing; just upload your server descriptor - immediately. If **AuthoritativeDirectory** is also set, this option - instructs the dirserver to bypass remote reachability testing too and list - all connected servers as running. + immediately. (Default: 0) + +[[AssumeReachableIPv6]] **AssumeReachableIPv6** **0**|**1**|**auto**:: + Like **AssumeReachable**, but affects only the relay's own IPv6 ORPort. + If this value is set to "auto", then Tor will look at **AssumeReachable** + instead. (Default: auto) [[BridgeRelay]] **BridgeRelay** **0**|**1**:: Sets the relay to act as a "bridge" with respect to relaying connections @@ -3004,6 +3046,12 @@ on the public Tor network. if there is some major bug in Ed25519 link authentication that causes us to label all the relays as not Running. (Default: 1) +[[AuthDirTestReachability]] **AuthDirTestReachability** **0**|**1**:: + Authoritative directories only. If set to 1, then we periodically + check every relay we know about to see whether it is running. + If set to 0, we vote Running for every relay, and don't perform + these tests. (Default: 1) + [[BridgePassword]] **BridgePassword** __Password__:: If set, contains an HTTP authenticator that tells a bridge authority to serve all requested bridge information. Used by the (only partially @@ -3358,7 +3406,6 @@ The following options are used for running a testing Tor network. DirAllowPrivateAddresses 1 EnforceDistinctSubnets 0 - AssumeReachable 1 AuthDirMaxServersPerAddr 0 ClientBootstrapConsensusAuthorityDownloadInitialDelay 0 ClientBootstrapConsensusFallbackDownloadInitialDelay 0 @@ -3826,8 +3873,4 @@ https://spec.torproject.org. See also **torsocks**(1) and **torify**(1). == BUGS Because Tor is still under development, there may be plenty of bugs. Please -report them at https://trac.torproject.org/. - -== AUTHORS - -Roger Dingledine [arma at mit.edu], Nick Mathewson [nickm at alum.mit.edu]. +report them at https://bugs.torproject.org/. diff --git a/doc/torify.1.txt b/doc/man/torify.1.txt index 716625f92d..46b13e18e5 100644 --- a/doc/torify.1.txt +++ b/doc/man/torify.1.txt @@ -34,7 +34,3 @@ different circuits. SEE ALSO -------- **tor**(1), **torsocks**(1) - -AUTHORS -------- -Peter Palfrader and Jacob Appelbaum wrote this manual. diff --git a/doc/state-contents.txt b/doc/state-contents.txt index 44716efc0c..772e108377 100644 --- a/doc/state-contents.txt +++ b/doc/state-contents.txt @@ -14,6 +14,21 @@ Recognized fields are: Time when this state file was written. Given in ISO format (YYYY-MM-DD HH:MM:SS) + + MinutesSinceUserActivity (integer) + Dormant (0, 1, or "auto") + + These values are used to keep track of how long Tor has been idle, + for the purpose of becoming 'dormant' after a long period without + any user-initiated requests. + + "MinutesSinceUserActivity" is the number of minutes since the last + time the user asked us to do something. It is set to zero if we're + dormant. + + "Dormant" is 1 if Tor was dormant when it wrote its state file, 0 if + Tor was active, and "auto" if Tor was starting for the first time. + AccountingBytesReadInInterval (memory unit) AccountingBytesWrittenInInterval (memory unit) AccountingExpectedUsage (memory unit) @@ -36,26 +51,6 @@ Recognized fields are: BytesAtSoftLimit. If we hit the soft limit already, we did so at SoftLimitHitAt. - EntryGuard - EntryGuardDownSince - EntryGuardUnlistedSince - EntryGuardAddedBy - - These lines form sections related to entry guards. Each section - starts with a single EntryGuard line, and is then followed by - information on the state of the Entry guard. - - The EntryGuard line contains a nickname, then an identity digest, of - the guard. - - The EntryGuardDownSince and EntryGuardUnlistedSince lines are present - if the entry guard is believed to be non-running or non-listed. If - present, they contain a line in ISO format (YYYY-MM-DD HH:MM:SS). - - The EntryGuardAddedBy line is optional. It contains three - space-separated fields: the identity of the entry guard, the version of - Tor that added it, and the ISO time at which it was added. - TransportProxy One or more of these may be present. @@ -65,41 +60,183 @@ Recognized fields are: this information to spawn pluggable transport listeners in the same IP address and TCP port even after tor client restarts. - BWHistoryReadEnds (ISO time) - BWHistoryReadInterval (integer, number of seconds) - BWHistoryReadValues (comma-separated list of integer) - BWHistoryReadMaxima (comma-separated list of integer) - BWHistoryWriteEnds - BWHistoryWriteInterval - BWHistoryWriteValues - BWHistoryWriteMaxima - BWHistoryDirReadEnds - BWHistoryDirReadInterval - BWHistoryDirReadValues - BWHistoryDirReadMaxima - BWHistoryDirWriteEnds - BWHistoryDirWriteInterval - BWHistoryDirWriteValues - BWHistoryDirWriteMaxima - - These values record bandwidth history. The "Values" fields are a list, for - some number of "Intervals", of the total amount read/written during that - integer. The "Maxima" are the highest burst for each interval. + BWHistory___Ends (ISO time) + BWHistory___Interval (integer, number of seconds) + BWHistory___Values (comma-separated list of integer) + BWHistory___Maxima (comma-separated list of integer) + + These values record bandwidth history. The "Values" fields are a list, + for some number of "Intervals", of the total amount read/written during + that integer. The "Maxima" are the highest burst for each interval. Interval duration is set by the "Interval" field, in seconds. The "Ends" field is the ending time of the last interval in each list. - The *Read* and *Write* fields are the total amount read and - written; the *DirRead* and *DirWrite* variants are for directory - traffic only. + Recognized values for "___" are: + Read -- total bytes read + Write -- total bytes written + DirRead -- total bytes read for directory connections. + DirWrite -- total bytes written for directory connections. + IPv6Read -- total bytes read on IPv6 connections + IPv6Write -- total bytes written on IPv6 connections LastRotatedOnionKey The last time that we changed our onion key for a new one. Given in ISO format (YYYY-MM-DD HH:MM:SS) + This field is used to ensure that onion key rotations happen with the + appropriate frequency. + TotalBuildTimes CircuitBuildAbandonedCount CircuitBuildTimeBin - XXXX writeme. + These fields are used by the Circuit Build Timeout code, which + tries to learn what times are reasonable for circuit construction, + so that it can reject circuits that take too long to build. + + CircuitBuildTimeBin is a count of circuits that were build + successfully in some timeframe. This entry can repeat; each of + these represents some bar on a histogram. The first integer is a + number of milliseconds; it tells the position of the center of the + histogram bin on the time axis. The second number is a count of + circuits in that bin. + + CircuitBuildTimeAbandonedCount is a count of circuits that we + simply gave up on building because they were taking far too long. + + TotalBuildTimes is the number of circuit build times that we + observed in order to build the above measurements fields. If it + reaches a cap, then older measurements get thrown away. + + Guard [key=value] [key=value]... + + Describes a single entry guard used by the client. Key=value + entries with unrecognized keys are persisted. Order is not + significant. For more information about terminology used here, + system, see guard-spec.txt in the tor specifications repository. + + Recognized keys are: + + in (string) + + The name of a guard selection that this guard is in. + + rsa_id (string) + + RSA fingerprint of this guard, without spaces. + + nickname (string) + + Declared nickname of this guard. + + sampled_on (Time in ISO YYYY-MM-DDTHH:MM:SS format) + + When was this guard added to the Guard sample? + + sampled_by (tor version) + + Which version of Tor added this Guard to the sample? + (Used to help with debugging.) + + sampled_idx (integer) + + Index of this guard among sampled guards. + + listed (boolean) + + Did this guard appear in the most recent consensus? + + unlisted_since (Time in ISO YYYY-MM-DDTHH:MM:SS format) + + If this guard is not listed, when is the earliest + consensus in which we found it unlisted? + + confirmed_on (Time in ISO YYYY-MM-DDTHH:MM:SS format) + + When did this guard become confirmed? + + confirmed_idx (integer) + + Index of this guard among confirmed guards. + + bridge_addr (address) + + If this guard is a bridge, its current address. + + pb_use_attempts + pb_use_successes + pb_circ_attempts + pb_successful_circuits_closed + pb_collapsed_circuits + pb_unusable_circuits + pb_timeouts + + Used by the pathbias subsystem to keep a record of the + behavior of circuits built through this guard, in hopes of + detecting guards try to that interfere with traffic. + + All of these fields are floating-point integers which + represent a count of circuits that have been trated in + various ways. These counts decay with time. + + "use_attempts" is a count of the circuits that we've built + and tried to use for traffic. + + "successful_circuits_closed" is a count of circuits that + have closed "naturally" without timeout or error. + + "use_successes" is a count of circuits that we've sent + traffic on, and which closed "naturally" without timeout + or error. + + "circ_attempts" is a count of circuits we've tried to + build through this guard. + + "collapsed_circuits" is a count of circuits that failed + after having been built, but before sending traffic. + + "unusable_circuits" is a count of circuits that we + built, but where streams or probes but which failed, + or which encountered questionable errors. + + "timeouts" is a count of circuits that encountered a + timeout while we were building them. + +Obsolete fields include: + + EntryGuard + EntryGuardDownSince + EntryGuardUnlistedSince + EntryGuardAddedBy + + These lines formed sections related to entry guards. Each section + starts with a single EntryGuard line, and is then followed by + information on the state of the Entry guard. + + The EntryGuard line contains a nickname, then an identity digest, of + the guard. + + The EntryGuardDownSince and EntryGuardUnlistedSince lines are present + if the entry guard is believed to be non-running or non-listed. If + present, they contain a line in ISO format (YYYY-MM-DD HH:MM:SS). + + The EntryGuardAddedBy line is optional. It contains three + space-separated fields: the identity of the entry guard, the version of + Tor that added it, and the ISO time at which it was added. + + EntryGuardPathBias and EntryGuardPathUseBias are superseded by + the `pb_...` elements in the Guard flag, and served a similar purpose. + + These entries have all been superseded by the Guard line type, + since Tor 0.3.0.1-alpha. + + HidServRevCounter + + It was once used to ensure that v3 onion service directory revision + numbers were strictly increasing; we now use an order-preserving + encryption scheme for that purpose. + + This option could appear multiple times; each time it does, it + applies to a different hidden service. diff --git a/doc/ticket40136 b/doc/ticket40136 new file mode 100644 index 0000000000..cd1c0682f6 --- /dev/null +++ b/doc/ticket40136 @@ -0,0 +1,3 @@ + o Documentation: + - Update doc/state-contents.txt to more accurately explain the + current contents of a Tor state file. Closes ticket 40136. |