diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/HACKING/CodingStandards.md | 147 | ||||
| -rw-r--r-- | doc/HACKING/CodingStandardsRust.md | 469 | ||||
| -rw-r--r-- | doc/HACKING/GettingStarted.md | 5 | ||||
| -rw-r--r-- | doc/HACKING/GettingStartedRust.md | 161 | ||||
| -rw-r--r-- | doc/HACKING/HelpfulTools.md | 22 | ||||
| -rw-r--r-- | doc/HACKING/HowToReview.md | 3 | ||||
| -rw-r--r-- | doc/HACKING/ReleasingTor.md | 33 | ||||
| -rw-r--r-- | doc/HACKING/WritingTests.md | 6 | ||||
| -rw-r--r-- | doc/tor.1.txt | 242 | ||||
| -rw-r--r-- | doc/torrc_format.txt | 11 |
10 files changed, 1010 insertions, 89 deletions
diff --git a/doc/HACKING/CodingStandards.md b/doc/HACKING/CodingStandards.md index c7787a72cc..dd21d6fd2c 100644 --- a/doc/HACKING/CodingStandards.md +++ b/doc/HACKING/CodingStandards.md @@ -6,8 +6,8 @@ tl;dr: - Run configure with `--enable-fatal-warnings` - Document your functions - Write unit tests - - Run `make test-full` to test against all unit and integration tests. - - Run `make distcheck` to ensure the distribution works + - Run `make check` before submitting a patch + - Run `make distcheck` if you have made changes to build system components - Add a file in `changes` for your branch. Patch checklist @@ -32,6 +32,16 @@ Did you remember... - To base your code on the appropriate branch? - To include a file in the `changes` directory as appropriate? +If you are submitting a major patch or new feature, or want to in the future... + + - Set up Chutney and Stem, see HACKING/WritingTests.md + - Run `make test-full` to test against all unit and integration tests. + +If you have changed build system components: + - Please run `make distcheck` + - For example, if you have changed Makefiles, autoconf files, or anything + else that affects the build system. + How we use Git branches ======================= @@ -52,8 +62,17 @@ before it gets merged into maint, but that's rare. If you're working on a bugfix for a bug that occurs in a particular version, base your bugfix branch on the "maint" branch for the first supported series -that has that bug. (As of June 2013, we're supporting 0.2.3 and later.) If -you're working on a new feature, base it on the master branch. +that has that bug. (As of June 2013, we're supporting 0.2.3 and later.) + +If you're working on a new feature, base it on the master branch. If you're +working on a new feature and it will take a while to implement and/or you'd +like to avoid the possibility of unrelated bugs in Tor while you're +implementing your feature, consider branching off of the latest maint- branch. +_Never_ branch off a relase- branch. Don't branch off a tag either: they come +from release branches. Doing so will likely produce a nightmare of merge +conflicts in the ChangeLog when it comes time to merge your branch into Tor. +Best advice: don't try to keep an independent branch forked for more than 6 +months and expect it to merge cleanly. Try to merge pieces early and often. How we log changes @@ -77,17 +96,34 @@ you can use `git describe --contains <sha1 of commit>`. If at all possible, try to create this file in the same commit where you are making the change. Please give it a distinctive name that no other branch will use for the lifetime of your change. To verify the format of the changes file, -you can use `make check-changes`. +you can use `make check-changes`. This is run automatically as part of +`make check` -- if it fails, we must fix it before we release. These +checks are implemented in `scripts/maint/lintChanges.py`. + +Changes file style guide: + * Changes files begin with " o Header (subheading):". The header + should usually be "Minor/Major bugfixes/features". The subheading is a + particular area within Tor. See the ChangeLog for examples. + + * Make everything terse. + + * Write from the user's point of view: describe the user-visible changes + right away. + + * Mention configuration options by name. If they're rare or unusual, + remind people what they're for. + + * Describe changes in the present tense and in the imperative: not past. + + * Every bugfix should have a sentence of the form "Fixes bug 1234; bugfix + on 0.1.2.3-alpha", describing what bug was fixed and where it came from. + + * "Relays", not "servers", "nodes", or "Tor relays". When we go to make a release, we will concatenate all the entries in changes to make a draft changelog, and clear the directory. We'll then edit the draft changelog into a nice readable format. -To make sure that stuff is in the right format, we use -scripts/maint/lintChanges.py to check the changes files for -(superficial) validity. You can run this script on your own changes -files! - What needs a changes file? * A not-exhaustive list: Anything that might change user-visible @@ -156,6 +192,79 @@ old C functions. Use `strlcat`, `strlcpy`, or `tor_snprintf/tor_asprintf` inste We don't call `memcmp()` directly. Use `fast_memeq()`, `fast_memneq()`, `tor_memeq()`, or `tor_memneq()` for most purposes. +Also see a longer list of functions to avoid in: +https://people.torproject.org/~nickm/tor-auto/internal/this-not-that.html + +Floating point math is hard +--------------------------- + +Floating point arithmetic as typically implemented by computers is +very counterintuitive. Failure to adequately analyze floating point +usage can result in surprising behavior and even security +vulnerabilities! + +General advice: + + - Don't use floating point. + - If you must use floating point, document how the limits of + floating point precision and calculation accuracy affect function + outputs. + - Try to do as much as possible of your calculations using integers + (possibly acting as fixed-point numbers) and convert to floating + point for display. + - If you must send floating point numbers on the wire, serialize + them in a platform-independent way. Tor avoids exchanging + floating-point values, but when it does, it uses ASCII numerals, + with a decimal point ("."). + - Binary fractions behave very differently from decimal fractions. + Make sure you understand how these differences affect your + calculations. + - Every floating point arithmetic operation is an opportunity to + lose precision, overflow, underflow, or otherwise produce + undesired results. Addition and subtraction tend to be worse + than multiplication and division (due to things like catastrophic + cancellation). Try to arrange your calculations to minimize such + effects. + - Changing the order of operations changes the results of many + floating-point calculations. Be careful when you simplify + calculations! If the order is significant, document it using a + code comment. + - Comparing most floating point values for equality is unreliable. + Avoid using `==`, instead, use `>=` or `<=`. If you use an + epsilon value, make sure it's appropriate for the ranges in + question. + - Different environments (including compiler flags and per-thread + state on a single platform!) can get different results from the + same floating point calculations. This means you can't use + floats in anything that needs to be deterministic, like consensus + generation. This also makes reliable unit tests of + floating-point outputs hard to write. + +For additional useful advice (and a little bit of background), see +[What Every Programmer Should Know About Floating-Point +Arithmetic](http://floating-point-gui.de/). + +A list of notable (and surprising) facts about floating point +arithmetic is at [Floating-point +complexities](https://randomascii.wordpress.com/2012/04/05/floating-point-complexities/). +Most of that [series of posts on floating +point](https://randomascii.wordpress.com/category/floating-point/) is +helpful. + +For more detailed (and math-intensive) background, see [What Every +Computer Scientist Should Know About Floating-Point +Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html). + +Other C conventions +------------------- + +The `a ? b : c` trinary operator only goes inside other expressions; +don't use it as a replacement for if. (You can ignore this inside macro +definitions when necessary.) + +Assignment operators shouldn't nest inside other expressions. (You can +ignore this inside macro definitions when necessary.) + Functions not to write ---------------------- @@ -217,7 +326,25 @@ end-users that they aren't expected to understand the message (perhaps with a string like "internal error"). Option (A) is to be preferred to option (B). +Assertions In Tor +----------------- + +Assertions should be used for bug-detection only. Don't use assertions to +detect bad user inputs, network errors, resource exhaustion, or similar +issues. + +Tor is always built with assertions enabled, so try to only use +`tor_assert()` for cases where you are absolutely sure that crashing is the +least bad option. Many bugs have been caused by use of `tor_assert()` when +another kind of check would have been safer. + +If you're writing an assertion to test for a bug that you _can_ recover from, +use `tor_assert_nonfatal()` in place of `tor_assert()`. If you'd like to +write a conditional that incorporates a nonfatal assertion, use the `BUG()` +macro, as in: + if (BUG(ptr == NULL)) + return -1; Doxygen comment conventions --------------------------- diff --git a/doc/HACKING/CodingStandardsRust.md b/doc/HACKING/CodingStandardsRust.md new file mode 100644 index 0000000000..d0b17c1604 --- /dev/null +++ b/doc/HACKING/CodingStandardsRust.md @@ -0,0 +1,469 @@ + + Rust Coding Standards +======================= + +You MUST follow the standards laid out in `.../doc/HACKING/CodingStandards.md`, +where applicable. + + Module/Crate Declarations +--------------------------- + +Each Tor C module which is being rewritten MUST be in its own crate. +See the structure of `.../src/rust` for examples. + +In your crate, you MUST use `lib.rs` ONLY for pulling in external +crates (e.g. `extern crate libc;`) and exporting public objects from +other Rust modules (e.g. `pub use mymodule::foo;`). For example, if +you create a crate in `.../src/rust/yourcrate`, your Rust code should +live in `.../src/rust/yourcrate/yourcode.rs` and the public interface +to it should be exported in `.../src/rust/yourcrate/lib.rs`. + +If your code is to be called from Tor C code, you MUST define a safe +`ffi.rs`. See the "Safety" section further down for more details. + +For example, in a hypothetical `tor_addition` Rust module: + +In `.../src/rust/tor_addition/addition.rs`: + + pub fn get_sum(a: i32, b: i32) -> i32 { + a + b + } + +In `.../src/rust/tor_addition/lib.rs`: + + pub use addition::*; + +In `.../src/rust/tor_addition/ffi.rs`: + + #[no_mangle] + pub extern "C" fn tor_get_sum(a: c_int, b: c_int) -> c_int { + get_sum(a, b) + } + +If your Rust code must call out to parts of Tor's C code, you must +declare the functions you are calling in the `external` crate, located +at `.../src/rust/external`. + +<!-- XXX get better examples of how to declare these externs, when/how they --> +<!-- XXX are unsafe, what they are expected to do —isis --> + +Modules should strive to be below 500 lines (tests excluded). Single +responsibility and limited dependencies should be a guiding standard. + +If you have any external modules as dependencies (e.g. `extern crate +libc;`), you MUST declare them in your crate's `lib.rs` and NOT in any +other module. + + Dependencies +-------------- + +In general, we use modules from only the Rust standard library +whenever possible. We will review including external crates on a +case-by-case basis. + + Documentation +--------------- + +You MUST include `#[deny(missing_docs)]` in your crate. + +For function/method comments, you SHOULD include a one-sentence, "first person" +description of function behaviour (see requirements for documentation as +described in `.../src/HACKING/CodingStandards.md`), then an `# Inputs` section +for inputs or initialisation values, a `# Returns` section for return +values/types, a `# Warning` section containing warnings for unsafe behaviours or +panics that could happen. For publicly accessible +types/constants/objects/functions/methods, you SHOULD also include an +`# Examples` section with runnable doctests. + +You MUST document your module with _module docstring_ comments, +i.e. `//!` at the beginning of each line. + + Style +------- + +You SHOULD consider breaking up large literal numbers with `_` when it makes it +more human readable to do so, e.g. `let x: u64 = 100_000_000_000`. + + Testing +--------- + +All code MUST be unittested and integration tested. + +Public functions/objects exported from a crate SHOULD include doctests +describing how the function/object is expected to be used. + +Integration tests SHOULD go into a `tests/` directory inside your +crate. Unittests SHOULD go into their own module inside the module +they are testing, e.g. in `.../src/rust/tor_addition/addition.rs` you +should put: + + #[cfg(test)] + mod test { + use super::*; + + #[test] + fn addition_with_zero() { + let sum: i32 = get_sum(5i32, 0i32); + assert_eq!(sum, 5); + } + } + + Benchmarking +-------------- + +The external `test` crate can be used for most benchmarking. However, using +this crate requires nightly Rust. Since we may want to switch to a more +stable Rust compiler eventually, we shouldn't do things which will automatically +break builds for stable compilers. Therefore, you MUST feature-gate your +benchmarks in the following manner. + +If you wish to benchmark some of your Rust code, you MUST put the +following in the `[features]` section of your crate's `Cargo.toml`: + + [features] + bench = [] + +Next, in your crate's `lib.rs` you MUST put: + + #[cfg(all(test, feature = "bench"))] + extern crate test; + +This ensures that the external crate `test`, which contains utilities +for basic benchmarks, is only used when running benchmarks via `cargo +bench --features bench`. + +Finally, to write your benchmark code, in +`.../src/rust/tor_addition/addition.rs` you SHOULD put: + + #[cfg(all(test, features = "bench"))] + mod bench { + use test::Bencher; + use super::*; + + #[bench] + fn addition_small_integers(b: &mut Bencher) { + b.iter(| | get_sum(5i32, 0i32)); + } + } + + Fuzzing +--------- + +If you wish to fuzz parts of your code, please see the +[`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz) crate, which uses +[libfuzzer-sys](https://github.com/rust-fuzz/libfuzzer-sys). + + Whitespace & Formatting +------------------------- + +You MUST run `rustfmt` (https://github.com/rust-lang-nursery/rustfmt) +on your code before your code will be merged. You can install rustfmt +by doing `cargo install rustfmt-nightly` and then run it with `cargo +fmt`. + + Safety +-------- + +You SHOULD read [the nomicon](https://doc.rust-lang.org/nomicon/) before writing +Rust FFI code. It is *highly advised* that you read and write normal Rust code +before attempting to write FFI or any other unsafe code. + +Here are some additional bits of advice and rules: + +0. Any behaviours which Rust considers to be undefined are forbidden + + From https://doc.rust-lang.org/reference/behavior-considered-undefined.html: + + > Behavior considered undefined + > + > The following is a list of behavior which is forbidden in all Rust code, + > including within unsafe blocks and unsafe functions. Type checking provides the + > guarantee that these issues are never caused by safe code. + > + > * Data races + > * Dereferencing a null/dangling raw pointer + > * Reads of [undef](http://llvm.org/docs/LangRef.html#undefined-values) + > (uninitialized) memory + > * Breaking the + > [pointer aliasing rules](http://llvm.org/docs/LangRef.html#pointer-aliasing-rules) + > with raw pointers (a subset of the rules used by C) + > * `&mut T` and `&T` follow LLVM’s scoped noalias model, except if the `&T` + > contains an `UnsafeCell<U>`. Unsafe code must not violate these aliasing + > guarantees. + > * Mutating non-mutable data (that is, data reached through a shared + > reference or data owned by a `let` binding), unless that data is + > contained within an `UnsafeCell<U>`. + > * Invoking undefined behavior via compiler intrinsics: + > - Indexing outside of the bounds of an object with + > `std::ptr::offset` (`offset` intrinsic), with the exception of + > one byte past the end which is permitted. + > - Using `std::ptr::copy_nonoverlapping_memory` (`memcpy32`/`memcpy64` + > intrinsics) on overlapping buffers + > * Invalid values in primitive types, even in private fields/locals: + > - Dangling/null references or boxes + > - A value other than `false` (0) or `true` (1) in a `bool` + > - A discriminant in an `enum` not included in the type definition + > - A value in a `char` which is a surrogate or above `char::MAX` + > - Non-UTF-8 byte sequences in a `str` + > * Unwinding into Rust from foreign code or unwinding from Rust into foreign + > code. Rust's failure system is not compatible with exception handling in other + > languages. Unwinding must be caught and handled at FFI boundaries. + +1. `unwrap()` + + If you call `unwrap()`, anywhere, even in a test, you MUST include + an inline comment stating how the unwrap will either 1) never fail, + or 2) should fail (i.e. in a unittest). + + You SHOULD NOT use `unwrap()` anywhere in which it is possible to handle the + potential error with either `expect()` or the eel operator, `?`. + For example, consider a function which parses a string into an integer: + + fn parse_port_number(config_string: &str) -> u16 { + u16::from_str_radix(config_string, 10).unwrap() + } + + There are numerous ways this can fail, and the `unwrap()` will cause the + whole program to byte the dust! Instead, either you SHOULD use `expect()` + (or another equivalent function which will return an `Option` or a `Result`) + and change the return type to be compatible: + + fn parse_port_number(config_string: &str) -> Option<u16> { + u16::from_str_radix(config_string, 10).expect("Couldn't parse port into a u16") + } + + or you SHOULD use `or()` (or another similar method): + + fn parse_port_number(config_string: &str) -> Option<u16> { + u16::from_str_radix(config_string, 10).or(Err("Couldn't parse port into a u16") + } + + Using methods like `or()` can be particularly handy when you must do + something afterwards with the data, for example, if we wanted to guarantee + that the port is high. Combining these methods with the eel operator (`?`) + makes this even easier: + + fn parse_port_number(config_string: &str) -> Result<u16, Err> { + let port = u16::from_str_radix(config_string, 10).or(Err("Couldn't parse port into a u16"))?; + + if port > 1024 { + return Ok(port); + } else { + return Err("Low ports not allowed"); + } + } + +2. `unsafe` + + If you use `unsafe`, you MUST describe a contract in your + documentation which describes how and when the unsafe code may + fail, and what expectations are made w.r.t. the interfaces to + unsafe code. This is also REQUIRED for major pieces of FFI between + C and Rust. + + When creating an FFI in Rust for C code to call, it is NOT REQUIRED + to declare the entire function `unsafe`. For example, rather than doing: + + #[no_mangle] + pub unsafe extern "C" fn increment_and_combine_numbers(mut numbers: [u8; 4]) -> u32 { + for number in &mut numbers { + *number += 1; + } + std::mem::transmute::<[u8; 4], u32>(numbers) + } + + You SHOULD instead do: + + #[no_mangle] + pub extern "C" fn increment_and_combine_numbers(mut numbers: [u8; 4]) -> u32 { + for index in 0..numbers.len() { + numbers[index] += 1; + } + unsafe { + std::mem::transmute::<[u8; 4], u32>(numbers) + } + } + +3. Pass only integer types and bytes over the boundary + + The only non-integer type which may cross the FFI boundary is + bytes, e.g. `&[u8]`. This SHOULD be done on the Rust side by + passing a pointer (`*mut libc::c_char`) and a length + (`libc::size_t`). + + One might be tempted to do this via doing + `CString::new("blah").unwrap().into_raw()`. This has several problems: + + a) If you do `CString::new("bl\x00ah")` then the unwrap() will fail + due to the additional NULL terminator, causing a dangling + pointer to be returned (as well as a potential use-after-free). + + b) Returning the raw pointer will cause the CString to run its deallocator, + which causes any C code which tries to access the contents to dereference a + NULL pointer. + + c) If we were to do `as_raw()` this would result in a potential double-free + since the Rust deallocator would run and possibly Tor's deallocator. + + d) Calling `into_raw()` without later using the same pointer in Rust to call + `from_raw()` and then deallocate in Rust can result in a + [memory leak](https://doc.rust-lang.org/std/ffi/struct.CString.html#method.into_raw). + + [It was determined](https://github.com/rust-lang/rust/pull/41074) that this + is safe to do if you use the same allocator in C and Rust and also specify + the memory alignment for CString (except that there is no way to specify + the alignment for CString). It is believed that the alignment is always 1, + which would mean it's safe to dealloc the resulting `*mut c_char` in Tor's + C code. However, the Rust developers are not willing to guarantee the + stability of, or a contract for, this behaviour, citing concerns that this + is potentially extremely and subtly unsafe. + +4. Perform an allocation on the other side of the boundary + + After crossing the boundary, the other side MUST perform an + allocation to copy the data and is therefore responsible for + freeing that memory later. + +5. No touching other language's enums + + Rust enums should never be touched from C (nor can they be safely + `#[repr(C)]`) nor vice versa: + + > "The chosen size is the default enum size for the target platform's C + > ABI. Note that enum representation in C is implementation defined, so this is + > really a "best guess". In particular, this may be incorrect when the C code + > of interest is compiled with certain flags." + + (from https://gankro.github.io/nomicon/other-reprs.html) + +6. Type safety + + Wherever possible and sensical, you SHOULD create new types in a + manner which prevents type confusion or misuse. For example, + rather than using an untyped mapping between strings and integers + like so: + + use std::collections::HashMap; + + pub fn get_elements_with_over_9000_points(map: &HashMap<String, usize>) -> Vec<String> { + ... + } + + It would be safer to define a new type, such that some other usage + of `HashMap<String, usize>` cannot be confused for this type: + + pub struct DragonBallZPowers(pub HashMap<String, usize>); + + impl DragonBallZPowers { + pub fn over_nine_thousand<'a>(&'a self) -> Vec<&'a String> { + let mut powerful_enough: Vec<&'a String> = Vec::with_capacity(5); + + for (character, power) in &self.0 { + if *power > 9000 { + powerful_enough.push(character); + } + } + powerful_enough + } + } + + Note the following code, which uses Rust's type aliasing, is valid + but it does NOT meet the desired type safety goals: + + pub type Power = usize; + + pub fn over_nine_thousand(power: &Power) -> bool { + if *power > 9000 { + return true; + } + false + } + + // We can still do the following: + let his_power: usize = 9001; + over_nine_thousand(&his_power); + +7. Unsafe mucking around with lifetimes + + Because lifetimes are technically, in type theory terms, a kind, i.e. a + family of types, individual lifetimes can be treated as types. For example, + one can arbitrarily extend and shorten lifetime using `std::mem::transmute`: + + struct R<'a>(&'a i32); + + unsafe fn extend_lifetime<'b>(r: R<'b>) -> R<'static> { + std::mem::transmute::<R<'b>, R<'static>>(r) + } + + unsafe fn shorten_invariant_lifetime<'b, 'c>(r: &'b mut R<'static>) -> &'b mut R<'c> { + std::mem::transmute::<&'b mut R<'static>, &'b mut R<'c>>(r) + } + + Calling `extend_lifetime()` would cause an `R` passed into it to live forever + for the life of the program (the `'static` lifetime). Similarly, + `shorten_invariant_lifetime()` could be used to take something meant to live + forever, and cause it to disappear! This is incredibly unsafe. If you're + going to be mucking around with lifetimes like this, first, you better have + an extremely good reason, and second, you may as be honest and explicit about + it, and for ferris' sake just use a raw pointer. + + In short, just because lifetimes can be treated like types doesn't mean you + should do it. + +8. Doing excessively unsafe things when there's a safer alternative + + Similarly to #7, often there are excessively unsafe ways to do a task and a + simpler, safer way. You MUST choose the safer option where possible. + + For example, `std::mem::transmute` can be abused in ways where casting with + `as` would be both simpler and safer: + + // Don't do this + let ptr = &0; + let ptr_num_transmute = unsafe { std::mem::transmute::<&i32, usize>(ptr)}; + + // Use an `as` cast instead + let ptr_num_cast = ptr as *const i32 as usize; + + In fact, using `std::mem::transmute` for *any* reason is a code smell and as + such SHOULD be avoided. + +9. Casting integers with `as` + + This is generally fine to do, but it has some behaviours which you should be + aware of. Casting down chops off the high bits, e.g.: + + let x: u32 = 4294967295; + println!("{}", x as u16); // prints 65535 + + Some cases which you MUST NOT do include: + + * Casting an `u128` down to an `f32` or vice versa (e.g. + `u128::MAX as f32` but this isn't only a problem with overflowing + as it is also undefined behaviour for `42.0f32 as u128`), + + * Casting between integers and floats when the thing being cast + cannot fit into the type it is being casted into, e.g.: + + println!("{}", 42949.0f32 as u8); // prints 197 in debug mode and 0 in release + println!("{}", 1.04E+17 as u8); // prints 0 in both modes + println!("{}", (0.0/0.0) as i64); // prints whatever the heck LLVM wants + + Because this behaviour is undefined, it can even produce segfaults in + safe Rust code. For example, the following program built in release + mode segfaults: + + #[inline(never)] + pub fn trigger_ub(sl: &[u8; 666]) -> &[u8] { + // Note that the float is out of the range of `usize`, invoking UB when casting. + let idx = 1e99999f64 as usize; + &sl[idx..] // The bound check is elided due to `idx` being of an undefined value. + } + + fn main() { + println!("{}", trigger_ub(&[1; 666])[999999]); // ~ out of bound + } + + And in debug mode panics with: + + thread 'main' panicked at 'slice index starts at 140721821254240 but ends at 666', /checkout/src/libcore/slice/mod.rs:754:4 diff --git a/doc/HACKING/GettingStarted.md b/doc/HACKING/GettingStarted.md index 0295adc1ff..0c42404634 100644 --- a/doc/HACKING/GettingStarted.md +++ b/doc/HACKING/GettingStarted.md @@ -11,8 +11,9 @@ whole Tor ecosystem.) If you are looking for a more bare-bones, less user-friendly information -dump of important information, you might like reading doc/HACKING -instead. You should probably read it before you write your first patch. +dump of important information, you might like reading the "torguts" +documents linked to below. You should probably read it before you write +your first patch. Required background diff --git a/doc/HACKING/GettingStartedRust.md b/doc/HACKING/GettingStartedRust.md new file mode 100644 index 0000000000..a5253b46a6 --- /dev/null +++ b/doc/HACKING/GettingStartedRust.md @@ -0,0 +1,161 @@ + + Hacking on Rust in Tor +======================== + + Getting Started +----------------- + +Please read or review our documentation on Rust coding standards +(`.../doc/HACKING/CodingStandardsRust.md`) before doing anything. + +Please also read +[the Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html). We aim +to follow the good example set by the Rust community and be excellent to one +another. Let's be careful with each other, so we can be memory-safe together! + +Next, please contact us before rewriting anything! Rust in Tor is still an +experiment. It is an experiment that we very much want to see succeed, so we're +going slowly and carefully. For the moment, it's also a completely +volunteer-driven effort: while many, if not most, of us are paid to work on Tor, +we are not yet funded to write Rust code for Tor. Please be patient with the +other people who are working on getting more Rust code into Tor, because they +are graciously donating their free time to contribute to this effort. + + Resources for learning Rust +----------------------------- + +**Beginning resources** + +The primary resource for learning Rust is +[The Book](https://doc.rust-lang.org/book/). If you'd like to start writing +Rust immediately, without waiting for anything to install, there is +[an interactive browser-based playground](https://play.rust-lang.org/). + +**Advanced resources** + +If you're interested in playing with various Rust compilers and viewing a very +nicely displayed output of the generated assembly, there is +[the Godbolt compiler explorer](https://rust.godbolt.org/) + +For learning how to write unsafe Rust, read +[The Rustonomicon](https://doc.rust-lang.org/nomicon/). + +For learning everything you ever wanted to know about Rust macros, there is +[The Little Book of Rust Macros](https://danielkeep.github.io/tlborm/book/index.html). + +For learning more about FFI and Rust, see Jake Goulding's +[Rust FFI Omnibus](http://jakegoulding.com/rust-ffi-omnibus/). + + Compiling Tor with Rust enabled +--------------------------------- + +You will need to run the `configure` script with the `--enable-rust` flag to +explicitly build with Rust. Additionally, you will need to specify where to +fetch Rust dependencies, as we allow for either fetching dependencies from Cargo +or specifying a local directory. + +**Fetch dependencies from Cargo** + + ./configure --enable-rust --enable-cargo-online-mode + +**Using a local dependency cache** + +**NOTE**: local dependency caches which were not *originally* created via + `--enable-cargo-online-mode` are broken. See https://bugs.torproject.org/22907 + +To specify a local directory: + + RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust + +(Note that RUST_DEPENDENCIES must be the full path to the directory; it cannot +be relative.) + +You'll need the following Rust dependencies (as of this writing): + + libc==0.2.22 + +To get them, do: + + mkdir path_to_dependencies_directory + cd path_to_dependencies_directory + git clone https://github.com/rust-lang/libc + cd libc + git checkout 0.2.22 + cargo package + cd .. + ln -s libc/target/package/libc-0.2.22 libc-0.2.22 + + + Identifying which modules to rewrite +====================================== + +The places in the Tor codebase that are good candidates for porting to Rust are: + +1. loosely coupled to other Tor submodules, +2. have high test coverage, and +3. would benefit from being implemented in a memory safe language. + +Help in either identifying places such as this, or working to improve existing +areas of the C codebase by adding regression tests and simplifying dependencies, +would be really helpful. + +Furthermore, as submodules in C are implemented in Rust, this is a good +opportunity to refactor, add more tests, and split modules into smaller areas of +responsibility. + +A good first step is to build a module-level callgraph to understand how +interconnected your target module is. + + git clone https://git.torproject.org/user/nickm/calltool.git + cd tor + CFLAGS=0 ./configure + ../calltool/src/main.py module_callgraph + +The output will tell you each module name, along with a set of every module that +the module calls. Modules which call fewer other modules are better targets. + + Writing your Rust module +========================== + +Strive to change the C API as little as possible. + +We are currently targetting Rust nightly, *for now*. We expect this to change +moving forward, as we understand more about which nightly features we need. It +is on our TODO list to try to cultivate good standing with various distro +maintainers of `rustc` and `cargo`, in order to ensure that whatever version we +solidify on is readily available. + + Adding your Rust module to Tor's build system +----------------------------------------------- + +0. Your translation of the C module should live in its own crate(s) + in the `.../tor/src/rust/` directory. +1. Add your crate to `.../tor/src/rust/Cargo.toml`, in the + `[workspace.members]` section. +2. Append your crate's static library to the `rust_ldadd` definition + (underneath `if USE_RUST`) in `.../tor/Makefile.am`. + + How to test your Rust code +---------------------------- + +Everything should be tested full stop. Even non-public functionality. + +Be sure to edit `.../tor/src/test/test_rust.sh` to add the name of your crate to +the `crates` variable! This will ensure that `cargo test` is run on your crate. + +Configure Tor's build system to build with Rust enabled: + + ./configure --enable-fatal-warnings --enable-rust --enable-cargo-online-mode + +Tor's test should be run by doing: + + make check + +Tor's integration tests should also pass: + + make test-stem + + Submitting a patch +===================== + +Please follow the instructions in `.../doc/HACKING/GettingStarted.md`. diff --git a/doc/HACKING/HelpfulTools.md b/doc/HACKING/HelpfulTools.md index b8ba2aa408..f919d08ec1 100644 --- a/doc/HACKING/HelpfulTools.md +++ b/doc/HACKING/HelpfulTools.md @@ -111,15 +111,19 @@ Running gcov for unit test coverage (On OSX, you'll need to start with `--enable-coverage CC=clang`.) -Then, look at the .gcov files in `coverage-output`. '-' before a line means -that the compiler generated no code for that line. '######' means that the -line was never reached. Lines with numbers were called that number of times. - If that doesn't work: * Try configuring Tor with `--disable-gcc-hardening` * You might need to run `make clean` after you run `./configure`. +Then, look at the .gcov files in `coverage-output`. '-' before a line means +that the compiler generated no code for that line. '######' means that the +line was never reached. Lines with numbers were called that number of times. + +For more details about how to read gcov output, see the [Invoking +gcov](https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html) chapter +of the GCC manual. + If you make changes to Tor and want to get another set of coverage results, you can run `make reset-gcov` to clear the intermediary gcov output. @@ -128,9 +132,13 @@ a meaningful diff between them, you can run: ./scripts/test/cov-diff coverage-output1 coverage-output2 | less -In this diff, any lines that were visited at least once will have coverage -"1". This lets you inspect what you (probably) really want to know: which -untested lines were changed? Are there any new untested lines? +In this diff, any lines that were visited at least once will have coverage "1", +and line numbers are deleted. This lets you inspect what you (probably) really +want to know: which untested lines were changed? Are there any new untested +lines? + +If you run ./scripts/test/cov-exclude, it marks excluded unreached +lines with 'x', and excluded reached lines with '!!!'. Running integration tests ------------------------- diff --git a/doc/HACKING/HowToReview.md b/doc/HACKING/HowToReview.md index d53318942f..2d1f3d1c9e 100644 --- a/doc/HACKING/HowToReview.md +++ b/doc/HACKING/HowToReview.md @@ -19,6 +19,8 @@ Top-level smell-checks - Does `make check-spaces` pass? +- Does `make check-changes` pass? + - Does it have a reasonable amount of tests? Do they pass? Do they leak memory? @@ -32,6 +34,7 @@ Top-level smell-checks - If this changes Tor's behavior on the wire, is there a design proposal? +- If this changes anything in the code, is there a "changes" file? Let's look at the code! diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index 4ece4d7a1d..62029b44f0 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -8,7 +8,11 @@ new Tor release: === 0. Preliminaries 1. Get at least three of weasel/arma/Sebastian/Sina to put the new - version number in their approved versions list. + version number in their approved versions list. Give them a few + days to do this if you can. + +2. If this is going to be an important security release, give the packagers + some advance warning: See this list of packagers in IV.3 below. === I. Make sure it works @@ -26,19 +30,24 @@ new Tor release: What about Coverity Scan? + What about clan scan-build? + Does 'make distcheck' complain? - How about 'make test-stem' and 'make test-network'? + How about 'make test-stem' and 'make test-network' and + `make test-network-full`? - Are all those tests still happy with --enable-expensive-hardening ? Any memory leaks? -=== II. Write a changelog. +=== II. Write a changelog -1. Gather the `changes/*` files into a changelog entry, rewriting many +1a. (Alpha release variant) + + Gather the `changes/*` files into a changelog entry, rewriting many of them and reordering to focus on what users and funders would find interesting and understandable. @@ -83,6 +92,15 @@ new Tor release: 4. Run `./scripts/maint/format_changelog.py --inplace` to make it prettier +1b. (old-stable release variant) + + For stable releases that backport things from later, we try to compose + their releases, we try to make sure that we keep the changelog entries + identical to their original versions, with a 'backport from 0.x.y.z' + note added to each section. So in this case, once you have the items + from the changes files copied together, don't use them to build a new + changelog: instead, look up the corrected versions that were merged + into ChangeLog in the master branch, and use those. 2. Compose a short release blurb to highlight the user-facing changes. Insert said release blurb into the ChangeLog stanza. If it's @@ -109,7 +127,7 @@ new Tor release: === III. Making the source release. 1. In `maint-0.?.x`, bump the version number in `configure.ac` and run - `scripts/maint/updateVersions.pl` to update version numbers in other + `perl scripts/maint/updateVersions.pl` to update version numbers in other places, and commit. Then merge `maint-0.?.x` into `release-0.?.x`. (NOTE: To bump the version number, edit `configure.ac`, and then run @@ -147,10 +165,13 @@ new Tor release: - {weasel,gk,mikeperry} at torproject dot org - {blueness} at gentoo dot org - {paul} at invizbox dot io + - {vincent} at invizbox dot com - {lfleischer} at archlinux dot org - {Nathan} at freitas dot net - {mike} at tig dot as - {tails-rm} at boum dot org + - {simon} at sdeziel.info + - {yuri} at rawbw.com 4. Add the version number to Trac. To do this, go to Trac, log in, select "Admin" near the top of the screen, then select "Versions" from @@ -162,7 +183,7 @@ new Tor release: 5. Mail the release blurb and ChangeLog to tor-talk (development release) or tor-announce (stable). - Post the changelog on the the blog as well. You can generate a + Post the changelog on the blog as well. You can generate a blog-formatted version of the changelog with the -B option to format-changelog. diff --git a/doc/HACKING/WritingTests.md b/doc/HACKING/WritingTests.md index 4dae41e922..cc393494ec 100644 --- a/doc/HACKING/WritingTests.md +++ b/doc/HACKING/WritingTests.md @@ -91,6 +91,9 @@ coverage percentage. For a summary of the test coverage for each _function_, run `./scripts/test/cov-display -f ${TMPDIR}/*`. +For more details on using gcov, including the helper scripts in +scripts/test, see HelpfulTools.md. + ### Comparing test coverage Sometimes it's useful to compare test coverage for a branch you're writing to @@ -117,7 +120,8 @@ with LCOV_EXCL_START... LCOV_EXCL_STOP. Note that older versions of lcov don't understand these lines. You can post-process .gcov files to make these lines 'unreached' by -running ./scripts/test/cov-exclude on them. +running ./scripts/test/cov-exclude on them. It marks excluded +unreached lines with 'x', and excluded reached lines with '!!!'. Note: you should never do this unless the line is meant to 100% unreachable by actual code. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index a6b4f2fc4b..aad8440a2c 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -128,6 +128,16 @@ COMMAND-LINE OPTIONS the passphrase, including any trailing newlines. Default: read from the terminal. +[[opt-key-expiration]] **--key-expiration** [**purpose**]:: + The **purpose** specifies which type of key certificate to determine + the expiration of. The only currently recognised **purpose** is + "sign". + + + + Running "tor --key-expiration sign" will attempt to find your signing + key certificate and will output, both in the logs as well as to stdout, + the signing key certificate's expiration time in ISO-8601 format. + For example, the output sent to stdout will be of the form: + "signing-cert-expiry: 2017-07-25 08:30:15 UTC" Other options can be specified on the command-line in the format "--option value", in the format "option value", or in a configuration file. For @@ -241,7 +251,9 @@ GENERAL OPTIONS [[ClientTransportPlugin]] **ClientTransportPlugin** __transport__ socks4|socks5 __IP__:__PORT__:: **ClientTransportPlugin** __transport__ exec __path-to-binary__ [options]:: In its first form, when set along with a corresponding Bridge line, the Tor - client forwards its traffic to a SOCKS-speaking proxy on "IP:PORT". It's the + client forwards its traffic to a SOCKS-speaking proxy on "IP:PORT". + (IPv4 addresses should written as-is; IPv6 addresses should be wrapped in + square brackets.) It's the duty of that proxy to properly forward the traffic to the bridge. + + In its second form, when set along with a corresponding Bridge line, the Tor @@ -258,7 +270,8 @@ GENERAL OPTIONS [[ServerTransportListenAddr]] **ServerTransportListenAddr** __transport__ __IP__:__PORT__:: When this option is set, Tor will suggest __IP__:__PORT__ as the listening address of any pluggable transport proxy that tries to - launch __transport__. + launch __transport__. (IPv4 addresses should written as-is; IPv6 + addresses should be wrapped in square brackets.) [[ServerTransportOptions]] **ServerTransportOptions** __transport__ __k=v__ __k=v__ ...:: When this option is set, Tor will pass the __k=v__ parameters to @@ -402,7 +415,7 @@ GENERAL OPTIONS DataDirectory. If the option is set to 1, make the DataDirectory readable by the default GID. (Default: 0) -[[FallbackDir]] **FallbackDir** __address__:__port__ orport=__port__ id=__fingerprint__ [weight=__num__] [ipv6=__address__:__orport__]:: +[[FallbackDir]] **FallbackDir** __ipv4address__:__port__ orport=__port__ id=__fingerprint__ [weight=__num__] [ipv6=**[**__ipv6address__**]**:__orport__]:: When we're unable to connect to any directory cache for directory info (usually because we don't know about any yet) we try a directory authority. Clients also simultaneously try a FallbackDir, to avoid hangs on client @@ -418,7 +431,7 @@ GENERAL OPTIONS FallbackDir line is present, it replaces the hard-coded FallbackDirs, regardless of the value of UseDefaultFallbackDirs.) (Default: 1) -[[DirAuthority]] **DirAuthority** [__nickname__] [**flags**] __address__:__port__ __fingerprint__:: +[[DirAuthority]] **DirAuthority** [__nickname__] [**flags**] __ipv4address__:__port__ __fingerprint__:: Use a nonstandard authoritative directory server at the provided address and port, with the specified key fingerprint. This option can be repeated many times, for multiple authoritative directory servers. Flags are @@ -432,13 +445,16 @@ GENERAL OPTIONS with probability proportional to that weight (default 1.0). If a flag "v3ident=**fp**" is given, the dirserver is a v3 directory authority whose v3 long-term signing key has the fingerprint **fp**. Lastly, - if an "ipv6=__address__:__orport__" flag is present, then the directory + if an "ipv6=**[**__ipv6address__**]**:__orport__" flag is present, then + the directory authority is listening for IPv6 connections on the indicated IPv6 address and OR Port. + + - Tor will contact the authority at __address__:__port__ (the DirPort) to - download directory documents. If an IPv6 address is supplied, Tor will - also download directory documents at the IPv6 address on the DirPort. + + Tor will contact the authority at __ipv4address__ to + download directory documents. The provided __port__ value is a dirport; + clients ignore this in favor of the specified "orport=" value. If an + IPv6 ORPort is supplied, Tor will + also download directory documents at the IPv6 ORPort. + + If no **DirAuthority** line is given, Tor will use the default directory authorities. NOTE: this option is intended for setting up a private Tor @@ -453,9 +469,9 @@ GENERAL OPTIONS should be 1.0 or less. The default is less than 1, to reduce load on authorities. (Default: 0.1) -[[AlternateDirAuthority]] **AlternateDirAuthority** [__nickname__] [**flags**] __address__:__port__ __fingerprint__ + +[[AlternateDirAuthority]] **AlternateDirAuthority** [__nickname__] [**flags**] __ipv4address__:__port__ __fingerprint__ + -[[AlternateBridgeAuthority]] **AlternateBridgeAuthority** [__nickname__] [**flags**] __address__:__port__ __ fingerprint__:: +[[AlternateBridgeAuthority]] **AlternateBridgeAuthority** [__nickname__] [**flags**] __ipv4address__:__port__ __ fingerprint__:: These options behave as DirAuthority, but they replace fewer of the default directory authorities. Using AlternateDirAuthority replaces the default Tor directory authorities, but @@ -522,13 +538,14 @@ GENERAL OPTIONS [[HTTPProxy]] **HTTPProxy** __host__[:__port__]:: Tor will make all its directory requests through this host:port (or host:80 if port is not specified), rather than connecting directly to any directory - servers. + servers. (DEPRECATED: As of 0.3.1.0-alpha you should use HTTPSProxy.) [[HTTPProxyAuthenticator]] **HTTPProxyAuthenticator** __username:password__:: If defined, Tor will use this username:password for Basic HTTP proxy authentication, as in RFC 2617. This is currently the only form of HTTP proxy authentication that Tor supports; feel free to submit a patch if you - want it to support others. + want it to support others. (DEPRECATED: As of 0.3.1.0-alpha you should use + HTTPSProxyAuthenticator.) [[HTTPSProxy]] **HTTPSProxy** __host__[:__port__]:: Tor will make all its OR (SSL) connections through this host:port (or @@ -546,8 +563,10 @@ GENERAL OPTIONS [[Sandbox]] **Sandbox** **0**|**1**:: If set to 1, Tor will run securely through the use of a syscall sandbox. Otherwise the sandbox will be disabled. The option is currently an - experimental feature. Can not be changed while tor is running. - + experimental feature. It only works on Linux-based operating systems, + and only when Tor has been built with the libseccomp library. This option + can not be changed while tor is running. + + When the Sandbox is 1, the following options can not be changed when tor is running: Address @@ -642,6 +661,7 @@ GENERAL OPTIONS is only useful when you have multiple network interfaces, and you want all of Tor's outgoing connections to use a single one. This option may be used twice, once with an IPv4 address and once with an IPv6 address. + IPv6 addresses should be wrapped in square brackets. This setting will be ignored for connections to the loopback addresses (127.0.0.0/8 and ::1). @@ -650,14 +670,17 @@ GENERAL OPTIONS originate from the IP address specified. This option overrides **OutboundBindAddress** for the same IP version. This option may be used twice, once with an IPv4 address and once with an IPv6 - address. This setting will be ignored for connections to the loopback + address. IPv6 addresses should be wrapped in square brackets. + This setting will be ignored for connections to the loopback addresses (127.0.0.0/8 and ::1). [[OutboundBindAddressExit]] **OutboundBindAddressExit** __IP__:: Make all outbound exit connections originate from the IP address specified. This option overrides **OutboundBindAddress** for the same IP version. This option may be used twice, once with an IPv4 - address and once with an IPv6 address. This setting will be ignored + address and once with an IPv6 address. + IPv6 addresses should be wrapped in square brackets. + This setting will be ignored for connections to the loopback addresses (127.0.0.0/8 and ::1). [[PidFile]] **PidFile** __FILE__:: @@ -761,11 +784,57 @@ GENERAL OPTIONS circuits. If the option is set to "default", we obey a parameter in the consensus document. (Default: auto) +[[NoExec]] **NoExec** **0**|**1**:: + If this option is set to 1, then Tor will never launch another + executable, regardless of the settings of PortForwardingHelper, + ClientTransportPlugin, or ServerTransportPlugin. Once this + option has been set to 1, it cannot be set back to 0 without + restarting Tor. (Default: 0) + +[[Schedulers]] **Schedulers** **KIST**|**KISTLite**|**Vanilla**:: + Specify the scheduler type that tor should use. The scheduler is + responsible for moving data around within a Tor process. This is an ordered + list by priority which means that the first value will be tried first and if + unavailable, the second one is tried and so on. It is possible to change + these values at runtime. This option mostly effects relays, and most + operators should leave it set to its default value. + (Default: KIST,KISTLite,Vanilla) + + + The possible scheduler types are: + + + **KIST**: Kernel-Informed Socket Transport. Tor will use TCP information + from the kernel to make informed decisions regarding how much data to send + and when to send it. KIST also handles traffic in batches (see + KISTSchedRunInterval) in order to improve traffic prioritization decisions. + As implemented, KIST will only work on Linux kernel version 2.6.39 or + higher. + + + **KISTLite**: Same as KIST but without kernel support. Tor will use all + the same mechanics as with KIST, including the batching, but its decisions + regarding how much data to send will not be as good. KISTLite will work on + all kernels and operating systems, and the majority of the benefits of KIST + are still realized with KISTLite. + + + **Vanilla**: The scheduler that Tor used before KIST was implemented. It + sends as much data as possible, as soon as possible. Vanilla will work on + all kernels and operating systems. + +[[KISTSchedRunInterval]] **KISTSchedRunInterval** __NUM__ **msec**:: + If KIST or KISTLite is used in the Schedulers option, this controls at which + interval the scheduler tick is. If the value is 0 msec, the value is taken + from the consensus if possible else it will fallback to the default 10 + msec. Maximum possible value is 100 msec. (Default: 0 msec) + +[[KISTSockBufSizeFactor]] **KISTSockBufSizeFactor** __NUM__:: + If KIST is used in Schedulers, this is a multiplier of the per-socket + limit calculation of the KIST algorithm. (Default: 1.0) + CLIENT OPTIONS -------------- The following options are useful only for clients (that is, if -**SocksPort**, **TransPort**, **DNSPort**, or **NATDPort** is non-zero): +**SocksPort**, **HTTPTunnelPort**, **TransPort**, **DNSPort**, or +**NATDPort** is non-zero): [[Bridge]] **Bridge** [__transport__] __IP__:__ORPort__ [__fingerprint__]:: When set along with UseBridges, instructs Tor to use the relay at @@ -902,7 +971,7 @@ The following options are useful only for clients (that is, if The ExcludeNodes option overrides this option: any node listed in both ExitNodes and ExcludeNodes is treated as excluded. + + - The .exit address notation, if enabled via AllowDotExit, overrides + The .exit address notation, if enabled via MapAddress, overrides this option. [[EntryNodes]] **EntryNodes** __node__,__node__,__...__:: @@ -942,7 +1011,7 @@ The following options are useful only for clients (that is, if **FascistFirewall** is set. This option is deprecated; use ReachableAddresses instead. (Default: 80, 443) -[[ReachableAddresses]] **ReachableAddresses** __ADDR__[/__MASK__][:__PORT__]...:: +[[ReachableAddresses]] **ReachableAddresses** __IP__[/__MASK__][:__PORT__]...:: A comma-separated list of IP addresses and ports that your firewall allows you to connect to. The format is as for the addresses in ExitPolicy, except that "accept" is understood unless "reject" is explicitly provided. For @@ -951,14 +1020,15 @@ The following options are useful only for clients (that is, if 99, rejects port 80 connections to net 18, and accepts connections to port 80 otherwise. (Default: \'accept \*:*'.) -[[ReachableDirAddresses]] **ReachableDirAddresses** __ADDR__[/__MASK__][:__PORT__]...:: +[[ReachableDirAddresses]] **ReachableDirAddresses** __IP__[/__MASK__][:__PORT__]...:: Like **ReachableAddresses**, a list of addresses and ports. Tor will obey these restrictions when fetching directory information, using standard HTTP GET requests. If not set explicitly then the value of **ReachableAddresses** is used. If **HTTPProxy** is set then these - connections will go through that proxy. + connections will go through that proxy. (DEPRECATED: This option has + had no effect for some time.) -[[ReachableORAddresses]] **ReachableORAddresses** __ADDR__[/__MASK__][:__PORT__]...:: +[[ReachableORAddresses]] **ReachableORAddresses** __IP__[/__MASK__][:__PORT__]...:: Like **ReachableAddresses**, a list of addresses and ports. Tor will obey these restrictions when connecting to Onion Routers, using TLS/SSL. If not set explicitly then the value of **ReachableAddresses** is used. If @@ -1089,7 +1159,9 @@ The following options are useful only for clients (that is, if Unsupported and force-disabled when using Unix domain sockets.) **IsolateSOCKSAuth**;; Don't share circuits with streams for which different - SOCKS authentication was provided. (On by default; + SOCKS authentication was provided. (For HTTPTunnelPort + connections, this option looks at the Proxy-Authorization and + X-Tor-Stream-Isolation headers. On by default; you can disable it with **NoIsolateSOCKSAuth**.) **IsolateClientProtocol**;; Don't share circuits with streams using a different protocol. @@ -1253,11 +1325,10 @@ The following options are useful only for clients (that is, if default to 1 if the consensus parameter isn't set. (Default: 0) [[NumDirectoryGuards]] **NumDirectoryGuards** __NUM__:: - If UseEntryGuardsAsDirectoryGuards is enabled, we try to make sure we have - at least NUM routers to use as directory guards. If this option is set to - 0, use the value from the guard-n-primary-dir-guards-to-use consensus - parameter, and default to 3 if the consensus parameter isn't set. - (Default: 0) + If UseEntryGuards is set to 1, we try to make sure we have at least NUM + routers to use as directory guards. If this option is set to 0, use the + value from the guard-n-primary-dir-guards-to-use consensus parameter, and + default to 3 if the consensus parameter isn't set. (Default: 0) [[GuardLifetime]] **GuardLifetime** __N__ **days**|**weeks**|**months**:: If nonzero, and UseEntryGuards is set, minimum time to keep a guard before @@ -1279,9 +1350,9 @@ The following options are useful only for clients (that is, if helps to determine whether an application using Tor is possibly leaking DNS requests. (Default: 0) -[[VirtualAddrNetworkIPv4]] **VirtualAddrNetworkIPv4** __Address__/__bits__ + +[[VirtualAddrNetworkIPv4]] **VirtualAddrNetworkIPv4** __IPv4Address__/__bits__ + -[[VirtualAddrNetworkIPv6]] **VirtualAddrNetworkIPv6** [__Address__]/__bits__:: +[[VirtualAddrNetworkIPv6]] **VirtualAddrNetworkIPv6** [__IPv6Address__]/__bits__:: When Tor needs to assign a virtual (unused) address because of a MAPADDRESS command from the controller or the AutomapHostsOnResolve feature, Tor picks an unassigned address from this range. (Defaults: @@ -1304,11 +1375,13 @@ The following options are useful only for clients (that is, if resolved. This helps trap accidental attempts to resolve URLs and so on. (Default: 0) -[[AllowDotExit]] **AllowDotExit** **0**|**1**:: - If enabled, we convert "www.google.com.foo.exit" addresses on the - SocksPort/TransPort/NATDPort into "www.google.com" addresses that exit from - the node "foo". Disabled by default since attacking websites and exit - relays can use it to manipulate your path selection. (Default: 0) +[[HTTPTunnelPort]] **HTTPTunnelPort** \['address':]__port__|**auto** [_isolation flags_]:: + Open this port to listen for proxy connections using the "HTTP CONNECT" + protocol instead of SOCKS. Set this to 0 + 0 if you don't want to allow "HTTP CONNECT" connections. Set the port + to "auto" to have Tor pick a port for you. This directive can be + specified multiple times to bind to multiple addresses/ports. See + SOCKSPort for an explanation of isolation flags. (Default: 0) [[TransPort]] **TransPort** \['address':]__port__|**auto** [_isolation flags_]:: Open this port to listen for transparent proxy connections. Set this to @@ -1379,12 +1452,12 @@ The following options are useful only for clients (that is, if [[ClientDNSRejectInternalAddresses]] **ClientDNSRejectInternalAddresses** **0**|**1**:: If true, Tor does not believe any anonymously retrieved DNS answer that tells it that an address resolves to an internal address (like 127.0.0.1 or - 192.168.0.1). This option prevents certain browser-based attacks; don't - turn it off unless you know what you're doing. (Default: 1) + 192.168.0.1). This option prevents certain browser-based attacks; it + is not allowed to be set on the default network. (Default: 1) [[ClientRejectInternalAddresses]] **ClientRejectInternalAddresses** **0**|**1**:: If true, Tor does not try to fulfill requests to connect to an internal - address (like 127.0.0.1 or 192.168.0.1) __unless a exit node is + address (like 127.0.0.1 or 192.168.0.1) __unless an exit node is specifically requested__ (for example, via a .exit hostname, or a controller request). If true, multicast DNS hostnames for machines on the local network (of the form *.local) are also rejected. (Default: 1) @@ -1521,7 +1594,8 @@ The following options are useful only for clients (that is, if server has both. (Tor also prefers an IPv6 DirPort if IPv4Client is set to 0.) If this option is set to auto, clients prefer IPv4. Other things may influence the choice. This option breaks a tie to the favor of IPv6. - (Default: auto) + (Default: auto) (DEPRECATED: This option has had no effect for some + time.) [[ClientPreferIPv6ORPort]] **ClientPreferIPv6ORPort** **0**|**1**|**auto**:: If this option is set to 1, Tor prefers an OR port with an IPv6 @@ -1550,8 +1624,8 @@ The following options are useful only for clients (that is, if live consensus). Only used by clients fetching from a list of fallback directory mirrors. This schedule is advanced by (potentially concurrent) connection attempts, unlike other schedules, which are advanced by - connection failures. (Default: 10, 11, 3600, 10800, 25200, 54000, - 111600, 262800) + connection failures. (Default: 6, 11, 3600, 10800, 25200, 54000, 111600, + 262800) [[ClientBootstrapConsensusFallbackDownloadSchedule]] **ClientBootstrapConsensusFallbackDownloadSchedule** __N__,__N__,__...__:: Schedule for when clients should download consensuses from fallback @@ -1581,7 +1655,7 @@ The following options are useful only for clients (that is, if [[ClientBootstrapConsensusMaxInProgressTries]] **ClientBootstrapConsensusMaxInProgressTries** __NUM__:: Try this many simultaneous connections to download a consensus before - waiting for one to complete, timeout, or error out. (Default: 4) + waiting for one to complete, timeout, or error out. (Default: 3) SERVER OPTIONS -------------- @@ -1590,12 +1664,13 @@ The following options are useful only for servers (that is, if ORPort is non-zero): [[Address]] **Address** __address__:: - The IP address or fully qualified domain name of this server (e.g. - moria.mit.edu). You can leave this unset, and Tor will guess your IP - address. This IP address is the one used to tell clients and other - servers where to find your Tor server; it doesn't affect the IP that your - Tor client binds to. To bind to a different address, use the - *ListenAddress and OutboundBindAddress options. + The IPv4 address of this server, or a fully qualified domain name of + this server that resolves to an IPv4 address. You can leave this + unset, and Tor will try to guess your IPv4 address. This IPv4 + address is the one used to tell clients and other servers where to + find your Tor server; it doesn't affect the address that your server + binds to. To bind to a different address, use the ORPort and + OutboundBindAddress options. [[AssumeReachable]] **AssumeReachable** **0**|**1**:: This option is used when bootstrapping a new Tor network. If set to 1, @@ -1610,6 +1685,17 @@ is non-zero): server descriptor to the bridge database, rather than to the public directory authorities. +[[BridgeDistribution]] **BridgeDistribution** __string__:: + If set along with BridgeRelay, Tor will include a new line in its + bridge descriptor which indicates to the BridgeDB service how it + would like its bridge address to be given out. Set it to "none" if + you want BridgeDB to avoid distributing your bridge address, or "any" to + let BridgeDB decide. (Default: any) + + + Note: as of Oct 2017, the BridgeDB part of this option is not yet + implemented. Until BridgeDB is updated to obey this option, your + bridge will make this request, but it will not (yet) be obeyed. + [[ContactInfo]] **ContactInfo** __email_address__:: Administrative contact information for this relay or bridge. This line can be used to contact you if your relay or bridge is misconfigured or @@ -1617,7 +1703,12 @@ is non-zero): descriptors containing these lines and that Google indexes them, so spammers might also collect them. You may want to obscure the fact that it's an email address and/or generate a new address for this - purpose. + purpose. + + + + ContactInfo **must** be set to a working address if you run more than one + relay or bridge. (Really, everybody running a relay or bridge should set + it.) + [[ExitRelay]] **ExitRelay** **0**|**1**|**auto**:: Tells Tor whether to run as an exit relay. If Tor is running as a @@ -1745,7 +1836,10 @@ is non-zero): compromise its concealment. + + When listing a node, it's better to list it by fingerprint than by - nickname: fingerprints are more reliable. + nickname: fingerprints are more reliable. + + + + If you run more than one relay, the MyFamily option on each relay + **must** list all other relays, as described above. [[Nickname]] **Nickname** __name__:: Set the server's nickname to \'name'. Nicknames must be between 1 and 19 @@ -1803,11 +1897,13 @@ is non-zero): + If this option is set to 0, Tor will not publish its descriptors to any directories. (This is useful if you're testing - out your server, or if you're using a Tor controller that handles directory - publishing for you.) Otherwise, Tor will publish its descriptors of all - type(s) specified. The default is "1", - which means "if running as a server, publish the - appropriate descriptors to the authorities". + out your server, or if you're using a Tor controller that handles + directory publishing for you.) Otherwise, Tor will publish its + descriptors of all type(s) specified. The default is "1", which + means "if running as a relay or bridge, publish descriptors to the + appropriate authorities". Other possibilities are "v3", meaning + "publish as if you're a relay", and "bridge", meaning "publish as + if you're a bridge". [[ShutdownWaitLength]] **ShutdownWaitLength** __NUM__:: When we get a SIGINT and we're a server, we begin shutting down: @@ -1901,7 +1997,7 @@ is non-zero): correct this. This option only affects name lookups that your server does on behalf of clients. (Default: 1) -[[ServerDNSTestAddresses]] **ServerDNSTestAddresses** __address__,__address__,__...__:: +[[ServerDNSTestAddresses]] **ServerDNSTestAddresses** __hostname__,__hostname__,__...__:: When we're detecting DNS hijacking, make sure that these __valid__ addresses aren't getting redirected. If they are, then our DNS is completely useless, and we'll reset our exit policy to "reject \*:*". This option only affects @@ -2328,7 +2424,7 @@ The following options are used to configure a hidden service. [[HiddenServiceVersion]] **HiddenServiceVersion** __version__,__version__,__...__:: A list of rendezvous service descriptor versions to publish for the hidden - service. Currently, only version 2 is supported. (Default: 2) + service. Currently, versions 2 and 3 are supported. (Default: 2) [[HiddenServiceAuthorizeClient]] **HiddenServiceAuthorizeClient** __auth-type__ __client-name__,__client-name__,__...__:: If configured, the hidden service is accessible for authorized clients @@ -2372,7 +2468,7 @@ The following options are used to configure a hidden service. [[HiddenServiceNumIntroductionPoints]] **HiddenServiceNumIntroductionPoints** __NUM__:: Number of introduction points the hidden service will have. You can't - have more than 10. (Default: 3) + have more than 10 for v2 service and 20 for v3. (Default: 3) [[HiddenServiceSingleHopMode]] **HiddenServiceSingleHopMode** **0**|**1**:: **Experimental - Non Anonymous** Hidden Services on a tor instance in @@ -2547,7 +2643,8 @@ The following options are used for running a testing Tor network. TestingClientDownloadSchedule 0, 0, 5, 10, 15, 20, 30, 60 TestingServerConsensusDownloadSchedule 0, 0, 5, 10, 15, 20, 30, 60 TestingClientConsensusDownloadSchedule 0, 0, 5, 10, 15, 20, 30, 60 - TestingBridgeDownloadSchedule 60, 30, 30, 60 + TestingBridgeDownloadSchedule 10, 30, 60 + TestingBridgeBootstrapDownloadSchedule 0, 0, 5, 10, 15, 20, 30, 60 TestingClientMaxIntervalWithoutRequest 5 seconds TestingDirConnectionMaxStall 30 seconds TestingConsensusMaxDownloadTries 80 @@ -2612,8 +2709,16 @@ The following options are used for running a testing Tor network. 1800, 3600, 3600, 3600, 10800, 21600, 43200) [[TestingBridgeDownloadSchedule]] **TestingBridgeDownloadSchedule** __N__,__N__,__...__:: - Schedule for when clients should download bridge descriptors. Changing this - requires that **TestingTorNetwork** is set. (Default: 3600, 900, 900, 3600) + Schedule for when clients should download each bridge descriptor when they + know that one or more of their configured bridges are running. Changing + this requires that **TestingTorNetwork** is set. (Default: 10800, 25200, + 54000, 111600, 262800) + +[[TestingBridgeBootstrapDownloadSchedule]] **TestingBridgeBootstrapDownloadSchedule** __N__,__N__,__...__:: + Schedule for when clients should download each bridge descriptor when they + have just started, or when they can not contact any of their bridges. + Changing this requires that **TestingTorNetwork** is set. (Default: 0, 30, + 90, 600, 3600, 10800, 25200, 54000, 111600, 262800) [[TestingClientMaxIntervalWithoutRequest]] **TestingClientMaxIntervalWithoutRequest** __N__ **seconds**|**minutes**:: When directory clients have only a few descriptors to request, they batch @@ -2940,6 +3045,16 @@ __DataDirectory__**/hashed-fingerprint**:: Only used by bridges. Holds the hashed fingerprint of the bridge's identity key. (That is, the hash of the hash of the identity key.) +__DataDirectory__**/approved-routers**:: + Only used by authoritative directory servers. This file lists + the status of routers by their identity fingerprint. + Each line lists a status and a fingerprint separated by + whitespace. See your **fingerprint** file in the __DataDirectory__ for an + example line. If the status is **!reject** then descriptors from the + given identity (fingerprint) are rejected by this server. If it is + **!invalid** then descriptors are accepted but marked in the directory as + not valid, that is, not recommended. + __DataDirectory__**/v3-status-votes**:: Only for v3 authoritative directory servers. This file contains status votes from all the authoritative directory servers. @@ -3004,6 +3119,11 @@ __HiddenServiceDirectory__**/hostname**:: The <base32-encoded-fingerprint>.onion domain name for this hidden service. If the hidden service is restricted to authorized clients only, this file also contains authorization data for all clients. + + + Note that clients will ignore any extra subdomains prepended to a hidden + service hostname. So if you have "xyz.onion" as your hostname, you + can tell clients to connect to "www.xyz.onion" or "irc.xyz.onion" + for virtual-hosting purposes. __HiddenServiceDirectory__**/private_key**:: The private key for this hidden service. diff --git a/doc/torrc_format.txt b/doc/torrc_format.txt index b175b9fb15..7a4a92a663 100644 --- a/doc/torrc_format.txt +++ b/doc/torrc_format.txt @@ -18,9 +18,10 @@ does, not what it should do. ; specified in RFC5234. ; A file is interpreted as every Entry in the file, in order. - TorrcFile = *Line + TorrcFile = *Line [ UnterminatedLine ] - Line = BlankLine / Entry + Line = BlankLine LF / Entry LF + UnterminatedLine = BlankLine / Entry BlankLine = *WSP OptComment LF BlankLine =/ *WSP LF @@ -69,6 +70,12 @@ does, not what it should do. ; Anything besides NUL and LF NonLF = %x01-%x09 / %x0b - %xff + ; Note that on windows, we open our configuration files in "text" mode, + ; which causes CRLF pairs to be interpreted as LF. So, on windows: + ; LF = [ %x0d ] %x0a + ; but everywhere else, + LF = %0x0a + OCTDIG = '0' - '7' KC = Any character except an isspace() character or '#' or NUL |