diff options
Diffstat (limited to 'doc/HACKING')
| -rw-r--r-- | doc/HACKING/CodingStandards.md | 40 | ||||
| -rw-r--r-- | doc/HACKING/CodingStandardsRust.md | 44 | ||||
| -rw-r--r-- | doc/HACKING/GettingStartedRust.md | 120 | ||||
| -rw-r--r-- | doc/HACKING/ReleasingTor.md | 21 | ||||
| -rw-r--r-- | doc/HACKING/Tracing.md | 2 | ||||
| -rw-r--r-- | doc/HACKING/android/Simpleperf.md | 98 |
6 files changed, 270 insertions, 55 deletions
diff --git a/doc/HACKING/CodingStandards.md b/doc/HACKING/CodingStandards.md index dd21d6fd2c..79a6a9f0ce 100644 --- a/doc/HACKING/CodingStandards.md +++ b/doc/HACKING/CodingStandards.md @@ -346,6 +346,46 @@ macro, as in: if (BUG(ptr == NULL)) return -1; +Allocator conventions +--------------------- + +By convention, any tor type with a name like `abc_t` should be allocated +by a function named `abc_new()`. This function should never return +NULL. + +Also, a type named `abc_t` should be freed by a function named `abc_free_()`. +Don't call this `abc_free_()` function directly -- instead, wrap it in a +macro called `abc_free()`, using the `FREE_AND_NULL` macro: + + void abc_free_(abc_t *obj); + #define abc_free(obj) FREE_AND_NULL(abc_t, abc_free_, (obj)) + +This macro will free the underlying `abc_t` object, and will also set +the object pointer to NULL. + +You should define all `abc_free_()` functions to accept NULL inputs: + + void + abc_free_(abc_t *obj) + { + if (!obj) + return; + tor_free(obj->name); + thing_free(obj->thing); + tor_free(obj); + } + +If you need a free function that takes a `void *` argument (for example, +to use it as a function callback), define it with a name like +`abc_free_void()`: + + static void + abc_free_void_(void *obj) + { + abc_free_(obj); + } + + Doxygen comment conventions --------------------------- diff --git a/doc/HACKING/CodingStandardsRust.md b/doc/HACKING/CodingStandardsRust.md index d0b17c1604..7c6405e624 100644 --- a/doc/HACKING/CodingStandardsRust.md +++ b/doc/HACKING/CodingStandardsRust.md @@ -54,13 +54,53 @@ If you have any external modules as dependencies (e.g. `extern crate libc;`), you MUST declare them in your crate's `lib.rs` and NOT in any other module. - Dependencies --------------- + Dependencies and versions +--------------------------- In general, we use modules from only the Rust standard library whenever possible. We will review including external crates on a case-by-case basis. +If a crate only contains traits meant for compatibility between Rust +crates, such as [the digest crate](https://crates.io/crates/digest) or +[the failure crate](https://crates.io/crates/failure), it is very likely +permissible to add it as a dependency. However, a brief review should +be conducted as to the usefulness of implementing external traits +(i.e. how widespread is the usage, how many other crates either +implement the traits or have trait bounds based upon them), as well as +the stability of the traits (i.e. if the trait is going to change, we'll +potentially have to re-do all our implementations of it). + +For large external libraries, especially which implement features which +would be labour-intensive to reproduce/maintain ourselves, such as +cryptographic or mathematical/statistics libraries, only crates which +have stabilised to 1.0.0 should be considered, however, again, we may +make exceptions on a case-by-case basis. + +Currently, Tor requires that you use the latest stable Rust version. At +some point in the future, we will freeze on a given stable Rust version, +to ensure backward compatibility with stable distributions that ship it. + + Updating/Adding Dependencies +------------------------------ + +To add/remove/update dependencies, first add your dependencies, +exactly specifying their versions, into the appropriate *crate-level* +`Cargo.toml` in `src/rust/` (i.e. *not* `/src/rust/Cargo.toml`, but +instead the one for your crate). Also, investigate whether your +dependency has any optional dependencies which are unnecessary but are +enabled by default. If so, you'll likely be able to enable/disable +them via some feature, e.g.: + +```toml +[dependencies] +foo = { version = "1.0.0", default-features = false } +``` + +Next, run `/scripts/maint/updateRustDependencies.sh`. Then, go into +`src/ext/rust` and commit the changes to the `tor-rust-dependencies` +repo. + Documentation --------------- diff --git a/doc/HACKING/GettingStartedRust.md b/doc/HACKING/GettingStartedRust.md index a5253b46a6..a533ba8a27 100644 --- a/doc/HACKING/GettingStartedRust.md +++ b/doc/HACKING/GettingStartedRust.md @@ -9,17 +9,19 @@ Please read or review our documentation on Rust coding standards (`.../doc/HACKING/CodingStandardsRust.md`) before doing anything. Please also read -[the Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html). We aim -to follow the good example set by the Rust community and be excellent to one -another. Let's be careful with each other, so we can be memory-safe together! - -Next, please contact us before rewriting anything! Rust in Tor is still an -experiment. It is an experiment that we very much want to see succeed, so we're -going slowly and carefully. For the moment, it's also a completely -volunteer-driven effort: while many, if not most, of us are paid to work on Tor, -we are not yet funded to write Rust code for Tor. Please be patient with the -other people who are working on getting more Rust code into Tor, because they -are graciously donating their free time to contribute to this effort. +[the Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html). We +aim to follow the good example set by the Rust community and be +excellent to one another. Let's be careful with each other, so we can +be memory-safe together! + +Next, please contact us before rewriting anything! Rust in Tor is still +an experiment. It is an experiment that we very much want to see +succeed, so we're going slowly and carefully. For the moment, it's also +a completely volunteer-driven effort: while many, if not most, of us are +paid to work on Tor, we are not yet funded to write Rust code for Tor. +Please be patient with the other people who are working on getting more +Rust code into Tor, because they are graciously donating their free time +to contribute to this effort. Resources for learning Rust ----------------------------- @@ -33,14 +35,15 @@ Rust immediately, without waiting for anything to install, there is **Advanced resources** -If you're interested in playing with various Rust compilers and viewing a very -nicely displayed output of the generated assembly, there is +If you're interested in playing with various Rust compilers and viewing +a very nicely displayed output of the generated assembly, there is [the Godbolt compiler explorer](https://rust.godbolt.org/) For learning how to write unsafe Rust, read [The Rustonomicon](https://doc.rust-lang.org/nomicon/). -For learning everything you ever wanted to know about Rust macros, there is +For learning everything you ever wanted to know about Rust macros, there +is [The Little Book of Rust Macros](https://danielkeep.github.io/tlborm/book/index.html). For learning more about FFI and Rust, see Jake Goulding's @@ -49,10 +52,10 @@ For learning more about FFI and Rust, see Jake Goulding's Compiling Tor with Rust enabled --------------------------------- -You will need to run the `configure` script with the `--enable-rust` flag to -explicitly build with Rust. Additionally, you will need to specify where to -fetch Rust dependencies, as we allow for either fetching dependencies from Cargo -or specifying a local directory. +You will need to run the `configure` script with the `--enable-rust` +flag to explicitly build with Rust. Additionally, you will need to +specify where to fetch Rust dependencies, as we allow for either +fetching dependencies from Cargo or specifying a local directory. **Fetch dependencies from Cargo** @@ -60,48 +63,48 @@ or specifying a local directory. **Using a local dependency cache** -**NOTE**: local dependency caches which were not *originally* created via - `--enable-cargo-online-mode` are broken. See https://bugs.torproject.org/22907 +You'll need the following Rust dependencies (as of this writing): -To specify a local directory: + libc==0.2.39 - RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust +We vendor our Rust dependencies in a separate repo using +[cargo-vendor](https://github.com/alexcrichton/cargo-vendor). To use +them, do: -(Note that RUST_DEPENDENCIES must be the full path to the directory; it cannot -be relative.) + git submodule init + git submodule update -You'll need the following Rust dependencies (as of this writing): +To specify the local directory containing the dependencies, (assuming +you are in the top level of the repository) configure tor with: - libc==0.2.22 + TOR_RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust -To get them, do: +(Note that TOR_RUST_DEPENDENCIES must be the full path to the directory; it +cannot be relative.) - mkdir path_to_dependencies_directory - cd path_to_dependencies_directory - git clone https://github.com/rust-lang/libc - cd libc - git checkout 0.2.22 - cargo package - cd .. - ln -s libc/target/package/libc-0.2.22 libc-0.2.22 +Assuming you used the above `git submodule` commands and you're in the +topmost directory of the repository, this would be: + + TOR_RUST_DEPENDENCIES=`pwd`/src/ext/rust/crates ./configure --enable-rust Identifying which modules to rewrite ====================================== -The places in the Tor codebase that are good candidates for porting to Rust are: +The places in the Tor codebase that are good candidates for porting to +Rust are: 1. loosely coupled to other Tor submodules, 2. have high test coverage, and 3. would benefit from being implemented in a memory safe language. -Help in either identifying places such as this, or working to improve existing -areas of the C codebase by adding regression tests and simplifying dependencies, -would be really helpful. +Help in either identifying places such as this, or working to improve +existing areas of the C codebase by adding regression tests and +simplifying dependencies, would be really helpful. Furthermore, as submodules in C are implemented in Rust, this is a good -opportunity to refactor, add more tests, and split modules into smaller areas of -responsibility. +opportunity to refactor, add more tests, and split modules into smaller +areas of responsibility. A good first step is to build a module-level callgraph to understand how interconnected your target module is. @@ -119,11 +122,22 @@ the module calls. Modules which call fewer other modules are better targets. Strive to change the C API as little as possible. -We are currently targetting Rust nightly, *for now*. We expect this to change -moving forward, as we understand more about which nightly features we need. It -is on our TODO list to try to cultivate good standing with various distro -maintainers of `rustc` and `cargo`, in order to ensure that whatever version we -solidify on is readily available. +We are currently targeting Rust nightly, *for now*. We expect this to +change moving forward, as we understand more about which nightly +features we need. It is on our TODO list to try to cultivate good +standing with various distro maintainers of `rustc` and `cargo`, in +order to ensure that whatever version we solidify on is readily +available. + +If parts of your Rust code needs to stay in sync with C code (such as +handling enums across the FFI boundary), annonotate these places in a +comment structured as follows: + + /// C_RUST_COUPLED: <path_to_file> `<name_of_c_object>` + +Where <name_of_c_object> can be an enum, struct, constant, etc. Then, +do the same in the C code, to note that rust will need to be changed +when the C does. Adding your Rust module to Tor's build system ----------------------------------------------- @@ -132,16 +146,22 @@ solidify on is readily available. in the `.../tor/src/rust/` directory. 1. Add your crate to `.../tor/src/rust/Cargo.toml`, in the `[workspace.members]` section. -2. Append your crate's static library to the `rust_ldadd` definition - (underneath `if USE_RUST`) in `.../tor/Makefile.am`. +2. Add your crate's files to src/rust/include.am + +If your crate should be available to C (rather than just being included as a +dependency of other Rust modules): +0. Declare the crate as a dependency of tor_rust in + `src/rust/tor_util/Cargo.toml` and include it in + `src/rust/tor_rust/lib.rs` How to test your Rust code ---------------------------- Everything should be tested full stop. Even non-public functionality. -Be sure to edit `.../tor/src/test/test_rust.sh` to add the name of your crate to -the `crates` variable! This will ensure that `cargo test` is run on your crate. +Be sure to edit `.../tor/src/test/test_rust.sh` to add the name of your +crate to the `crates` variable! This will ensure that `cargo test` is +run on your crate. Configure Tor's build system to build with Rust enabled: diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index 62029b44f0..6c8fa1331f 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -14,6 +14,9 @@ new Tor release: 2. If this is going to be an important security release, give the packagers some advance warning: See this list of packagers in IV.3 below. +3. Given the release date for Tor, ask the TB team about the likely release + date of a TB that contains it. See note below in "commit, upload, + announce". === I. Make sure it works @@ -22,6 +25,7 @@ new Tor release: resolve those. As applicable, merge the `maint-X` branch into the `release-X` branch. + But you've been doing that all along, right? 2. Are all of the jenkins builders happy? See jenkins.torproject.org. @@ -134,6 +138,9 @@ new Tor release: either `make`, or `perl scripts/maint/updateVersions.pl`, depending on your version.) + When you merge the maint branch forward to the next maint branch, or into + master, merge it with "-s ours" to avoid a needless version bump. + 2. Make distcheck, put the tarball up in somewhere (how about your homedir on your homedir on people.torproject.org?) , and tell `#tor` about it. Wait a while to see if anybody has problems building it. @@ -147,6 +154,9 @@ new Tor release: git tag -u <keyid> tor-0.3.x.y-status git push origin tag tor-0.3.x.y-status + (You must do this before you update the website: it relies on finding + the version by tag.) + 2. scp the tarball and its sig to the dist website, i.e. `/srv/dist-master.torproject.org/htdocs/` on dist-master. When you want it to go live, you run "static-update-component dist.torproject.org" @@ -171,7 +181,10 @@ new Tor release: - {mike} at tig dot as - {tails-rm} at boum dot org - {simon} at sdeziel.info - - {yuri} at rawbw.com + - {yuri} at freebsd.org + - {mh+tor} at scrit.ch + + Also, email tor-packagers@lists.torproject.org. 4. Add the version number to Trac. To do this, go to Trac, log in, select "Admin" near the top of the screen, then select "Versions" from @@ -180,7 +193,11 @@ new Tor release: 0.2.2.23-alpha" (or whatever the version is), and we select the date as the date in the ChangeLog. -5. Mail the release blurb and ChangeLog to tor-talk (development release) or +5. Double-check: did the version get recommended in the consensus yet? Is + the website updated? If not, don't announce until they have the + up-to-date versions, or people will get confused. + +6. Mail the release blurb and ChangeLog to tor-talk (development release) or tor-announce (stable). Post the changelog on the blog as well. You can generate a diff --git a/doc/HACKING/Tracing.md b/doc/HACKING/Tracing.md index a5fb5165e2..349aade23a 100644 --- a/doc/HACKING/Tracing.md +++ b/doc/HACKING/Tracing.md @@ -6,7 +6,7 @@ tracing framework. ## Basics ### -Event tracing is seperated in two concepts, trace events and a tracer. The +Event tracing is separated in two concepts, trace events and a tracer. The tracing subsystem can be found in `src/trace`. The `events.h` header file is the main file that maps the different tracers to trace events. diff --git a/doc/HACKING/android/Simpleperf.md b/doc/HACKING/android/Simpleperf.md new file mode 100644 index 0000000000..25f39a3d23 --- /dev/null +++ b/doc/HACKING/android/Simpleperf.md @@ -0,0 +1,98 @@ +# Using `simpleperf` to collect CPU profiling on Android + +This document describes how you can use Android's `simpleperf` +command-line tool to get CPU profiling information from Tor via the +Orbot application. The tool is particularly useful for Tor development +because it is able to profile native applications on the platform +whereas a lot of the normal tooling for the Android platform is only +able to collect information from Java-based applications. + +## Prerequisites + +Before using `simpleperf` there is a couple of steps that must be +followed. You should make sure you have both a recent installation of +the Android Software Development Kit (SDK) and Native Development Kit +(NDK) installed. These can be found on the Android Developers website. + +1. Follow the build instructions from the `BUILD` file in the Orbot + repository and build an Orbot APK (Android Package) file with + debugging enabled. Make sure that when you build the native content of + the Orbot application that you run the `make -C external` command with + an additional `DEBUG=1` as parameter to ensure that the Orbot build + process does not strip the debug symbols from the Tor binary. + +2. (Optional) Uninstall and clean-up your old Orbot installation that + is most likely downloaded from Google's Play Store or via fdroid: + + $ adb shell pm clear org.torproject.android + $ adb uninstall org.torproject.android + +3. Install the Android Package you generated in step 1: + + $ adb install /path/to/your/app-fullperm-debug.apk + +4. Check on your device that the newly installed Orbot actually works + and behaves in the way you expect it to. + +## Profiling using `simpleperf` + +The `simpleperf` tool can be found in the `simpleperf/` directory in +the directory where you installed the Android NDK to. In this +directory there is a set of Python files that will help you deploy the +tool to a device and collect the measurement data such that you can +analyze the results on your computer rather than on your phone. + +1. Change directory to the location of the `simpleperf` directory. +2. Open the `app_profiler.config` file and change + `app_package_name` to `org.torproject.android`, `apk_file_path` to + the path of your Orbot Android Package (APK file). +3. Optionally change the duration parameter in the `record_options` + variable in `app_profiler.config` to the duration which you would like + to collect samples in. The value is specified in seconds. +4. Run the app profiler using `python app_profiler.py`. This helper + script will push the `simpleperf` tool to your device, start the + profiler, and once it has completed copy the generated `perf.data` + file over to your computer with the results. + +### Analyzing the results + +You can inspect your resulting `perf.data` file via a simple GUI +program `python report.py` or via the command-line tool `simpleperf +report`. I've found the GUI tool to be easier to navigate around with +than the command-line tool. + +The `-g` option can be passed to the command line `simpleperf report` +tool allows you to see the call graph of functions and how much time +was spend on the call. + +## Tips & Tricks + +- When you have installed Orbot the first time, you will notice that + if you get a shell on the Android device that there is no Tor binary + available. This is because Orbot unpacks the Tor binary first time it + is executed and places it under the `app_bin/` directory on the + device. + + To access binaries, `torrc` files, and other useful information on + the device do the following: + + $ adb shell + (device):/ $ run-as org.torproject.android + (device):/data/data/org.torproject.android $ ls + app_bin app_data cache databases files lib shared_prefs + + Descriptors, control authentication cookie, state, and other files can be + found in the `app_data` directory. The `torrc` can be found in the `app_bin/` + directory. + +- You can enable logging in Tor via the syslog (or android) log + mechanism with: + + $ adb shell + (device):/ $ run-as org.torproject.android + (device):/data/data/org.torproject.android $ echo -e "\nLog info syslog" >> app_bin/torrc + + Start Tor the normal way via Orbot and collect the logs from your computer using + + $ adb logcat + |