diff options
Diffstat (limited to 'doc/HACKING')
| -rw-r--r-- | doc/HACKING/CodeStructure.md | 129 | ||||
| -rw-r--r-- | doc/HACKING/CodingStandards.md | 27 | ||||
| -rw-r--r-- | doc/HACKING/CodingStandardsRust.md | 2 | ||||
| -rw-r--r-- | doc/HACKING/HelpfulTools.md | 36 | ||||
| -rw-r--r-- | doc/HACKING/Module.md | 30 | ||||
| -rw-r--r-- | doc/HACKING/ReleasingTor.md | 2 | ||||
| -rw-r--r-- | doc/HACKING/Tracing.md | 2 | ||||
| -rw-r--r-- | doc/HACKING/WritingTests.md | 4 |
8 files changed, 194 insertions, 38 deletions
diff --git a/doc/HACKING/CodeStructure.md b/doc/HACKING/CodeStructure.md new file mode 100644 index 0000000000..736d6cd484 --- /dev/null +++ b/doc/HACKING/CodeStructure.md @@ -0,0 +1,129 @@ + +TODO: revise this to talk about how things are, rather than how things +have changed. + +TODO: Make this into good markdown. + + + +For quite a while now, the program "tor" has been built from source +code in just two directories: src/common and src/or. + +This has become more-or-less untenable, for a few reasons -- most +notably of which is that it has led our code to become more +spaghetti-ish than I can endorse with a clean conscience. + +So to fix that, we've gone and done a huge code movement in our git +master branch, which will land in a release once Tor 0.3.5.1-alpha is +out. + +Here's what we did: + + * src/common has been turned into a set of static libraries. These +all live in the "src/lib/*" directories. The dependencies between +these libraries should have no cycles. The libraries are: + + arch -- Headers to handle architectural differences + cc -- headers to handle differences among compilers + compress -- wraps zlib, zstd, lzma + container -- high-level container types + crypt_ops -- Cryptographic operations. Planning to split this into +a higher and lower level library + ctime -- Operations that need to run in constant-time. (Properly, +data-invariant time) + defs -- miscelaneous definitions needed throughout Tor. + encoding -- transforming one data type into another, and various +data types into strings. + err -- lowest-level error handling, in cases where we can't use +the logs because something that the logging system needs has broken. + evloop -- Generic event-loop handling logic + fdio -- Low-level IO wrapper functions for file descriptors. + fs -- Operations on the filesystem + intmath -- low-level integer math and misc bit-twiddling hacks + lock -- low-level locking code + log -- Tor's logging module. This library sits roughly halfway up +the library dependency diagram, since everything it depends on has to +be carefully crafted to *not* log. + malloc -- Low-level wrappers for the platform memory allocation functions. + math -- Higher-level mathematical functions, and floating-point math + memarea -- An arena allocator + meminfo -- Functions for querying the current process's memory +status and resources + net -- Networking compatibility and convenience code + osinfo -- Querying information about the operating system + process -- Launching and querying the status of other processes + sandbox -- Backend for the linux seccomp2 sandbox + smartlist_core -- The lowest-level of the smartlist_t data type. +Separated from the rest of the containers library because the logging +subsystem depends on it. + string -- Compatibility and convenience functions for manipulating +C strings. + term -- Terminal-related functions (currently limited to a getpass +function). + testsupport -- Macros for mocking, unit tests, etc. + thread -- Higher-level thread compatibility code + time -- Higher-level time management code, including format +conversions and monotonic time + tls -- Our wrapper around our TLS library + trace -- Formerly src/trace -- a generic event tracing API + wallclock -- Low-level time code, used by the log module. + + * To ensure that the dependency graph in src/common remains under +control, there is a tool that you can run called "make +check-includes". It verifies that each module in Tor only includes +the headers that it is permitted to include, using a per-directory +".may_include" file. + + * The src/or/or.h header has been split into numerous smaller +headers. Notably, many important structures are now declared in a +header called foo_st.h, where "foo" is the name of the structure. + + * The src/or directory, which had most of Tor's code, had been split +up into several directories. This is still a work in progress: This +code has not itself been refactored, and its dependency graph is still +a tangled web. I hope we'll be working on that over the coming +releases, but it will take a while to do. + + The new top-level source directories are: + + src/core -- Code necessary to actually perform or use onion routing. + src/feature -- Code used only by some onion routing +configurations, or only for a special purpose. + src/app -- Top-level code to run, invoke, and configure the +lower-level code + + The new second-level source directories are: + src/core/crypto -- High-level cryptographic protocols used in Tor + src/core/mainloop -- Tor's event loop, connection-handling, and +traffic-routing code. + src/core/or -- Parts related to handling onion routing itself + src/core/proto -- support for encoding and decoding different +wire protocols + + src/feature/api -- Support for making Tor embeddable + src/feature/client -- Functionality which only Tor clients need + src/feature/control -- Controller implementation + src/feature/dirauth -- Directory authority + src/feature/dircache -- Directory cache + src/feature/dirclient -- Directory client + src/feature/dircommon -- Shared code between the other directory modules + src/feature/hibernate -- Hibernating when Tor is out of bandwidth +or shutting down + src/feature/hs -- v3 onion service implementation + src/feature/hs_common -- shared code between both onion service +implementations + src/feature/nodelist -- storing and accessing the list of relays on +the network. + src/feature/relay -- code that only relay servers and exit servers need. + src/feature/rend -- v2 onion service implementation + src/feature/stats -- statistics and history + + src/app/config -- configuration and state for Tor + src/app/main -- Top-level functions to invoke the rest or Tor. + + * The "tor" executable is now built in src/app/tor rather than src/or/tor. + + * There are more static libraries than before that you need to build +into your application if you want to embed Tor. Rather than +maintaining this list yourself, I recommend that you run "make +show-libs" to have Tor emit a list of what you need to link. diff --git a/doc/HACKING/CodingStandards.md b/doc/HACKING/CodingStandards.md index 3711f70198..4f229348e4 100644 --- a/doc/HACKING/CodingStandards.md +++ b/doc/HACKING/CodingStandards.md @@ -172,7 +172,6 @@ deviations from our C whitespace style. Generally, we use: - Unix-style line endings - K&R-style indentation - No space before newlines - - A blank line at the end of each file - Never more than one blank line in a row - Always spaces, never tabs - No more than 79-columns per line. @@ -185,6 +184,9 @@ deviations from our C whitespace style. Generally, we use: `puts (x)`. - Function declarations at the start of the line. +If you use an editor that has plugins for editorconfig.org, the file +`.editorconfig` will help you to conform this coding style. + We try hard to build without warnings everywhere. In particular, if you're using gcc, you should invoke the configure script with the option `--enable-fatal-warnings`. This will tell the compiler @@ -198,8 +200,8 @@ We have some wrapper functions like `tor_malloc`, `tor_free`, `tor_strdup`, and always succeed or exit.) You can get a full list of the compatibility functions that Tor provides by -looking through `src/common/util*.h` and `src/common/compat*.h`. You can see the -available containers in `src/common/containers*.h`. You should probably +looking through `src/lib/*/*.h`. You can see the +available containers in `src/lib/containers/*.h`. You should probably familiarize yourself with these modules before you write too much code, or else you'll wind up reinventing the wheel. @@ -212,6 +214,24 @@ We don't call `memcmp()` directly. Use `fast_memeq()`, `fast_memneq()`, Also see a longer list of functions to avoid in: https://people.torproject.org/~nickm/tor-auto/internal/this-not-that.html +What code can use what other code? +---------------------------------- + +We're trying to simplify Tor's structure over time. In the long run, we want +Tor to be structured as a set of modules with *no circular dependencies*. + +This property is currently provided by the modules in src/lib, but not +throughout the rest of Tor. In general, higher-level libraries may use +lower-level libraries, but never the reverse. + +To prevent new circular dependencies from landing, we have a tool that +you can invoke with `make check-includes`, and which is run +automatically as part of `make check`. This tool will verify that, for +every source directory with a `.may_include` file, no local headers are +included except those specifically permitted by the `.may_include` file. +When editing one of these files, please make sure that you are not +introducing any cycles into Tor's dependency graph. + Floating point math is hard --------------------------- @@ -434,4 +454,3 @@ the functions that call your function rely on it doing something, then your function should mention that it does that something in the documentation. If you rely on a function doing something beyond what is in its documentation, then you should watch out, or it might do something else later. - diff --git a/doc/HACKING/CodingStandardsRust.md b/doc/HACKING/CodingStandardsRust.md index d9496c08f7..fc562816db 100644 --- a/doc/HACKING/CodingStandardsRust.md +++ b/doc/HACKING/CodingStandardsRust.md @@ -104,7 +104,7 @@ repo. Documentation --------------- -You MUST include `#[deny(missing_docs)]` in your crate. +You MUST include `#![deny(missing_docs)]` in your crate. For function/method comments, you SHOULD include a one-sentence, "first person" description of function behaviour (see requirements for documentation as diff --git a/doc/HACKING/HelpfulTools.md b/doc/HACKING/HelpfulTools.md index a0795076e0..d499238526 100644 --- a/doc/HACKING/HelpfulTools.md +++ b/doc/HACKING/HelpfulTools.md @@ -4,9 +4,16 @@ Useful tools These aren't strictly necessary for hacking on Tor, but they can help track down bugs. -Travis CI ---------- -It's CI. Looks like this: https://travis-ci.org/torproject/tor. +Travis/Appveyor CI +------------------ +It's CI. + +Looks like this: +* https://travis-ci.org/torproject/tor +* https://ci.appveyor.com/project/torproject/tor + +Travis builds and runs tests on Linux, and eventually macOS (#24629). +Appveyor builds and runs tests on Windows (using Windows Services for Linux). Runs automatically on Pull Requests sent to torproject/tor. You can set it up for your fork to build commits outside of PRs too: @@ -16,6 +23,8 @@ for your fork to build commits outside of PRs too: https://help.github.com/articles/fork-a-repo/ 3. follow https://docs.travis-ci.com/user/getting-started/#To-get-started-with-Travis-CI. skip steps involving `.travis.yml` (we already have one). +4. go to https://ci.appveyor.com/login , log in with github, and select + "NEW PROJECT" Builds should show up on the web at travis-ci.com and on IRC at #tor-ci on OFTC. If they don't, ask #tor-dev (also on OFTC). @@ -23,22 +32,21 @@ OFTC. If they don't, ask #tor-dev (also on OFTC). Jenkins ------- - https://jenkins.torproject.org +It's CI/builders. Looks like this: https://jenkins.torproject.org -Dmalloc -------- +Runs automatically on commits merged to git.torproject.org. We CI the +master branch and all supported tor versions. We also build nightly debian +packages from master. -The dmalloc library will keep track of memory allocation, so you can find out -if we're leaking memory, doing any double-frees, or so on. +Builds Linux and Windows cross-compilation. Runs Linux tests. - dmalloc -l -/dmalloc.log - (run the commands it tells you) - ./configure --with-dmalloc +Builds should show up on the web at jenkins.torproject.org and on IRC at +#tor-bots on OFTC. If they don't, ask #tor-dev (also on OFTC). Valgrind -------- - valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/or/tor + valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/app/tor (Note that if you get a zillion openssl warnings, you will also need to pass `--undef-value-errors=no` to valgrind, or rebuild your openssl @@ -242,10 +250,10 @@ Beforehand, install google-perftools. Now you can run Tor with profiling enabled, and use the pprof utility to look at performance! See the gperftools manual for more info, but basically: -2. Run `env CPUPROFILE=/tmp/profile src/or/tor -f <path/torrc>`. The profile file +2. Run `env CPUPROFILE=/tmp/profile src/app/tor -f <path/torrc>`. The profile file is not written to until Tor finishes execuction. -3. Run `pprof src/or/tor /tm/profile` to start the REPL. +3. Run `pprof src/app/tor /tm/profile` to start the REPL. Generating and analyzing a callgraph ------------------------------------ diff --git a/doc/HACKING/Module.md b/doc/HACKING/Module.md index 1028a029d9..9cf36090b4 100644 --- a/doc/HACKING/Module.md +++ b/doc/HACKING/Module.md @@ -12,9 +12,9 @@ Currently, there is only one module: - Directory Authority subsystem (dirauth) -It is located in its own directory in `src/or/dirauth/`. To disable it, one -need to pass `--disable-module-dirauth` at configure time. All modules are -currently enabled by default. +It is located in its own directory in `src/feature/dirauth/`. To disable it, +one need to pass `--disable-module-dirauth` at configure time. All modules +are currently enabled by default. ## Build System ## @@ -32,10 +32,10 @@ The changes to the build system are pretty straightforward. the C code to conditionally compile things for your module. And the `BUILD_MODULE_<name>` is also defined for automake files (e.g: include.am). -3. In the `src/or/include.am` file, locate the `MODULE_DIRAUTH_SOURCES` value. - You need to create your own `_SOURCES` variable for your module and then - conditionally add the it to `LIBTOR_A_SOURCES` if you should build the - module. +3. In the `src/core/include.am` file, locate the `MODULE_DIRAUTH_SOURCES` + value. You need to create your own `_SOURCES` variable for your module + and then conditionally add the it to `LIBTOR_A_SOURCES` if you should + build the module. It is then **very** important to add your SOURCES variable to `src_or_libtor_testing_a_SOURCES` so the tests can build it. @@ -51,7 +51,7 @@ always build everything in order to tests everything. ## Coding ## As mentioned above, a module must be isolated in its own directory (name of -the module) in `src/or/`. +the module) in `src/feature/`. There are couples of "rules" you want to follow: @@ -87,22 +87,22 @@ There are couples of "rules" you want to follow: making the code much more difficult to follow/understand. * It is possible that you end up with code that needs to be used by the rest - of the code base but is still part of your module. As a good example, if you - look at `src/or/shared_random_client.c`: it contains code needed by the hidden - service subsystem but mainly related to the shared random subsystem very - specific to the dirauth module. + of the code base but is still part of your module. As a good example, if + you look at `src/feature/shared_random_client.c`: it contains code needed + by the hidden service subsystem but mainly related to the shared random + subsystem very specific to the dirauth module. This is fine but try to keep it as lean as possible and never use the same filename as the one in the module. For example, this is a bad idea and should never be done: - - `src/or/shared_random.c` - - `src/or/dirauth/shared_random.c` + - `src/feature/dirclient/shared_random.c` + - `src/feature/dirauth/shared_random.c` * When you include headers from the module, **always** use the full module path in your statement. Example: - `#include "dirauth/dirvote.h"` + `#include "feature/dirauth/dirvote.h"` The main reason is that we do **not** add the module include path by default so it needs to be specified. But also, it helps our human brain understand diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index e70416c354..55a40fc89b 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -7,7 +7,7 @@ new Tor release: === 0. Preliminaries -1. Get at least three of weasel/arma/Sebastian/Sina to put the new +1. Get at least two of weasel/arma/Sebastian to put the new version number in their approved versions list. Give them a few days to do this if you can. diff --git a/doc/HACKING/Tracing.md b/doc/HACKING/Tracing.md index 349aade23a..24fa761310 100644 --- a/doc/HACKING/Tracing.md +++ b/doc/HACKING/Tracing.md @@ -69,7 +69,7 @@ configure option: ## Instrument Tor ## This is pretty easy. Let's say you want to add a trace event in -`src/or/rendcache.c`, you only have to add this include statement: +`src/feature/rend/rendcache.c`, you only have to add this include statement: #include "trace/events.h" diff --git a/doc/HACKING/WritingTests.md b/doc/HACKING/WritingTests.md index cc393494ec..05de8e0be8 100644 --- a/doc/HACKING/WritingTests.md +++ b/doc/HACKING/WritingTests.md @@ -7,8 +7,8 @@ keep from introducing bugs. The major ones are: 1. Unit tests written in C and shipped with the Tor distribution. - 2. Integration tests written in Python and shipped with the Tor - distribution. + 2. Integration tests written in Python 2 (>= 2.7) or Python 3 + (>= 3.1) and shipped with the Tor distribution. 3. Integration tests written in Python and shipped with the Stem library. Some of these use the Tor controller protocol. |