aboutsummaryrefslogtreecommitdiff
path: root/doc/HACKING
diff options
context:
space:
mode:
Diffstat (limited to 'doc/HACKING')
-rw-r--r--doc/HACKING/CodingStandards.md4
-rw-r--r--doc/HACKING/Fuzzing.md123
-rw-r--r--doc/HACKING/HelpfulTools.md55
-rw-r--r--doc/HACKING/ReleasingTor.md18
-rw-r--r--doc/HACKING/WritingTests.md2
5 files changed, 194 insertions, 8 deletions
diff --git a/doc/HACKING/CodingStandards.md b/doc/HACKING/CodingStandards.md
index f1c65850a4..01212a9919 100644
--- a/doc/HACKING/CodingStandards.md
+++ b/doc/HACKING/CodingStandards.md
@@ -93,6 +93,10 @@ What needs a changes file?
rewrites. Anything about which somebody might plausibly wonder "when
did that happen, and/or why did we do that" 6 months down the line.
+What does not need a changes file?
+
+ * Bugfixes for code that hasn't shipped in any released version of Tor
+
Why use changes files instead of Git commit messages?
* Git commit messages are written for developers, not users, and they
diff --git a/doc/HACKING/Fuzzing.md b/doc/HACKING/Fuzzing.md
new file mode 100644
index 0000000000..2039d6a4c0
--- /dev/null
+++ b/doc/HACKING/Fuzzing.md
@@ -0,0 +1,123 @@
+= Fuzzing Tor
+
+== The simple version (no fuzzing, only tests)
+
+Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place
+where you checked it out.
+
+To run the fuzzing test cases in a deterministic fashion, use:
+ make test-fuzz-corpora
+
+This won't actually fuzz Tor! It will just run all the fuzz binaries
+on our existing set of testcases for the fuzzer.
+
+
+== Different kinds of fuzzing
+
+Right now we support three different kinds of fuzzer.
+
+First, there's American Fuzzy Lop (AFL), a fuzzer that works by forking
+a target binary and passing it lots of different inputs on stdin. It's the
+trickiest one to set up, so I'll be describing it more below.
+
+Second, there's libFuzzer, a llvm-based fuzzer that you link in as a library,
+and it runs a target function over and over. To use this one, you'll need to
+have a reasonably recent clang and libfuzzer installed. At that point, you
+just build with --enable-expensive-hardening and --enable-libfuzzer. That
+will produce a set of binaries in src/test/fuzz/lf-fuzz-* . These programs
+take as input a series of directories full of fuzzing examples. For more
+information on libfuzzer, see http://llvm.org/docs/LibFuzzer.html
+
+Third, there's Google's OSS-Fuzz infrastructure, which expects to get all of
+its. For more on this, see https://github.com/google/oss-fuzz and the
+projects/tor subdirectory. You'll need to mess around with Docker a bit to
+test this one out; it's meant to run on Google's infrastructure.
+
+In all cases, you'll need some starting examples to give the fuzzer when it
+starts out. There's a set in the "fuzzing-corpora" git repository. Try
+setting TOR_FUZZ_CORPORA to point to a checkout of that repository
+
+== Writing Tor fuzzers
+
+A tor fuzzing harness should have:
+* a fuzz_init() function to set up any necessary global state.
+* a fuzz_main() function to receive input and pass it to a parser.
+* a fuzz_cleanup() function to clear global state.
+
+Most fuzzing frameworks will produce many invalid inputs - a tor fuzzing
+harness should rejecting invalid inputs without crashing or behaving badly.
+
+But the fuzzing harness should crash if tor fails an assertion, triggers a
+bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
+"interesting" cases.
+
+
+== Guided Fuzzing with AFL
+
+There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so
+its integrity can't be verified. That said, you really shouldn't fuzz on a
+machine you care about, anyway.
+
+To Build:
+ Get AFL from http://lcamtuf.coredump.cx/afl/ and unpack it
+ cd afl
+ make
+ cd ../tor
+ PATH=$PATH:../afl/ CC="../afl/afl-gcc" ./configure --enable-expensive-hardening
+ AFL_HARDEN=1 make clean fuzzers
+
+To Find The ASAN Memory Limit: (64-bit only)
+
+On 64-bit platforms, afl needs to know how much memory ASAN uses,
+because ASAN tends to allocate a ridiculous amount of virtual memory,
+and then not actually use it.
+
+Read afl/docs/notes_for_asan.txt for more details.
+
+ Download recidivm from http://jwilk.net/software/recidivm
+ Download the signature
+ Check the signature
+ tar xvzf recidivm*.tar.gz
+ cd recidivm*
+ make
+ /path/to/recidivm -v src/test/fuzz/fuzz-http
+ Use the final "ok" figure as the input to -m when calling afl-fuzz
+ (Normally, recidivm would output a figure automatically, but in some cases,
+ the fuzzing harness will hang when the memory limit is too small.)
+
+You could also just say "none" instead of the memory limit below, if you
+don't care about memory limits.
+
+
+To Run:
+ mkdir -p src/test/fuzz/fuzz_http_findings
+ ../afl/afl-fuzz -i ${TOR_FUZZ_CORPORA}/http -o src/test/fuzz/fuzz_http_findings -m <asan-memory-limit> -- src/test/fuzz/fuzz-http
+
+
+AFL has a multi-core mode, check the documentation for details.
+You might find the included fuzz-multi.sh script useful for this.
+
+macOS (OS X) requires slightly more preparation, including:
+* using afl-clang (or afl-clang-fast from the llvm directory)
+* disabling external crash reporting (AFL will guide you through this step)
+
+== Triaging Issues
+
+Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code.
+
+Hangs might be interesting, but they might also be spurious machine slowdowns.
+Check if a hang is reproducible before reporting it. Sometimes, processing
+valid inputs may take a second or so, particularly with the fuzzer and
+sanitizers enabled.
+
+To see what fuzz-http is doing with a test case, call it like this:
+ src/test/fuzz/fuzz-http --debug < /path/to/test.case
+
+(Logging is disabled while fuzzing to increase fuzzing speed.)
+
+== Reporting Issues
+
+Please report any issues discovered using the process in Tor's security issue
+policy:
+
+https://trac.torproject.org/projects/tor/wiki/org/meetings/2016SummerDevMeeting/Notes/SecurityIssuePolicy
diff --git a/doc/HACKING/HelpfulTools.md b/doc/HACKING/HelpfulTools.md
index a7f36e6c7e..67481ace43 100644
--- a/doc/HACKING/HelpfulTools.md
+++ b/doc/HACKING/HelpfulTools.md
@@ -142,6 +142,12 @@ run `make test-network`.
We also have scripts to run integration tests using Stem. To try them, set
`STEM_SOURCE_DIR` to your Stem source directory, and run `test-stem`.
+Profiling Tor
+-------------
+
+Ongoing notes about Tor profiling can be found at
+https://pad.riseup.net/p/profiling-tor
+
Profiling Tor with oprofile
---------------------------
@@ -168,6 +174,55 @@ Here are some basic instructions
* `opreport -l that_dir/*`
- Profit
+Profiling Tor with perf
+-----------------------
+
+This works with a running Tor, and requires root.
+
+1. Decide how long you want to profile for. Start with (say) 30 seconds. If that
+ works, try again with longer times.
+
+2. Find the PID of your running tor process.
+
+3. Run `perf record --call-graph dwarf -p <PID> sleep <SECONDS>`
+
+ (You may need to do this as root.)
+
+ You might need to add `-e cpu-clock` as an option to the perf record line
+ above, if you are on an older CPU without access to hardware profiling
+ events, or in a VM, or something.
+
+4. Now you have a perf.data file. Have a look at it with `perf report
+ --no-children --sort symbol,dso` or `perf report --no-children --sort
+ symbol,dso --stdio --header`. How does it look?
+
+5a. Once you have a nice big perf.data file, you can compress it, encrypt it,
+ and send it to your favorite Tor developers.
+
+5b. Or maybe you'd rather not send a nice big perf.data file. Who knows what's
+ in that!? It's kinda scary. To generate a less scary file, you can use `perf
+ report -g > <FILENAME>.out`. Then you can compress that and put it somewhere
+ public.
+
+Profiling Tor with gperftools aka Google-performance-tools
+----------------------------------------------------------
+
+This should work on nearly any unixy system. It doesn't seem to be compatible
+with RunAsDaemon though.
+
+Beforehand, install google-perftools.
+
+1. You need to rebuild Tor, hack the linking steps to add `-lprofiler` to the
+ libs. You can do this by adding `LIBS=-lprofiler` when you call `./configure`.
+
+Now you can run Tor with profiling enabled, and use the pprof utility to look at
+performance! See the gperftools manual for more info, but basically:
+
+2. Run `env CPUPROFILE=/tmp/profile src/or/tor -f <path/torrc>`. The profile file
+ is not written to until Tor finishes execuction.
+
+3. Run `pprof src/or/tor /tm/profile` to start the REPL.
+
Generating and analyzing a callgraph
------------------------------------
diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md
index 7595398241..4761ca9a37 100644
--- a/doc/HACKING/ReleasingTor.md
+++ b/doc/HACKING/ReleasingTor.md
@@ -28,7 +28,7 @@ new Tor release:
Is make check-spaces happy?
- Does 'make distcheck' compain?
+ Does 'make distcheck' complain?
How about 'make test-stem' and 'make test-network'?
@@ -98,7 +98,7 @@ new Tor release:
to a release-0.2.x branch, manually commit the changelogs to the later
git branches too.
-3. If you're doing the first stable release in a series, you need to
+3. If you're doing the first stable release in a series, you need to
create a ReleaseNotes for the series as a whole. To get started
there, copy all of the Changelog entries from the series into a new
file, and run `./scripts/maint/sortChanges.py` on it. That will
@@ -164,12 +164,15 @@ new Tor release:
0.2.2.23-alpha" (or whatever the version is), and we select the date as
the date in the ChangeLog.
-5. Wait up to a day or two (for a development release), or until most
- packages are up (for a stable release), and mail the release blurb and
- changelog to tor-talk or tor-announce.
+5. Mail the release blurb and ChangeLog to tor-talk (development release) or
+ tor-announce (stable).
- (We might be moving to faster announcements, but don't announce until
- the website is at least updated.)
+ Post the changelog on the the blog as well. You can generate a
+ blog-formatted version of the changelog with the -B option to
+ format-changelog.
+
+ When you post, include an estimate of when the next TorBrowser releases
+ will come out that include this Tor release.
=== V. Aftermath and cleanup
@@ -182,4 +185,5 @@ new Tor release:
2. Forward-port the ChangeLog (and ReleaseNotes if appropriate).
+3. Keep an eye on the blog post, to moderate comments and answer questions.
diff --git a/doc/HACKING/WritingTests.md b/doc/HACKING/WritingTests.md
index de80bbdef2..4dae41e922 100644
--- a/doc/HACKING/WritingTests.md
+++ b/doc/HACKING/WritingTests.md
@@ -48,7 +48,7 @@ isolation, you just run `./src/test/test-memwipe`.
To run tests within the unit test programs, you can specify the name
of the test. The string ".." can be used as a wildcard at the end of the
test name. For example, to run all the cell format tests, enter
-`./src/test/test cellfmt/..`. To run
+`./src/test/test cellfmt/..`.
Many tests that need to mess with global state run in forked subprocesses in
order to keep from contaminating one another. But when debugging a failing test,