diff options
Diffstat (limited to 'doc/WritingTests.txt')
| -rw-r--r-- | doc/WritingTests.txt | 273 |
1 files changed, 0 insertions, 273 deletions
diff --git a/doc/WritingTests.txt b/doc/WritingTests.txt deleted file mode 100644 index 62a17e3709..0000000000 --- a/doc/WritingTests.txt +++ /dev/null @@ -1,273 +0,0 @@ - -Writing tests for Tor: an incomplete guide -========================================== - -Tor uses a variety of testing frameworks and methodologies to try to -keep from introducing bugs. The major ones are: - - 1. Unit tests written in C and shipped with the Tor distribution. - - 2. Integration tests written in Python and shipped with the Tor - distribution. - - 3. Integration tests written in Python and shipped with the Stem - library. Some of these use the Tor controller protocol. - - 4. System tests written in Python and SH, and shipped with the - Chutney package. These work by running many instances of Tor - locally, and sending traffic through them. - - 5. The Shadow network simulator. - -How to run these tests ----------------------- - -=== The easy version - -To run all the tests that come bundled with Tor, run "make check" - -To run the Stem tests as well, fetch stem from the git repository, -set STEM_SOURCE_DIR to the checkout, and run "make test-stem". - -To run the Chutney tests as well, fetch chutney from the git repository, -set CHUTNEY_PATH to the checkout, and run "make test-network". - -To run all of the above, run "make test-full". - -To run all of the above, plus tests that require a working connection to the -internet, run "make test-full-online". - -=== Running particular subtests - -The Tor unit tests are divided into separate programs and a couple of -bundled unit test programs. - -Separate programs are easy. For example, to run the memwipe tests in -isolation, you just run ./src/test/test-memwipe . - -To run tests within the unit test programs, you can specify the name -of the test. The string ".." can be used as a wildcard at the end of the -test name. For example, to run all the cell format tests, enter -"./src/test/test cellfmt/..". To run - -Many tests that need to mess with global state run in forked subprocesses in -order to keep from contaminating one another. But when debugging a failing test, -you might want to run it without forking a subprocess. To do so, use the -"--no-fork" option with a single test. (If you specify it along with -multiple tests, they might interfere.) - -You can turn on logging in the unit tests by passing one of "--debug", -"--info", "--notice", or "--warn". By default only errors are displayed. - -Unit tests are divided into "./src/test/test" and "./src/test/test-slow". -The former are those that should finish in a few seconds; the latter tend to -take more time, and may include CPU-intensive operations, deliberate delays, -and stuff like that. - -=== Finding test coverage - -When you configure Tor with the --enable-coverage option, it should -build with support for coverage in the unit tests, and in a special -"tor-cov" binary. - -Then, run the tests you'd like to see coverage from. If you have old -coverage output, you may need to run "reset-gcov" first. - -Now you've got a bunch of files scattered around your build directories -called "*.gcda". In order to extract the coverage output from them, make a -temporary directory for them and run "./scripts/test/coverage ${TMPDIR}", -where ${TMPDIR} is the temporary directory you made. This will create a -".gcov" file for each source file under tests, containing that file's source -annotated with the number of times the tests hit each line. (You'll need to -have gcov installed.) - -You can get a summary of the test coverage for each file by running -"./scripts/test/cov-display ${TMPDIR}/*" . Each line lists the file's name, -the number of uncovered lines, the number of uncovered lines, and the -coverage percentage. - -For a summary of the test coverage for each _function_, run -"./scripts/test/cov-display -f ${TMPDIR}/*" . - -=== Comparing test coverage - -Sometimes it's useful to compare test coverage for a branch you're writing to -coverage from another branch (such as git master, for example). But you -can't run "diff" on the two coverage outputs directly, since the actual -number of times each line is executed aren't so important, and aren't wholly -deterministic. - -Instead, follow the instructions above for each branch, creating a separate -temporary directory for each. Then, run "./scripts/test/cov-diff ${D1} -${D2}", where D1 and D2 are the directories you want to compare. This will -produce a diff of the two directories, with all lines normalized to be either -covered or uncovered. - -To count new or modified uncovered lines in D2, you can run: - - "./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' |wc -l - - -What kinds of test should I write? ----------------------------------- - -Integration testing and unit testing are complementary: it's probably a -good idea to make sure that your code is hit by both if you can. - -If your code is very-low level, and its behavior is easily described in -terms of a relation between inputs and outputs, or a set of state -transitions, then it's a natural fit for unit tests. (If not, please -consider refactoring it until most of it _is_ a good fit for unit -tests!) - -If your code adds new externally visible functionality to Tor, it would -be great to have a test for that functionality. That's where -integration tests more usually come in. - -Unit and regression tests: Does this function do what it's supposed to? ------------------------------------------------------------------------ - -Most of Tor's unit tests are made using the "tinytest" testing framework. -You can see a guide to using it in the tinytest manual at - - https://github.com/nmathewson/tinytest/blob/master/tinytest-manual.md - -To add a new test of this kind, either edit an existing C file in src/test/, -or create a new C file there. Each test is a single function that must -be indexed in the table at the end of the file. We use the label "done:" as -a cleanup point for all test functions. - -(Make sure you read tinytest-manual.md before proceeding.) - -I use the term "unit test" and "regression tests" very sloppily here. - -=== A simple example - -Here's an example of a test function for a simple function in util.c: - - static void - test_util_writepid(void *arg) - { - (void) arg; - - char *contents = NULL; - const char *fname = get_fname("tmp_pid"); - unsigned long pid; - char c; - - write_pidfile(fname); - - contents = read_file_to_str(fname, 0, NULL); - tt_assert(contents); - - int n = sscanf(contents, "%lu\n%c", &pid, &c); - tt_int_op(n, OP_EQ, 1); - tt_int_op(pid, OP_EQ, getpid()); - - done: - tor_free(contents); - } - -This should look pretty familiar to you if you've read the tinytest -manual. One thing to note here is that we use the testing-specific -function "get_fname" to generate a file with respect to a temporary -directory that the tests use. You don't need to delete the file; -it will get removed when the tests are done. - -Also note our use of OP_EQ instead of == in the tt_int_op() calls. -We define OP_* macros to use instead of the binary comparison -operators so that analysis tools can more easily parse our code. -(Coccinelle really hates to see == used as a macro argument.) - -Finally, remember that by convention, all *_free() functions that -Tor defines are defined to accept NULL harmlessly. Thus, you don't -need to say "if (contents)" in the cleanup block. - -=== Exposing static functions for testing - -Sometimes you need to test a function, but you don't want to expose -it outside its usual module. - -To support this, Tor's build system compiles a testing version of -each module, with extra identifiers exposed. If you want to -declare a function as static but available for testing, use the -macro "STATIC" instead of "static". Then, make sure there's a -macro-protected declaration of the function in the module's header. - -For example, crypto_curve25519.h contains: - -#ifdef CRYPTO_CURVE25519_PRIVATE -STATIC int curve25519_impl(uint8_t *output, const uint8_t *secret, - const uint8_t *basepoint); -#endif - -The crypto_curve25519.c file and the test_crypto.c file both define -CRYPTO_CURVE25519_PRIVATE, so they can see this declaration. - -=== Mock functions for testing in isolation - -Often we want to test that a function works right, but the function to -be tested depends on other functions whose behavior is hard to observe, -or which require a working Tor network, or something like that. - -To write tests for this case, you can replace the underlying functions -with testing stubs while your unit test is running. You need to declare -the underlying function as 'mockable', as follows: - - MOCK_DECL(returntype, functionname, (argument list)); - -and then later implement it as: - - MOCK_IMPL(returntype, functionname, (argument list)) - { - /* implementation here */ - } - -For example, if you had a 'connect to remote server' function, you could -declare it as: - - - MOCK_DECL(int, connect_to_remote, (const char *name, status_t *status)); - -When you declare a function this way, it will be declared as normal in -regular builds, but when the module is built for testing, it is declared -as a function pointer initialized to the actual implementation. - -In your tests, if you want to override the function with a temporary -replacement, you say: - - MOCK(functionname, replacement_function_name); - -And later, you can restore the original function with: - - UNMOCK(functionname); - -For more information, see the definitions of this mocking logic in -testsupport.h. - - -=== Advanced techniques: Namespaces - -XXXX write this. danah boyd made us some really awesome stuff here. - - -Integration tests: Calling Tor from the outside ------------------------------------------------ - -XXXX WRITEME - -Writing integration tests with Stem ------------------------------------ - -XXXX WRITEME - -System testing with Chutney ---------------------------- - -XXXX WRITEME - -Who knows what evil lurks in the timings of networks? The Shadow knows! ------------------------------------------------------------------------ - -XXXX WRITEME - |