Merge commit 'karsten/proposal-166-impl-master'

4 files changed, 234 insertions, 10 deletions

diff --git a/doc/spec/dir-spec.txt b/doc/spec/dir-spec.txt
index 4bec7b00d0..483a33b73c 100644
--- a/doc/spec/dir-spec.txt
+++ b/doc/spec/dir-spec.txt
@@ -641,6 +641,200 @@
         "geoip-start" is the time at which we began collecting geoip
         statistics.
 
+    "dirreq-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        A "dirreq-stats-end" line, as well as any other "dirreq-*" line,
+        is only added when the relay has opened its Dir port and after 24
+        hours of measuring directory requests.
+
+    "dirreq-v2-ips" CC=N,CC=N,... NL
+        [At most once.]
+    "dirreq-v3-ips" CC=N,CC=N,... NL
+        [At most once.]
+
+        List of mappings from two-letter country codes to the number of
+        unique IP addresses that have connected from that country to
+        request a v2/v3 network status, rounded up to the nearest multiple
+        of 8. Only those IP addresses are counted that the directory can
+        answer with a 200 OK status code.
+
+    "dirreq-v2-reqs" CC=N,CC=N,... NL
+        [At most once.]
+    "dirreq-v3-reqs" CC=N,CC=N,... NL
+        [At most once.]
+
+        List of mappings from two-letter country codes to the number of
+        requests for v2/v3 network statuses from that country, rounded up
+        to the nearest multiple of 8. Only those requests are counted that
+        the directory can answer with a 200 OK status code.
+
+    "dirreq-v2-share" num% NL
+        [At most once.]
+    "dirreq-v3-share" num% NL
+        [At most once.]
+
+        The share of v2/v3 network status requests that the directory
+        expects to receive from clients based on its advertised bandwidth
+        compared to the overall network bandwidth capacity. Shares are
+        formatted in percent with two decimal places. Shares are
+        calculated as means over the whole 24-hour interval.
+
+    "dirreq-v2-resp" status=num,... NL
+        [At most once.]
+    "dirreq-v3-resp" status=nul,... NL
+        [At most once.]
+
+        List of mappings from response statuses to the number of requests
+        for v2/v3 network statuses that were answered with that response
+        status, rounded up to the nearest multiple of 4. Only response
+        statuses with at least 1 response are reported. New response
+        statuses can be added at any time. The current list of response
+        statuses is as follows:
+
+        "ok": a network status request is answered; this number
+           corresponds to the sum of all requests as reported in
+           "dirreq-v2-reqs" or "dirreq-v3-reqs", respectively, before
+           rounding up.
+        "not-enough-sigs: a version 3 network status is not signed by a
+           sufficient number of requested authorities.
+        "unavailable": a requested network status object is unavailable.
+        "not-found": a requested network status is not found.
+        "not-modified": a network status has not been modified since the
+           If-Modified-Since time that is included in the request.
+        "busy": the directory is busy.
+
+    "dirreq-v2-direct-dl" key=val,... NL
+        [At most once.]
+    "dirreq-v3-direct-dl" key=val,... NL
+        [At most once.]
+    "dirreq-v2-tunneled-dl" key=val,... NL
+        [At most once.]
+    "dirreq-v3-tunneled-dl" key=val,... NL
+        [At most once.]
+
+        List of statistics about possible failures in the download process
+        of v2/v3 network statuses. Requests are either "direct"
+        HTTP-encoded requests over the relay's directory port, or
+        "tunneled" requests using a BEGIN_DIR cell over the relay's OR
+        port. The list of possible statistics can change, and statistics
+        can be left out from reporting. The current list of statistics is
+        as follows:
+
+        Successful downloads and failures:
+
+        "complete": a client has finished the download successfully.
+        "timeout": a download did not finish within 10 minutes after
+           starting to send the response.
+        "running": a download is still running at the end of the
+           measurement period for less than 10 minutes after starting to
+           send the response.
+
+        Download times:
+
+        "min", "max": smallest and largest measured bandwidth in B/s.
+        "d[1-4,6-9]": 1st to 4th and 6th to 9th decile of measured
+           bandwidth in B/s. For a given decile i, i/10 of all downloads
+           had a smaller bandwidth than di, and (10-i)/10 of all downloads
+           had a larger bandwidth than di.
+        "q[1,3]": 1st and 3rd quartile of measured bandwidth in B/s. One
+           fourth of all downloads had a smaller bandwidth than q1, one
+           fourth of all downloads had a larger bandwidth than q3, and the
+           remaining half of all downloads had a bandwidth between q1 and
+           q3.
+        "md": median of measured bandwidth in B/s. Half of the downloads
+           had a smaller bandwidth than md, the other half had a larger
+           bandwidth than md.
+
+    "entry-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        An "entry-stats-end" line, as well as any other "entry-*"
+        line, is first added after the relay has been running for at least
+        24 hours.
+
+    "entry-ips" CC=N,CC=N,... NL
+        [At most once.]
+
+        List of mappings from two-letter country codes to the number of
+        unique IP addresses that have connected from that country to the
+        relay and which are no known other relays, rounded up to the
+        nearest multiple of 8.
+
+    "cell-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        A "cell-stats-end" line, as well as any other "cell-*" line,
+        is first added after the relay has been running for at least 24
+        hours.
+
+    "cell-processed-cells" num,...,num NL
+        [At most once.]
+
+        Mean number of processed cells per circuit, subdivided into
+        deciles of circuits by the number of cells they have processed in
+        descending order from loudest to quietest circuits.
+
+    "cell-queued-cells" num,...,num NL
+        [At most once.]
+
+        Mean number of cells contained in queues by circuit decile. These
+        means are calculated by 1) determining the mean number of cells in
+        a single circuit between its creation and its termination and 2)
+        calculating the mean for all circuits in a given decile as
+        determined in "cell-processed-cells". Numbers have a precision of
+        two decimal places.
+
+    "cell-time-in-queue" num,...,num NL
+        [At most once.]
+
+        Mean time cells spend in circuit queues in milliseconds. Times are
+        calculated by 1) determining the mean time cells spend in the
+        queue of a single circuit and 2) calculating the mean for all
+        circuits in a given decile as determined in
+        "cell-processed-cells".
+
+    "cell-circuits-per-decile" num NL
+        [At most once.]
+
+        Mean number of circuits that are included in any of the deciles,
+        rounded up to the next integer.
+
+    "exit-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        An "exit-stats-end" line, as well as any other "exit-*" line, is
+        first added after the relay has been running for at least 24 hours
+        and only if the relay permits exiting (where exiting to a single
+        port and IP address is sufficient).
+
+    "exit-kibibytes-written" port=N,port=N,... NL
+        [At most once.]
+    "exit-kibibytes-read" port=N,port=N,... NL
+        [At most once.]
+
+        List of mappings from ports to the number of kibibytes that the
+        relay has written to or read from exit connections to that port,
+        rounded up to the next full kibibyte.
+
+    "exit-streams-opened" port=N,port=N,... NL
+        [At most once.]
+
+        List of mappings from ports to the number of opened exit streams
+        to that port, rounded up to the nearest multiple of 4.
+
     "router-signature" NL Signature NL
         [At end, exactly once.]
 
diff --git a/doc/spec/proposals/000-index.txt b/doc/spec/proposals/000-index.txt
index af1f40bf6d..d2d3ca5d72 100644
--- a/doc/spec/proposals/000-index.txt
+++ b/doc/spec/proposals/000-index.txt
@@ -86,7 +86,7 @@ Proposals by number:
 163  Detecting whether a connection comes from a client [OPEN]
 164  Reporting the status of server votes [OPEN]
 165  Easy migration for voting authority sets [OPEN]
-166  Including Network Statistics in Extra-Info Documents [OPEN]
+166  Including Network Statistics in Extra-Info Documents [ACCEPTED]
 
 
 Proposals by status:
@@ -114,7 +114,6 @@ Proposals by status:
    163  Detecting whether a connection comes from a client [for 0.2.2]
    164  Reporting the status of server votes [for 0.2.2]
    165  Easy migration for voting authority sets
-   166  Including Network Statistics in Extra-Info Documents [for 0.2.2]
  ACCEPTED:
    110  Avoiding infinite length circuits [for 0.2.1.x] [in 0.2.1.3-alpha]
    117  IPv6 exits [for 0.2.1.x]
@@ -122,6 +121,7 @@ Proposals by status:
    140  Provide diffs between consensuses [for 0.2.2.x]
    147  Eliminate the need for v2 directories in generating v3 directories [for 0.2.1.x]
    157  Make certificate downloads specific [for 0.2.1.x]
+   166  Including Network Statistics in Extra-Info Documents [for 0.2.2]
  META:
    000  Index of Tor Proposals
    001  The Tor Proposal Process
diff --git a/doc/spec/proposals/166-statistics-extra-info-docs.txt b/doc/spec/proposals/166-statistics-extra-info-docs.txt
index 3716c049cc..ab2716a71c 100644
--- a/doc/spec/proposals/166-statistics-extra-info-docs.txt
+++ b/doc/spec/proposals/166-statistics-extra-info-docs.txt
@@ -3,7 +3,7 @@ Title: Including Network Statistics in Extra-Info Documents
 Author: Karsten Loesing
 Created: 21-Jul-2009
 Target: 0.2.2
-Status: Open
+Status: Accepted
 
 Change history:
 
@@ -298,7 +298,7 @@ Exit statistics:
 
   The last type of statistics affects exit nodes counting the number of
   bytes written and read and the number of streams opened per port and
-  per 24 hours. Exit port statistics can be measured from looking of
+  per 24 hours. Exit port statistics can be measured from looking at
   headers of BEGIN and DATA cells. A BEGIN cell contains the exit port
   that is required for the exit node to open a new exit stream.
   Subsequent DATA cells coming from the client or being sent back to the
@@ -361,7 +361,7 @@ Implementation notes:
      basically means renaming keywords.
 
   2. The timing of writing the four *-stats files should be unified, so
-     that they are written exactly after 24 hours after starting the
+     that they are written exactly 24 hours after starting the
      relay. Right now, the measurement intervals for dirreq, entry, and
      exit stats starts with the first observed request, and files are
      written when observing the first request that occurs more than 24
@@ -373,14 +373,14 @@ Implementation notes:
      directory until they are included in extra-info documents. The
      reason is that the 24-hour measurement interval can be very
      different from the 18-hour publication interval of extra-info
-     documents. When a relay crashed after finishing a measurement
+     documents. When a relay crashes after finishing a measurement
      interval, but before publishing the next extra-info document,
      statistics would get lost. Therefore, statistics are written to
      disk when finishing a measurement interval and read from disk when
-     generating an extra-info document. As a result, the *-stats files
-     need to be overwritten after 24 hours, rather than appending new
-     statistics to them. Further, the contents of the *-stats files need
-     to be checked in the process of generating extra-info documents.
+     generating an extra-info document. Only the statistics that were
+     appended to the *-stats files within the past 24 hours are included
+     in extra-info documents. Further, the contents of the *-stats files
+     need to be checked in the process of generating extra-info documents.
 
   4. With the statistics patches being tested, the ./configure options
      should be removed and the statistics code be compiled by default.
diff --git a/doc/tor.1.in b/doc/tor.1.in
index 9386b90a14..ba703079c8 100644
--- a/doc/tor.1.in
+++ b/doc/tor.1.in
@@ -1075,6 +1075,36 @@ behalf of clients.
 .TP
 \fBGeoIPFile \fR\fIfilename\fP
 A filename containing GeoIP data, for use with BridgeRecordUsageByCountry.
+.LP
+.TP
+\fBCellStatistics \fR\fB0\fR|\fB1\fR\fP
+When this option is enabled, Tor writes statistics on the mean time that
+cells spend in circuit queues to disk every 24 hours. Cannot be changed
+while Tor is running. (Default: 0)
+.LP
+.TP
+\fBDirReqStatistics \fR\fB0\fR|\fB1\fR\fP
+When this option is enabled, Tor writes statistics on the number and
+response time of network status requests to disk every 24 hours. Cannot be
+changed while Tor is running. (Default: 0)
+.LP
+.TP
+\fBEntryStatistics \fR\fB0\fR|\fB1\fR\fP
+When this option is enabled, Tor writes statistics on the number of
+directly connecting clients to disk every 24 hours. Cannot be changed
+while Tor is running. (Default: 0)
+.LP
+.TP
+\fBExitPortStatistics \fR\fB0\fR|\fB1\fR\fP
+When this option is enabled, Tor writes statistics on the number of
+relayed bytes and opened stream per exit port to disk every 24 hours.
+Cannot be changed while Tor is running. (Default: 0)
+.LP
+.TP
+\fBExtraInfoStatistics \fR\fB0\fR|\fB1\fR\fP
+When this option is enabled, Tor includes previously gathered statistics
+in its extra-info documents that it uploads to the directory authorities.
+(Default: 0)
 
 .SH DIRECTORY SERVER OPTIONS
 .PP