``` Filename: 166-statistics-extra-info-docs.txt Title: Including Network Statistics in Extra-Info Documents Author: Karsten Loesing Created: 21-Jul-2009 Target: 0.2.2 Status: Closed Change history: 21-Jul-2009 Initial proposal for or-dev Overview: The Tor network has grown to almost two thousand relays and millions of casual users over the past few years. With growth has come increasing performance problems and attempts by some countries to block access to the Tor network. In order to address these problems, we need to learn more about the Tor network. This proposal suggests to measure additional statistics and include them in extra-info documents to help us understand the Tor network better. Introduction: As of May 2009, relays, bridges, and directories gather the following data for statistical purposes: - Relays and bridges count the number of bytes that they have pushed in 15-minute intervals over the past 24 hours. Relays and bridges include these data in extra-info documents that they send to the directory authorities whenever they publish their server descriptor. - Bridges further include a rough number of clients per country that they have seen in the past 48 hours in their extra-info documents. - Directories can be configured to count the number of clients they see per country in the past 24 hours and to write them to a local file. Since then we extended the network statistics in Tor. These statistics include: - Directories now gather more precise statistics about connecting clients. Fixes include measuring in intervals of exactly 24 hours, counting unsuccessful requests, measuring download times, etc. The directories append their statistics to a local file every 24 hours. - Entry guards count the number of clients per country per day like bridges do and write them to a local file every 24 hours. - Relays measure statistics of the number of cells in their circuit queues and how much time these cells spend waiting there. Relays write these statistics to a local file every 24 hours. - Exit nodes count the number of read and written bytes on exit connections per port as well as the number of opened exit streams per port in 24-hour intervals. Exit nodes write their statistics to a local file. The following four sections contain descriptions for adding these statistics to the relays' extra-info documents. Directory request statistics: The first type of statistics aims at measuring directory requests sent by clients to a directory mirror or directory authority. More precisely, these statistics aim at requests for v2 and v3 network statuses only. These directory requests are sent non-anonymously, either via HTTP-like requests to a directory's Dir port or tunneled over a 1-hop circuit. Measuring directory request statistics is useful for several reasons: First, the number of locally seen directory requests can be used to estimate the total number of clients in the Tor network. Second, the country-wise classification of requests using a GeoIP database can help counting the relative and absolute number of users per country. Third, the download times can give hints on the available bandwidth capacity at clients. Directory requests do not give any hints on the contents that clients send or receive over the Tor network. Every client requests network statuses from the directories, so that there are no anonymity-related concerns to gather these statistics. It might be, though, that clients wish to hide the fact that they are connecting to the Tor network. Therefore, IP addresses are resolved to country codes in memory, events are accumulated over 24 hours, and numbers are rounded up to multiples of 4 or 8. "dirreq-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL [At most once.] YYYY-MM-DD HH:MM:SS defines the end of the included measurement interval of length NSEC seconds (86400 seconds by default). A "dirreq-stats-end" line, as well as any other "dirreq-*" line, is only added when the relay has opened its Dir port and after 24 hours of measuring directory requests. "dirreq-v2-ips" CC=N,CC=N,... NL [At most once.] "dirreq-v3-ips" CC=N,CC=N,... NL [At most once.] List of mappings from two-letter country codes to the number of unique IP addresses that have connected from that country to request a v2/v3 network status, rounded up to the nearest multiple of 8. Only those IP addresses are counted that the directory can answer with a 200 OK status code. "dirreq-v2-reqs" CC=N,CC=N,... NL [At most once.] "dirreq-v3-reqs" CC=N,CC=N,... NL [At most once.] List of mappings from two-letter country codes to the number of requests for v2/v3 network statuses from that country, rounded up to the nearest multiple of 8. Only those requests are counted that the directory can answer with a 200 OK status code. "dirreq-v2-share" num% NL [At most once.] "dirreq-v3-share" num% NL [At most once.] The share of v2/v3 network status requests that the directory expects to receive from clients based on its advertised bandwidth compared to the overall network bandwidth capacity. Shares are formatted in percent with two decimal places. Shares are calculated as means over the whole 24-hour interval. "dirreq-v2-resp" status=num,... NL [At most once.] "dirreq-v3-resp" status=nul,... NL [At most once.] List of mappings from response statuses to the number of requests for v2/v3 network statuses that were answered with that response status, rounded up to the nearest multiple of 4. Only response statuses with at least 1 response are reported. New response statuses can be added at any time. The current list of response statuses is as follows: "ok": a network status request is answered; this number corresponds to the sum of all requests as reported in "dirreq-v2-reqs" or "dirreq-v3-reqs", respectively, before rounding up. "not-enough-sigs: a version 3 network status is not signed by a sufficient number of requested authorities. "unavailable": a requested network status object is unavailable. "not-found": a requested network status is not found. "not-modified": a network status has not been modified since the If-Modified-Since time that is included in the request. "busy": the directory is busy. "dirreq-v2-direct-dl" key=val,... NL [At most once.] "dirreq-v3-direct-dl" key=val,... NL [At most once.] "dirreq-v2-tunneled-dl" key=val,... NL [At most once.] "dirreq-v3-tunneled-dl" key=val,... NL [At most once.] List of statistics about possible failures in the download process of v2/v3 network statuses. Requests are either "direct" HTTP-encoded requests over the relay's directory port, or "tunneled" requests using a BEGIN_DIR cell over the relay's OR port. The list of possible statistics can change, and statistics can be left out from reporting. The current list of statistics is as follows: Successful downloads and failures: "complete": a client has finished the download successfully. "timeout": a download did not finish within 10 minutes after starting to send the response. "running": a download is still running at the end of the measurement period for less than 10 minutes after starting to send the response. Download times: "min", "max": smallest and largest measured bandwidth in B/s. "d[1-4,6-9]": 1st to 4th and 6th to 9th decile of measured bandwidth in B/s. For a given decile i, i/10 of all downloads had a smaller bandwidth than di, and (10-i)/10 of all downloads had a larger bandwidth than di. "q[1,3]": 1st and 3rd quartile of measured bandwidth in B/s. One fourth of all downloads had a smaller bandwidth than q1, one fourth of all downloads had a larger bandwidth than q3, and the remaining half of all downloads had a bandwidth between q1 and q3. "md": median of measured bandwidth in B/s. Half of the downloads had a smaller bandwidth than md, the other half had a larger bandwidth than md. Entry guard statistics: Entry guard statistics include the number of clients per country and per day that are connecting directly to an entry guard. Entry guard statistics are important to learn more about the distribution of clients to countries. In the future, this knowledge can be useful to detect if there are or start to be any restrictions for clients connecting from specific countries. The information which client connects to a given entry guard is very sensitive. This information must not be combined with the information what contents are leaving the network at the exit nodes. Therefore, entry guard statistics need to be aggregated to prevent them from becoming useful for de-anonymization. Aggregation includes resolving IP addresses to country codes, counting events over 24-hour intervals, and rounding up numbers to the next multiple of 8. "entry-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL [At most once.] YYYY-MM-DD HH:MM:SS defines the end of the included measurement interval of length NSEC seconds (86400 seconds by default). An "entry-stats-end" line, as well as any other "entry-*" line, is first added after the relay has been running for at least 24 hours. "entry-ips" CC=N,CC=N,... NL [At most once.] List of mappings from two-letter country codes to the number of unique IP addresses that have connected from that country to the relay and which are no known other relays, rounded up to the nearest multiple of 8. Cell statistics: The third type of statistics have to do with the time that cells spend in circuit queues. In order to gather these statistics, the relay memorizes when it puts a given cell in a circuit queue and when this cell is flushed. The relay further notes the life time of the circuit. These data are sufficient to determine the mean number of cells in a queue over time and the mean time that cells spend in a queue. Cell statistics are necessary to learn more about possible reasons for the poor network performance of the Tor network, especially high latencies. The same statistics are also useful to determine the effects of design changes by comparing today's data with future data. There are basically no privacy concerns from measuring cell statistics, regardless of a node being an entry, middle, or exit node. "cell-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL [At most once.] YYYY-MM-DD HH:MM:SS defines the end of the included measurement interval of length NSEC seconds (86400 seconds by default). A "cell-stats-end" line, as well as any other "cell-*" line, is first added after the relay has been running for at least 24 hours. "cell-processed-cells" num,...,num NL [At most once.] Mean number of processed cells per circuit, subdivided into deciles of circuits by the number of cells they have processed in descending order from loudest to quietest circuits. "cell-queued-cells" num,...,num NL [At most once.] Mean number of cells contained in queues by circuit decile. These means are calculated by 1) determining the mean number of cells in a single circuit between its creation and its termination and 2) calculating the mean for all circuits in a given decile as determined in "cell-processed-cells". Numbers have a precision of two decimal places. "cell-time-in-queue" num,...,num NL [At most once.] Mean time cells spend in circuit queues in milliseconds. Times are calculated by 1) determining the mean time cells spend in the queue of a single circuit and 2) calculating the mean for all circuits in a given decile as determined in "cell-processed-cells". "cell-circuits-per-decile" num NL [At most once.] Mean number of circuits that are included in any of the deciles, rounded up to the next integer. Exit statistics: The last type of statistics affects exit nodes counting the number of bytes written and read and the number of streams opened per port and per 24 hours. Exit port statistics can be measured from looking at headers of BEGIN and DATA cells. A BEGIN cell contains the exit port that is required for the exit node to open a new exit stream. Subsequent DATA cells coming from the client or being sent back to the client contain a length field stating how many bytes of application data are contained in the cell. Exit port statistics are important to measure in order to identify possible load-balancing problems with respect to exit policies. Exit nodes that permit more ports than others are very likely overloaded with traffic for those ports plus traffic for other ports. Improving load balancing in the Tor network improves the overall utilization of bandwidth capacity. Exit traffic is one of the most sensitive parts of network data in the Tor network. Even though these statistics do not require looking at traffic contents, statistics are aggregated so that they are not useful for de-anonymizing users. Only those ports are reported that have seen at least 0.1% of exiting or incoming bytes, numbers of bytes are rounded up to full kibibytes (KiB), and stream numbers are rounded up to the next multiple of 4. "exit-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL [At most once.] YYYY-MM-DD HH:MM:SS defines the end of the included measurement interval of length NSEC seconds (86400 seconds by default). An "exit-stats-end" line, as well as any other "exit-*" line, is first added after the relay has been running for at least 24 hours and only if the relay permits exiting (where exiting to a single port and IP address is sufficient). "exit-kibibytes-written" port=N,port=N,... NL [At most once.] "exit-kibibytes-read" port=N,port=N,... NL [At most once.] List of mappings from ports to the number of kibibytes that the relay has written to or read from exit connections to that port, rounded up to the next full kibibyte. "exit-streams-opened" port=N,port=N,... NL [At most once.] List of mappings from ports to the number of opened exit streams to that port, rounded up to the nearest multiple of 4. Implementation notes: Right now, relays that are configured accordingly write similar statistics to those described in this proposal to disk every 24 hours. With this proposal being implemented, relays include the contents of these files in extra-info documents. The following steps are necessary to implement this proposal: 1. The current format of [dirreq|entry|buffer|exit]-stats files needs to be adapted to the description in this proposal. This step basically means renaming keywords. 2. The timing of writing the four *-stats files should be unified, so that they are written exactly 24 hours after starting the relay. Right now, the measurement intervals for dirreq, entry, and exit stats starts with the first observed request, and files are written when observing the first request that occurs more than 24 hours after the beginning of the measurement interval. With this proposal, the measurement intervals should all start at the same time, and files should be written exactly 24 hours later. 3. It is advantageous to cache statistics in local files in the data directory until they are included in extra-info documents. The reason is that the 24-hour measurement interval can be very different from the 18-hour publication interval of extra-info documents. When a relay crashes after finishing a measurement interval, but before publishing the next extra-info document, statistics would get lost. Therefore, statistics are written to disk when finishing a measurement interval and read from disk when generating an extra-info document. Only the statistics that were appended to the *-stats files within the past 24 hours are included in extra-info documents. Further, the contents of the *-stats files need to be checked in the process of generating extra-info documents. 4. With the statistics patches being tested, the ./configure options should be removed and the statistics code be compiled by default. It is still required for relay operators to add configuration options (DirReqStatistics, ExitPortStatistics, etc.) to enable gathering statistics. However, in the near future, statistics shall be enabled gathered by all relays by default, where requiring a ./configure option would be a barrier for many relay operators. ```