prop340: Tweak how we handle optional stream IDs

Introduce an optional header called the "message routing header" which
contains a `stream_id` and is only present for commands that do require
it.

We also specify that if an unrecognized command is encountered, the
circuit MUST be destroyed immediately.

Signed-off-by: David Goulet <dgoulet@torproject.org>

1 files changed, 117 insertions, 121 deletions

diff --git a/proposals/340-packed-and-fragmented.md b/proposals/340-packed-and-fragmented.md
index 6fb1ca5..121be68 100644
--- a/proposals/340-packed-and-fragmented.md
+++ b/proposals/340-packed-and-fragmented.md
@@ -34,7 +34,8 @@ This proposal combines ideas from
 [next-generation relay cryptography](./308-counter-galois-onion).
 
 Additionally, this proposal has been revised to incorporate another
-protocol change, and remove StreamId from the relay cell header.
+protocol change, and move StreamId from the relay cell header into a new,
+optional header.
 
 ## A preliminary change: Relay encryption, version 1.5
 
@@ -54,36 +55,50 @@ The new format for a decrypted relay _cell_ will be:
    digest     [14 bytes]
    body       [509 - 16 = 493 bytes]
 
-"Digest" and "recognized" are computed as before; the only difference
-is that they occur _before_ the rest of the cell, and that "digest" is
-truncated to 14 bytes instead of 4.
+The `recognized` and `digest` fields are computed as before; the only
+difference is that they occur _before_ the rest of the cell, and that `digest`
+is truncated to 14 bytes instead of 4.
 
 If we are lucky, we won't have to build this encryption at all, and we
 can just move to some version of GCM-UIV or other RPRP that reserves 16
 bytes for an authentication tag or similar cryptographic object.
 
+The `body` MUST contain exactly 493 bytes as cells have a fixed size.
+
 ## New relay message packing
 
-Relay _messages_ now follow the following format:
+We define this new format for a relay message which has to fit within one
+relay cell. However, the body can be split accross many relay cells:
+
+```
+  Message Header
+    command         u8
+    length          u16
+  Message Routing Header (optional)
+    stream_id       u16
+  Message Body
+    data            u8[length]
+```
 
-  Header
-    command   u8
-    length    u16
-  Body
-    data      u8[length]
+One big change from the current tor protocol is something that has become
+optional: we have moved `stream_id` into a separate inner header that only
+appears sometimes named the Message Routing Header. The command value tells us
+if the header is to be expected or not.
+
+The following message types take required stream IDs: `BEGIN`, `DATA`, `END`,
+`CONNECTED`, `RESOLVE`, `RESOLVED`, and `BEGIN_DIR`, `XON`, `XOFF`.
 
-We require that "command" is never 0.
+The following message types from proposal 339 (UDP) take required stream IDs:
+`CONNECT_UDP`, `CONNECTED_UDP` and `DATAGRAM`.
 
-One big change from the current tor protocol is something that is _not_
-here: we have moved `stream_id` into the body of the relay message of
-those commands that need it.
+No other message types take stream IDs. The `stream_id` field, when present,
+MUST NOT be zero.
 
 Messages can be split across relay cells; multiple messages can occur in
 a single relay cell.  We enforce the following rules:
 
   * Headers may not be split across cells.
-  * If a 0 byte follows any relay message, there are no more messages in
-    that cell.
+  * If a 0 byte follows a message body, there are no more messages.
   * A relay cell may not be "empty": it must have at least some part of
     some message.
 
@@ -91,15 +106,15 @@ Unless specified elsewhere, **all** message types may be packed, and
 **all** message types may be fragmented.
 
 Every command has an associated maximum length for its messages.  If not
-specified elsewhere, the maximum length for every message is 498 bytes
-(for legacy reasons).
-
-Receivers MUST validate that headers are well-formed and have valid
-lengths while handling the cell in which the header is encoded.  If the
-header is invalid, the receiver must destroy the circuit.
-
+specified elsewhere, the maximum length for every message is 498 bytes (for
+legacy reasons).
 
+Receivers MUST validate that the cell `header` and the `message header` are
+well-formed and have valid lengths while handling the cell in which the header
+is encoded. If any of them is invalid, the circuit MUST be destroyed.
 
+An unrecognized `command` is considered invalid and thus MUST result in the
+circuit being immediately destroyed.
 
 ## Negotiation
 
@@ -115,42 +130,6 @@ The `version` field is the `Relay` subprotocol version that the client
 wants to use. The relay must send back the same extension in its ntor3
 handshake to acknowledge support.
 
-
-## Specifying the message format with moved stream ID.
-
-Here, we'll specify how to adjust tor-spec to describe the `stream_id`
-move correctly.
-
-Below, suppose that `Relay=V` denotes whatever version of the relay
-message subprotocol denotes support for this proposal.
-
-For all relay message types that include a stream ID, we insert
-the following at the beginning of their fields:
-
->```
->        STREAM_ID [2 bytes] (Present when Relay protocol version >= V).
->```
-
-We add a note saying:
-
-> STREAM_ID is part of many message types, when using Relay protocol
-> version V or later.  Earlier versions of the Relay protocol put
-> STREAM_ID in the RELAY header.  In those versions, the field is
-> STREAM_ID omitted from the message.
->
-> Except when noted, STREAM_ID may not be zero.
-
-The following message types take required stream IDs: `BEGIN`, `DATA`, `END`,
-`CONNECTED`, `RESOLVE`, `RESOLVED`, and `BEGIN_DIR`, `XON`, `XOFF`.
-
-The following message type takes an *optional* stream ID: `SENDME`.
-(*Stream-level sendmes are not a thing anymore with proposal 324, but I
-want to give implementations the freedom to implement prop324 and this
-proposal in either order.*)
-
-The following message types from proposal 339 (UDP) take required stream
-IDs: `CONNECT_UDP`, `CONNECTED_UDP` and `DATAGRAM`.
-
 ## Migration
 
 We add a consensus parameter, "streamed-relay-messages", with default
@@ -204,7 +183,7 @@ this.)
 
 ### Extending message-length maxima
 
-For now, the maximum length for every message is 498 bytes, except as
+For now, the maximum length for every message body is 493 bytes, except as
 follows:
 
    - `DATAGRAM` messages (see proposal 339) have a maximum body length
@@ -218,149 +197,166 @@ saying so, and reserving a new subprotocol version.)
 
 # Appendix: Example cells
 
-
-Here is an example of the simplest case: one message, sent in one relay
-cell.  Here it is a BEGIN message.
+Here is an example of the simplest case: one message, sent in one relay cell:
 
 ```
   Cell 1:
-    Cell header
+    header:
        circid         ..                [4 bytes]
        command        RELAY             [1 byte]
-    relay cell header
+    relay cell header:
        recognized     0                 [2 bytes]
        digest         (...)             [14 bytes]
     message header:
        command        BEGIN             [1 byte]
        length         23                [2 bytes]
-    message body
-       stream_id      (...)             [2 bytes]
+    message routing header:
+       stream_id      42                [2 bytes]
+    message body:
       "www.torproject.org:443\0"        [23 bytes]
-    end-of-messages marker
+    end-of-messages marker:
       0                                 [1 byte]
-    padding up to end of cell
+    padding up to end of cell:
       random                            [464 bytes]
-
 ```
 
+Total of 514 bytes which is the absolute maximum cell size.
+
 Here's an example with fragmentation only: a large EXTEND2 message split
 across two relay cells.
 
 ```
   Cell 1:
-    Cell header
-       circid         ..                [4 bytes]
-       command        RELAY_EARLY       [1 byte]
-    relay cell header
-       recognized     0                 [2 bytes]
-       digest         (...)             [14 bytes]
+    header:
+       circid         ..               [4 bytes]
+       command        RELAY_EARLY      [1 byte]
+    relay cell header:
+       recognized     0                [2 bytes]
+       digest         (...)            [14 bytes]
     message header:
-       command        EXTEND            [1 byte]
-       length         800               [2 bytes]
-    message body
-       stream_id      0                 [2 bytes]
-       (extend body, part 1)            [488 bytes]
+       command        EXTEND           [1 byte]
+       length         800              [2 bytes]
+    message body:
+       (extend body, part 1)           [490 bytes]
 
   Cell 2:
-    Cell header
-       circid         ..                [4 bytes]
-       command        RELAY             [1 byte]
-    relay cell header
+    header:
+       circid         ..               [4 bytes]
+       command        RELAY            [1 byte]
+    relay cell header:
       recognized     0                 [2 bytes]
       digest         (...)             [14 bytes]
-    message body, continued
-      (extend body, part 2)            [312 bytes]
-    end-of-messages marker
+    message body, continued:
+      (extend body, part 2)            [310 bytes] (310+490=800)
+    end-of-messages marker:
       0                                [1 byte]
-    padding up to end of cell
-      random                           [180 bytes]
+    padding up to end of cell:
+      random                           [182 bytes]
 
 ```
 
-Here is an example with packing only: A begin_dir message and a data
-message in the same cell.
+Each cells are 514 bytes for a message body totalling 800 bytes.
+
+Here is an example with packing only: A `BEGIN_DIR` message and a data message
+in the same cell.
 
 ```
   Cell 1:
-    Cell header
+    header:
        circid         ..                [4 bytes]
        command        RELAY             [1 byte]
-    relay cell header
+    relay cell header:
        recognized     0                 [2 bytes]
        digest         (...)             [14 bytes]
+
+    # First relay message
     message header:
        command        BEGIN_DIR         [1 byte]
        length         0                 [2 bytes]
-    message body:
+    message routing header:
        stream_id      32                [2 bytes]
+
+    # Second relay message
     message header:
        command        DATA              [1 byte]
        length         25                [2 bytes]
-    message body:
+    message routing header:
        stream_id      32                [2 bytes]
+    message body:
        "HTTP/1.0 GET /tor/foo\r\n\r\n"  [25 bytes]
-    end-of-messages marker
+
+    end-of-messages marker:
       0                                 [1 byte]
-    padding up to end of cell
+    padding up to end of cell:
       random                            [457 bytes]
 
 ```
 
 Here is an example with packing and fragmentation: a large DATAGRAM cell, a
-SENDME cell, and an XON cell.  (Note that this sequence of cells would not
-actually be generated by the algorithm described in "Packing decisions"
-above; this is only an example of what parties need to accept.)
+SENDME cell, and an XON cell.
+
+(Note that this sequence of cells would not actually be generated by the
+algorithm described in "Packing decisions" above; this is only an example of
+what parties need to accept.)
 
 ```
   Cell 1:
-    Cell header
-       circid         ..                [4 bytes]
-       command        RELAY             [1 byte]
-    relay cell header
+    header:
+       circid         ..               [4 bytes]
+       command        RELAY            [1 byte]
+    relay cell header:
        recognized     0                [2 bytes]
        digest         (...)            [14 bytes]
+
+    # First message
     message header:
        command        DATAGRAM         [1 byte]
        length         1200             [2 bytes]
-    message body
+    message routing header:
        stream_id      99               [2 bytes]
+    message body:
        (datagram body, part 1)         [488 bytes]
 
   Cell 2:
-    Cell header
-       circid         ..                [4 bytes]
-       command        RELAY             [1 byte]
-    relay cell header
+    header:
+       circid         ..               [4 bytes]
+       command        RELAY            [1 byte]
+    relay cell header:
       recognized     0                 [2 bytes]
       digest         (...)             [14 bytes]
-    message body, continued
+    message body, continued:
       (datagram body, part 2)          [493 bytes]
 
   Cell 3:
-    Cell header
-       circid         ..                [4 bytes]
-       command        RELAY             [1 byte]
-    relay cell header
+    header:
+       circid         ..               [4 bytes]
+       command        RELAY            [1 byte]
+    relay cell header:
       recognized     0                 [2 bytes]
       digest         (...)             [14 bytes]
-    message body, continued
+    message body, continued:
       (datagram body, part 3)          [219 bytes] (488+493+219=1200)
+
+    # Second message
     message header:
        command        SENDME           [1 byte]
        length         23               [2 bytes]
     message body:
-       stream_id      0                [2 bytes]
        version        1                [1 byte]
        datalen        20               [2 bytes]
        data           (digest to ack)  [20 bytes]
+
+    # Third message
     message header:
        command        XON              [1 byte]
        length         1                [2 bytes]
-    message body:
+    message routing header:
        stream_id      50               [2 bytes]
+    message body:
        version        1                [1 byte]
-    end-of-messages marker
+
+    end-of-messages marker:
       0                                [1 byte]
-    padding up to end of cell
-      random                           [239 bytes]
+    padding up to end of cell:
+      random                           [241 bytes]
 ```