Give tor_cert_get_id_digests() fail-fast behavior

Right now we can take the digests only of an RSA key, and only expect to
take the digests of an RSA key.  The old tor_cert_get_id_digests() would
return a good set of digests for an RSA key, and an all-zero one for a
non-RSA key.  This behavior is too error-prone: it carries the risk that
we will someday check two non-RSA keys for equality and conclude that
they must be equal because they both have the same (zero) "digest".

Instead, let's have tor_cert_get_id_digests() return NULL for keys we
can't handle, and make its callers explicitly test for NULL.

1 files changed, 8 insertions, 3 deletions

diff --git a/src/or/connection_or.c b/src/or/connection_or.c
index 7cdea82191..a5b965b8d1 100644
--- a/src/or/connection_or.c
+++ b/src/or/connection_or.c
@@ -2060,12 +2060,17 @@ connection_or_compute_authenticate_cell_body(or_connection_t *conn,
 
   {
     const tor_cert_t *id_cert=NULL, *link_cert=NULL;
+    const digests_t *my_digests, *their_digests;
     const uint8_t *my_id, *their_id, *client_id, *server_id;
     if (tor_tls_get_my_certs(0, &link_cert, &id_cert))
       return -1;
-    my_id = (uint8_t*)tor_cert_get_id_digests(id_cert)->d[DIGEST_SHA256];
-    their_id = (uint8_t*)
-      tor_cert_get_id_digests(conn->handshake_state->id_cert)->d[DIGEST_SHA256];
+    my_digests = tor_cert_get_id_digests(id_cert);
+    their_digests = tor_cert_get_id_digests(conn->handshake_state->id_cert);
+    tor_assert(my_digests);
+    tor_assert(their_digests);
+    my_id = (uint8_t*)my_digests->d[DIGEST_SHA256];
+    their_id = (uint8_t*)their_digests->d[DIGEST_SHA256];
+
     client_id = server ? their_id : my_id;
     server_id = server ? my_id : their_id;