Merge remote-tracking branch 'tor-gitlab/mr/479' into main

2 files changed, 323 insertions, 193 deletions

diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md
index 490c100fcb..12f5d1b16b 100644
--- a/doc/HACKING/ReleasingTor.md
+++ b/doc/HACKING/ReleasingTor.md
@@ -1,239 +1,114 @@
-# CHECKLIST
-
-Here's a summary checklist, with the things that Nick messes up most often.
-
-Did you:
-
- * [ ] Copy the ChangeLog to the ReleaseNotes?
- * [ ] Check that the new versions got approved?
- * [ ] Check the release date in the ChangeLog?
- * [ ] Update the GeoIP file?
-
-# Putting out a new release
+# How to Release Tor
 
 Here are the steps that the maintainer should take when putting out a
-new Tor release:
-
-## 0. Preliminaries
-
-1. Get at least three of weasel/arma/Sebastian/Sina to put the new
-   version number in their approved versions list.  Give them a few
-   days to do this if you can.
-
-2. If this is going to be an important security release, give the packagers
-   advance warning, via `tor-packagers@lists.torproject.org`.
-
-
-3. Given the release date for Tor, ask the TB team about the likely release
-   date of a TB that contains it.  See note below in "commit, upload,
-   announce".
-
-## I. Make sure it works
-
-1. Make sure that CI passes: have a look at the branches on gitlab.
-
-   _Optionally_, have a look at Travis
-   (https://travis-ci.org/torproject/tor/branches), Appveyor
-   (https://ci.appveyor.com/project/torproject/tor/history), and
-   Jenkins (https://jenkins.torproject.org/view/tor/).
-   Make sure you're looking at the right branches.
-
-   If there are any unexplained failures, try to fix them or figure them
-   out.
-
-2. Verify that there are no big outstanding issues.  You might find such
-   issues --
-
-    * On Gitlab
-
-    * On coverity scan
-
-    * On OSS-Fuzz
-
-## II. Write a changelog
-
-
-1a. (Alpha release variant)
-
-   Gather the `changes/*` files into a changelog entry, rewriting many
-   of them and reordering to focus on what users and funders would find
-   interesting and understandable.
-
-   To do this, run `./scripts/maint/sortChanges.py changes/* > changelog.in`
-   to combine headings and sort the entries.  Copy the changelog.in file into
-   the ChangeLog.  Run `format_changelog.py --inplace` (see below) to clean up
-   the line breaks.
-
-   Remove the `changes/*` files that you just merged into the ChangeLog.
-
-   After that, it's time to hand-edit and fix the issues that
-   lintChanges can't find:
-
-   1. Within each section, sort by "version it's a bugfix on", else by
-      numerical ticket order.
-
-   2. Clean them up:
-
-      Make stuff very terse
-
-      Describe the user-visible problem right away
-
-      Mention relevant config options by name.  If they're rare or unusual,
-      remind people what they're for
-
-      Avoid starting lines with open-paren
-
-      Present and imperative tense: not past.
-
-      "Relays", not "servers" or "nodes" or "Tor relays".
-
-      "Onion services", not "hidden services".
+new Tor release. It is split in 3 stages and coupled with our Tor CI Release
+pipeline.
 
-      "Stop FOOing", not "Fix a bug where we would FOO".
+Before we begin, first rule is to make sure:
 
-      Try not to let any given section be longer than about a page. Break up
-      long sections into subsections by some sort of common subtopic. This
-      guideline is especially important when organizing Release Notes for
-      new stable releases.
+   - Our CI pass for each version to release
+   - Coverity has no new alerts
 
-      If a given changes stanza showed up in a different release (e.g.
-      maint-0.2.1), be sure to make the stanzas identical (so people can
-      distinguish if these are the same change).
+## 0. Security Release
 
-   3. Clean everything one last time.
+To start with, if you are doing a security release, this must be done few days
+prior to the release:
 
-   4. Run `./scripts/maint/format_changelog.py --inplace` to make it prettier
+   1. If this is going to be an important security release, give the packagers
+      advance warning, via `tor-packagers@lists.torproject.org`.
 
-1b. (old-stable release variant)
 
-   For stable releases that backport things from later, we try to compose
-   their releases, we try to make sure that we keep the changelog entries
-   identical to their original versions, with a "backport from 0.x.y.z"
-   note added to each section.  So in this case, once you have the items
-   from the changes files copied together, don't use them to build a new
-   changelog: instead, look up the corrected versions that were merged
-   into ChangeLog in the main branch, and use those.
+## 1. Preliminaries
 
-   Add "backport from X.Y.Z" in the section header for these entries.
+The following must be done 2 days at the very least prior to the release:
 
-2. Compose a short release blurb to highlight the user-facing
-   changes. Insert said release blurb into the ChangeLog stanza. If it's
-   a stable release, add it to the ReleaseNotes file too. If we're adding
-   to a release-* branch, manually commit the changelogs to the later
-   git branches too.
+   1. Add the version(s) in the dirauth-conf git repository as the
+      RecommendedVersion and RequiredVersion so they can be approved by the
+      authorities and be in the consensus before the release.
 
-3. If there are changes that require or suggest operator intervention
-   before or during the update, mail operators (either dirauth or relays
-   list) with a headline that indicates that an action is required or
-   appreciated.
+   2. Send a pre-release announcement to `tor-project@lists.torproject.org` in
+      order to inform every teams in Tor of the upcoming release. This is so
+      we can avoid creating release surprises and sync with other teams.
 
-4. If you're doing the first stable release in a series, you need to
-   create a ReleaseNotes for the series as a whole.  To get started
-   there, copy all of the Changelog entries from the series into a new
-   file, and run `./scripts/maint/sortChanges.py` on it.  That will
-   group them by category.  Then kill every bugfix entry for fixing
-   bugs that were introduced within that release series; those aren't
-   relevant changes since the last series.  At that point, it's time
-   to start sorting and condensing entries.  (Generally, we don't edit the
-   text of existing entries, though.)
+   3. Ask the network-team to review the `changes/` files in all versions we
+      are about to release. This step is encouraged but not mandatory.
 
-## III. Making the source release.
 
-1. In `maint-0.?.x`, bump the version number in `configure.ac` and run
-   `./scripts/main/update_versions.py` to update version numbers in other
-   places, and commit.  Then merge `maint-0.?.x` into `release-0.?.x`.
+## 2. Tarballs
 
-   When you merge the maint branch forward to the next maint branch, or into
-   main, merge it with `-s ours` to avoid conflict with the version
-   bump.
+To build the tarballs to release, we need to launch the CI release pipeline:
 
-2. In `release-0.?.x`, run `make distcheck`, put the tarball up in somewhere
-   (how about your homedir on people.torproject.org?) , and tell `#tor-dev`
-   about it.
+   https://gitlab.torproject.org/tpo/core/tor-ci-release
 
-   If you want, wait until at least one person has built it
-   successfully.  (We used to say "wait for others to test it", but our
-   CI has successfully caught these kinds of errors for the last several
-   years.)
+The `versions.yml` needs to be modified with the Tor versions you want to
+release. Once done, git commit and push to trigger the release pipeline.
 
-3. Make sure that the new version is recommended in the latest consensus.
-   (Otherwise, users will get confused when it complains to them
-   about its status.)
+The first two stages (Preliminary and Patches) will be run automatically. The
+Build stage needs to be triggered manually once all generated patches have
+been merged upstream.
 
-   If it is not, you'll need to poke Roger, Weasel, Sebastian, and Sina
-   again: see the note at the start of the document.
+   1. Download the generated patches from the `Patches` stage.
 
-## IV. Commit, upload, announce
+   2. For the ChangeLog and ReleaseNotes, you need to write a blurb at the top
+      explaining a bit the release.
 
-1. Sign the tarball, then sign and push the git tag:
+   3. Review, modify if needed, and merged them upstream.
 
-```console
-$ gpg -ba <the_tarball>
-$ git tag -s tor-0.4.x.y-<status>
-$ git push origin tag tor-0.4.x.y-<status>
-```
+   4. Manually trigger the `maintained` job in the `Build` stage so the CI can
+      build the tarballs without errors.
 
-   (You must do this before you update the website: the website scripts
-   rely on finding the version by tag.)
+Once this is done, each selected developers need to build the tarballs in a
+reproducible way using:
 
-   (If your default PGP key is not the one you want to sign with, then say
-   "-u <keyid>" instead of "-s".)
+   https://gitlab.torproject.org/tpo/core/tor-ci-reproducible
 
-2. scp the tarball and its sig to the dist website, i.e.
-   `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run
-   "static-update-component dist.torproject.org" on dist-master.
+Simply run the `./build.sh` which will commit interactively the signature for
+you. You then only need to git push.
 
-   In the `project/web/tpo.git` repository, update `databags/versions.ini`
-   to note the new version.  Push these changes to `master`.
+Once all signatures have been committed:
 
-   (NOTE: Due to #17805, there can only be one stable version listed at
-   once.  Nonetheless, do not call your version "alpha" if it is stable,
-   or people will get confused.)
+   1. Manually trigger the `signature` job in the `Post-process` stage of the
+      CI release pipeline.
 
-   (NOTE: It will take a while for the website update scripts to update
-   the website.)
+   2. If it passes, the tarball(s) and signature(s) will be available as
+      artifacts and should be used for the release.
 
-3. Email the tor-packagers@lists.torproject.org mailing list to tell them
-   about the new release.
+   3. Put them on `dist.torproject.org`:
 
-   Also, email tor-packagers@lists.torproject.org.
+      Upload the tarball and its sig to the dist website, i.e.
+      `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run
+      "static-update-component dist.torproject.org" on dist-master.
 
-   Mention where to download the tarball (`https://dist.torproject.org/`).
+      In the `project/web/tpo.git` repository, update `databags/versions.ini`
+      to note the new version. Push these changes to `master`.
 
-   Include a link to the changelog.
+      (NOTE: Due to #17805, there can only be one stable version listed at once.
+      Nonetheless, do not call your version "alpha" if it is stable, or people
+      will get confused.)
 
-4. Wait for the download page to be updated. (If you don't do this before you
-   announce, people will be confused.)
+      (NOTE: It will take a while for the website update scripts to update the
+      website.)
 
-5. Mail the release blurb and ChangeLog to tor-talk (development release) or
-   tor-announce (stable).
 
-   Post the changelog on the blog as well. You can generate a
-   blog-formatted version of the changelog with
-      `./scripts/maint/format_changelog.py -B`
+## 3. Post Process
 
-   When you post, include an estimate of when the next TorBrowser
-   releases will come out that include this Tor release.  This will
-   usually track https://wiki.mozilla.org/RapidRelease/Calendar , but it
-   can vary.
+Once the tarballs have been uploaded and are ready to be announced, we need to
+do the following:
 
-   For templates to use when announcing, see:
-       https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/AnnouncementTemplates
+   1. Merge upstream the artifacts from the `patches` job in the
+      `Post-process` stage of the CI release pipeline.
 
-## V. Aftermath and cleanup
+   2. Write and post the release announcement for the `forum.torproject.net`
+      in the `News -> Tor Release Announcement` category.
 
-1. If it's a stable release, bump the version number in the
-   `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`
-   merge to avoid taking that change into main.
+      Mention in which Tor Browser version (with dates) the release will be
+      in. This usually only applies to the latest stable.
 
-2. If there is a new `maint-x.y.z` branch, create a Travis CI cron job that
-   builds the release every week. (It's ok to skip the weekly build if the
-   branch was updated in the last 24 hours.)
+### New Stable
 
-3. Forward-port the ChangeLog (and ReleaseNotes if appropriate) to the
-   main branch.
+   1. Create the `maint-x.y.z` and `release-x.y.z` branches and update the
+      `./scripts/git/git-list-tor-branches.sh` with the new version.
 
-4. Keep an eye on the blog post, to moderate comments and answer questions.
 
 ## Appendix: An alternative means to notify packagers
 
diff --git a/doc/HACKING/ReleasingTor.md.old b/doc/HACKING/ReleasingTor.md.old
new file mode 100644
index 0000000000..490c100fcb
--- /dev/null
+++ b/doc/HACKING/ReleasingTor.md.old
@@ -0,0 +1,255 @@
+# CHECKLIST
+
+Here's a summary checklist, with the things that Nick messes up most often.
+
+Did you:
+
+ * [ ] Copy the ChangeLog to the ReleaseNotes?
+ * [ ] Check that the new versions got approved?
+ * [ ] Check the release date in the ChangeLog?
+ * [ ] Update the GeoIP file?
+
+# Putting out a new release
+
+Here are the steps that the maintainer should take when putting out a
+new Tor release:
+
+## 0. Preliminaries
+
+1. Get at least three of weasel/arma/Sebastian/Sina to put the new
+   version number in their approved versions list.  Give them a few
+   days to do this if you can.
+
+2. If this is going to be an important security release, give the packagers
+   advance warning, via `tor-packagers@lists.torproject.org`.
+
+
+3. Given the release date for Tor, ask the TB team about the likely release
+   date of a TB that contains it.  See note below in "commit, upload,
+   announce".
+
+## I. Make sure it works
+
+1. Make sure that CI passes: have a look at the branches on gitlab.
+
+   _Optionally_, have a look at Travis
+   (https://travis-ci.org/torproject/tor/branches), Appveyor
+   (https://ci.appveyor.com/project/torproject/tor/history), and
+   Jenkins (https://jenkins.torproject.org/view/tor/).
+   Make sure you're looking at the right branches.
+
+   If there are any unexplained failures, try to fix them or figure them
+   out.
+
+2. Verify that there are no big outstanding issues.  You might find such
+   issues --
+
+    * On Gitlab
+
+    * On coverity scan
+
+    * On OSS-Fuzz
+
+## II. Write a changelog
+
+
+1a. (Alpha release variant)
+
+   Gather the `changes/*` files into a changelog entry, rewriting many
+   of them and reordering to focus on what users and funders would find
+   interesting and understandable.
+
+   To do this, run `./scripts/maint/sortChanges.py changes/* > changelog.in`
+   to combine headings and sort the entries.  Copy the changelog.in file into
+   the ChangeLog.  Run `format_changelog.py --inplace` (see below) to clean up
+   the line breaks.
+
+   Remove the `changes/*` files that you just merged into the ChangeLog.
+
+   After that, it's time to hand-edit and fix the issues that
+   lintChanges can't find:
+
+   1. Within each section, sort by "version it's a bugfix on", else by
+      numerical ticket order.
+
+   2. Clean them up:
+
+      Make stuff very terse
+
+      Describe the user-visible problem right away
+
+      Mention relevant config options by name.  If they're rare or unusual,
+      remind people what they're for
+
+      Avoid starting lines with open-paren
+
+      Present and imperative tense: not past.
+
+      "Relays", not "servers" or "nodes" or "Tor relays".
+
+      "Onion services", not "hidden services".
+
+      "Stop FOOing", not "Fix a bug where we would FOO".
+
+      Try not to let any given section be longer than about a page. Break up
+      long sections into subsections by some sort of common subtopic. This
+      guideline is especially important when organizing Release Notes for
+      new stable releases.
+
+      If a given changes stanza showed up in a different release (e.g.
+      maint-0.2.1), be sure to make the stanzas identical (so people can
+      distinguish if these are the same change).
+
+   3. Clean everything one last time.
+
+   4. Run `./scripts/maint/format_changelog.py --inplace` to make it prettier
+
+1b. (old-stable release variant)
+
+   For stable releases that backport things from later, we try to compose
+   their releases, we try to make sure that we keep the changelog entries
+   identical to their original versions, with a "backport from 0.x.y.z"
+   note added to each section.  So in this case, once you have the items
+   from the changes files copied together, don't use them to build a new
+   changelog: instead, look up the corrected versions that were merged
+   into ChangeLog in the main branch, and use those.
+
+   Add "backport from X.Y.Z" in the section header for these entries.
+
+2. Compose a short release blurb to highlight the user-facing
+   changes. Insert said release blurb into the ChangeLog stanza. If it's
+   a stable release, add it to the ReleaseNotes file too. If we're adding
+   to a release-* branch, manually commit the changelogs to the later
+   git branches too.
+
+3. If there are changes that require or suggest operator intervention
+   before or during the update, mail operators (either dirauth or relays
+   list) with a headline that indicates that an action is required or
+   appreciated.
+
+4. If you're doing the first stable release in a series, you need to
+   create a ReleaseNotes for the series as a whole.  To get started
+   there, copy all of the Changelog entries from the series into a new
+   file, and run `./scripts/maint/sortChanges.py` on it.  That will
+   group them by category.  Then kill every bugfix entry for fixing
+   bugs that were introduced within that release series; those aren't
+   relevant changes since the last series.  At that point, it's time
+   to start sorting and condensing entries.  (Generally, we don't edit the
+   text of existing entries, though.)
+
+## III. Making the source release.
+
+1. In `maint-0.?.x`, bump the version number in `configure.ac` and run
+   `./scripts/main/update_versions.py` to update version numbers in other
+   places, and commit.  Then merge `maint-0.?.x` into `release-0.?.x`.
+
+   When you merge the maint branch forward to the next maint branch, or into
+   main, merge it with `-s ours` to avoid conflict with the version
+   bump.
+
+2. In `release-0.?.x`, run `make distcheck`, put the tarball up in somewhere
+   (how about your homedir on people.torproject.org?) , and tell `#tor-dev`
+   about it.
+
+   If you want, wait until at least one person has built it
+   successfully.  (We used to say "wait for others to test it", but our
+   CI has successfully caught these kinds of errors for the last several
+   years.)
+
+3. Make sure that the new version is recommended in the latest consensus.
+   (Otherwise, users will get confused when it complains to them
+   about its status.)
+
+   If it is not, you'll need to poke Roger, Weasel, Sebastian, and Sina
+   again: see the note at the start of the document.
+
+## IV. Commit, upload, announce
+
+1. Sign the tarball, then sign and push the git tag:
+
+```console
+$ gpg -ba <the_tarball>
+$ git tag -s tor-0.4.x.y-<status>
+$ git push origin tag tor-0.4.x.y-<status>
+```
+
+   (You must do this before you update the website: the website scripts
+   rely on finding the version by tag.)
+
+   (If your default PGP key is not the one you want to sign with, then say
+   "-u <keyid>" instead of "-s".)
+
+2. scp the tarball and its sig to the dist website, i.e.
+   `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run
+   "static-update-component dist.torproject.org" on dist-master.
+
+   In the `project/web/tpo.git` repository, update `databags/versions.ini`
+   to note the new version.  Push these changes to `master`.
+
+   (NOTE: Due to #17805, there can only be one stable version listed at
+   once.  Nonetheless, do not call your version "alpha" if it is stable,
+   or people will get confused.)
+
+   (NOTE: It will take a while for the website update scripts to update
+   the website.)
+
+3. Email the tor-packagers@lists.torproject.org mailing list to tell them
+   about the new release.
+
+   Also, email tor-packagers@lists.torproject.org.
+
+   Mention where to download the tarball (`https://dist.torproject.org/`).
+
+   Include a link to the changelog.
+
+4. Wait for the download page to be updated. (If you don't do this before you
+   announce, people will be confused.)
+
+5. Mail the release blurb and ChangeLog to tor-talk (development release) or
+   tor-announce (stable).
+
+   Post the changelog on the blog as well. You can generate a
+   blog-formatted version of the changelog with
+      `./scripts/maint/format_changelog.py -B`
+
+   When you post, include an estimate of when the next TorBrowser
+   releases will come out that include this Tor release.  This will
+   usually track https://wiki.mozilla.org/RapidRelease/Calendar , but it
+   can vary.
+
+   For templates to use when announcing, see:
+       https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/AnnouncementTemplates
+
+## V. Aftermath and cleanup
+
+1. If it's a stable release, bump the version number in the
+   `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`
+   merge to avoid taking that change into main.
+
+2. If there is a new `maint-x.y.z` branch, create a Travis CI cron job that
+   builds the release every week. (It's ok to skip the weekly build if the
+   branch was updated in the last 24 hours.)
+
+3. Forward-port the ChangeLog (and ReleaseNotes if appropriate) to the
+   main branch.
+
+4. Keep an eye on the blog post, to moderate comments and answer questions.
+
+## Appendix: An alternative means to notify packagers
+
+If for some reason you need to contact a bunch of packagers without
+using the publicly archived tor-packagers list, you can try these
+people:
+
+       - {weasel,sysrqb,mikeperry} at torproject dot org
+       - {blueness} at gentoo dot org
+       - {paul} at invizbox dot io
+       - {vincent} at invizbox dot com
+       - {lfleischer} at archlinux dot org
+       - {Nathan} at freitas dot net
+       - {mike} at tig dot as
+       - {tails-rm} at boum dot org
+       - {simon} at sdeziel.info
+       - {yuri} at freebsd.org
+       - {mh+tor} at scrit.ch
+       - {security} at brave.com