Motif hints: Respect maximum border style configuration set by user

Context:
Motif hints [1] allow applications to request specific window manager
frame decorations. Most applications like alacritty, chromium, and
godot, use the hints as a binary flag, setting or un-setting
`MWM_DECOR_ALL`.
Previously [2], we had disallowed applications to set the "normal"
border style through motif hints. This effectively meant that users that
had set `default_border pixel` would not see applications spawning with
normal decorations [3].
However, that meant that applications like godot [4] could not toggle
their border between none and normal so the behaviour changed with
v4.21 [5].
That change however also allowed applications to override the default
none/pixel border style the user set. For example, alacritty can be
configured to either have all or no decorations [6] and they always set
the motif hint on startup, completely overriding i3 user's preference:
1. If decorations are disabled with alacritty's config then they will
   override `default_border normal` and no title will be used.
2. If decorations are enabled (also the default behavior) with
   alacritty's config then they will override `default_border pixel` and
   a title will be used.

This patch redefines how we interpret motif hints. When a client sets
`MWM_DECOR_ALL`, we interpret it as "the maximum decoration the user has
allowed for this window". I.e., if a client was all decorations and the
user expects the window to not have a title, we don't include the title
in "all" decorations.

The user's preference is determined by these:
1. For new tiling windows, as set by `default_border`
2. For new floating windows, as set by `default_floating_border`
3. For all windows that the user runs the `border` command, whatever is
   the result of that command for that window.
Example:
- User opens new tiling window with `default_border pixel` => maximum
  decoration = PIXEL
- Window requests all/title decorations => i3 enforces the user maximum
  decoration, PIXEL (no change)
- Window requests no decorations => i3 accepts it and sets border to
  NONE, maximum decoration remains PIXEL
- User toggles the border, next style is NORMAL => maximum decoration is
  now NORMAL
- Window requests no decorations => i3 accepts it and sets border to
  NONE
- Window requests all/title decorations => i3 accepts it and sets the
  maximum border, NORMAL
- User toggles the border, next style is NONE => maximum decoration is
  now NONE
- Window requests all/title decorations => i3 enforces the user maximum
  decoration, NONE (no change)

With this, we will still allow behaviour where windows can toggle their
border style with motif hints [4][7].

Reference/footnotes:
[1]: https://linux.die.net/man/3/vendorshell
[2]: https://github.com/i3/i3/pull/2386
[3]: Notice how there is apparently a gap because `default border none`
settings would not be respected if an application wanted just "border"
decorations but this was never reported, probably because of the rare
conjunction of applications requesting that and users defaulting to none
borders.
[4]: https://github.com/godotengine/godot/issues/40037
[5]: https://github.com/i3/i3/pull/5135
[6]: Set by an underlying library here:
https://github.com/rust-windowing/winit/blob/fafdedfb7d3a7370ca4b01108f7713b685633164/src/platform_impl/linux/x11/util/hint.rs#L113-L142
called by alactitty here:
https://github.com/alacritty/alacritty/blob/4ddb608563d985060d69594d1004550a680ae3bd/alacritty/src/display/window.rs#L341
[7]: https://github.com/i3/i3/issues/3678

Closes #3678
Fixes #5149

2 files changed, 10 insertions, 1 deletions

diff --git a/include/con.h b/include/con.h
index b8bff080..8d344994 100644
--- a/include/con.h
+++ b/include/con.h
@@ -451,7 +451,7 @@ int con_border_style(Con *con);
  * floating window.
  *
  */
-void con_set_border_style(Con *con, int border_style, int border_width);
+void con_set_border_style(Con *con, border_style_t border_style, int border_width);
 
 /**
  * This function changes the layout of a given container. Use it to handle
diff --git a/include/data.h b/include/data.h
index 9bf43d0c..8078fd3e 100644
--- a/include/data.h
+++ b/include/data.h
@@ -723,7 +723,16 @@ struct Con {
      * layout in workspace_layout and creates a new split container with that
      * layout whenever a new container is attached to the workspace. */
     layout_t layout, last_split_layout, workspace_layout;
+
     border_style_t border_style;
+    /* When the border style of a con changes because of motif hints, we don't
+     * want to set more decoration that the user wants. The user's preference is determined by these:
+     * 1. For new tiling windows, as set by `default_border`
+     * 2. For new floating windows, as set by `default_floating_border`
+     * 3. For all windows that the user runs the `border` command, whatever is
+     * the result of that command for that window. */
+    border_style_t max_user_border_style;
+
     /** floating? (= not in tiling layout) This cannot be simply a bool
      * because we want to keep track of whether the status was set by the
      * application (by setting _NET_WM_WINDOW_TYPE appropriately) or by the