Support Wayland as a backend

[sminez]

⚠️ This tracking issue is intended as an initial attempt at providing Wayland support for Penrose ⚠️

There are likely going to be a number of internal API changes required to get this to work (not least the fact that XConn will no longer be a good name for the trait that defines the back end we are relying on!)
For now, take this as an investigation into what would be required to get this all hooked up rather than a commitment to actually pulling it off 😉

Overall goal

The current design of Penrose involves a relatively clean split between "pure" logic that describes the user facing behaviour of the window manager itself, and the "X" back end layer that talks to the X server in order to actually do the heavy lifting. Ideally this split should be maintained when supporting a Wayland back end (this I feel should be doable no matter what) but there are going to be some changes required, at least in terms of naming, around some of the user facing functionality that reaches down to the X layer.

As a simple example, manage hooks often rely on an X Query to determine whether or not they should run. The actual interaction with the X server is handled by the XConn implementation but X concepts such as class names and properties are used in order to write those queries. This is fine so long as the XConn trait has the semantics of "interface to an X server" rather than "interface to an arbitrary back end". This is one area of the Penrose design that I've never been particularly satisfied with as it forces users to care about the underlying X layer when (really) they shouldn't have to care one way or the other.

As a starting point, it's probably going to be best to design a modified trait to replace XConn that is capable of supporting Wayland as a back end as well as X. Getting that in place and updating Penrose to make use of it should then mean that we are "just" left with the task of implementing that back end, rather than having to overhaul Penrose to only support Wayland.

Fist steps

• Write up the required interfaces for a Wayland back end so that the new trait can be defined
• Implement the new trait using x11rb
• Wire in the new trait, replacing XConn
  • This should then be released as 0.4.0
• Update the docs and book to cover the new APIs

Future work

In addition to the API changes required to support the Wayland protocol as well as X11, there is the primary reason I haven't tackled this sooner: graphics. A Wayland compositor is responsible for significantly more than an X11 window manager so there is going to be a large amount of new code needed for handling that side of things. (This is an area where I am hoping to lean on a library such as Smithay if possible).

Beyond that, there are a number of extensions and pieces of functionality that may or may not need to be implemented inside of Penrose itself.

References

As a starting point, we need an understanding the differences between the X11 protocols and Wayland protocols and how that will affect the internal APIs of Penrose. I've made some progress with this but not enough to be able to confidently say what the required changes will look like yet.

• Wayland itself has a variety of documentation available:
  • https://wayland.freedesktop.org/
  • https://wayland.app/protocols/
  • https://wayland-book.com/
    • This seems to be broken at the moment? Not sure what's up...

In terms of a library layer to build on top of (as opposed to implementing the wire protocols myself) I'm most likely going to go with Smithay. Smithay is getting closer to its 0.4 release so lets see where things currently stand:

• smithay itself: https://github.com/Smithay/smithay
  • wayland-rs (lower level crates): https://github.com/Smithay/wayland-rs
  • calloop (the event loop): https://github.com/Smithay/calloop
  • input.rs (bindings to libinput): https://github.com/Smithay/input.rs
• smithay mainline docs: https://smithay.github.io/smithay/smithay/index.html
• existing compositors built ontop of smithay:
  • cosmic-comp: https://github.com/pop-os/cosmic-comp
  • niri: https://github.com/YaLTeR/niri
  • magamaWM: https://github.com/MagmaWM/MagmaWM
  • strata: https://github.com/StrataWM/strata
  • pinnacle: https://github.com/pinnacle-comp/pinnacle

Outside of the compositors written on top of Smithay, there are several that seem worthwhile looking at as a reference for how a working Wayland compositor is structured. I've not tried any of these out personally so this is more as a pointer to ways of approaching certain problems rather than known good implementations that should be taken as the "correct" way to do things. From what I can find online, the state of the art for wayland compositors is still pretty variable so I'm unlikely to find a relatively stable implementation I can refer to like dwm or xmonad.

• dwl: https://codeberg.org/dwl/dwl
• hyprland: https://github.com/hyprwm/Hyprland
• sway: https://github.com/swaywm/sway
• tinywl+: https://github.com/keshto/tinywl_plus
• vivarium: https://github.com/inclement/vivarium

And finally, it's probably worthwhile also listing out alternative building blocks / back ends to evaluate:

• Mir: https://github.com/MirServer/mir
• swc: https://github.com/michaelforney/swc
• wlroots: https://gitlab.freedesktop.org/wlroots/wlroots

[nabajour]

Looking forward to this! I played a bit with penrose when looking for a programmable WM to program with rust, but then switched my system over to wayland, and haven't found any programmable compositor that I could get to the point I was with xmonad (despite my lack of haskell programming knowledge).

Some info that can be useful to you:

• I also started reading the wayland book, it's from the guy who wrote wlroots. It looks like it's hosted on SourceHut, which seems to be currently offline because of a massive DDOS attack.
• there is also jay that works as a programmable compositor to build one's tiling wm that I'm playing with.

[sminez]

Thanks for the links @nabajour! Particularly the context around why the wayland book is currently offline (hopefully the source is available somewhere so I can nab an offline copy for future reference 🤞) I'd spotted jay previously but hadn't included it in my list as it looks like it handles implementing the Wayland protocol itself which probably isn't the way I'm going to go with this? Still worth a look though, just might be a little harder to unpick the pieces I want to learn from!

If I'm able to pull off what I'm aiming for then you should be able to swap out the X back end in Penrose for a new Wayland one and mostly keep the rest of your setup unchanged. I need to read up and experiment with things around how interacting with client programs works (particularly the sorts of things that you might do with something like xdotool) before I can settle on the new trait that will be replacing XConn. If you have any tips or links along those lines I'd appreciate a steer in right direction 🙂

[nabajour]

hopefully the source is available somewhere so I can nab an offline copy for future reference

The wayland book is still offline, but it looks like the source is available on Drew Devault's sourcehut. It's buildable with mdbook.

I'd spotted jay previously but hadn't included it in my list as it looks like it handles implementing the Wayland protocol itself which probably isn't the way I'm going to go with this? Still worth a look though, just might be a little harder to unpick the pieces I want to learn from!

It's interesting to see how stuff is done from scratch, but yes, smithay might be a better starting point, or something based on wlroots. I saw some things missing in jay that depend on extensions, so those might need to be reimplemented from scratch in this case...

[SomeGuyNamedMay]

qtile: https://github.com/qtile/qtile may be a good reference as well as they have a both a wayland and x11 backend