WIP: Add new custom event loop for I/O layer

Introduce ev.h and ev.c, establishing the foundation for the new custom event loop, `pgagroal_ev`. Replace previous dependencies on libev with the custom event loop. For Linux, implement support for io_uring with a fallback to epoll if io_uring is unavailable. For BSD, implement support for kqueue.
agroal · Nov 4, 2024 · bade190 · bade190
1 parent 7298300
commit bade190
Show file tree

Hide file tree

Showing 37 changed files with 2,752 additions and 557 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -23,9 +23,9 @@ What is the version of pgagroal ?
 
 What is the version of PostgreSQL ?
 
-**libev**
+**liburing**
 
-What is the version of libev ?
+What is the version of liburing ?
 
 **OpenSSL**
 

diff --git a/.github/config/pg_hba.conf b/.github/config/pg_hba.conf
@@ -0,0 +1,4 @@
+local   all             all trust
+host    all             all all   trust
+local   replication     all peer
+host    replication     all all   trust
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -51,19 +51,19 @@ set(SUPPORTED_COMPILERS "GNU" "Clang" "AppleClang")
 
 # Check for a supported compiler
 if (NOT CMAKE_C_COMPILER_ID IN_LIST SUPPORTED_COMPILERS)
-   message(FATAL_ERROR "Unsupported compiler ${CMAKE_C_COMPILER_ID}. Supported compilers are: ${SUPPORTED_COMPILERS}")
+  message(FATAL_ERROR "Unsupported compiler ${CMAKE_C_COMPILER_ID}. Supported compilers are: ${SUPPORTED_COMPILERS}")
 endif ()
 
 CHECK_C_COMPILER_FLAG("-std=c17" COMPILER_SUPPORTS_C17)
 if(NOT COMPILER_SUPPORTS_C17)
   message(FATAL_ERROR "The compiler ${CMAKE_C_COMPILER} has no C17 support. Please use a different C compiler.")
 endif()
 
-find_package(Libev 4.11)
-if (LIBEV_FOUND)
-  message(STATUS "libev found")
-else ()
-  message(FATAL_ERROR "libev needed")
+find_package(Liburing 2.5)
+if (LIBURING_FOUND)
+  message(STATUS "liburing found")
+else()
+  message(STATUS "liburing NOT found")
 endif()
 
 find_package(OpenSSL)

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -34,7 +34,7 @@ Don't forget to indicate your pgagroal version.
 You can use the follow command, if you are using a [Fedora](https://getfedora.org/) based platform:
 
 ```
-dnf install git gcc cmake make libev libev-devel openssl openssl-devel systemd systemd-devel python3-docutils
+dnf install git gcc cmake make openssl openssl-devel systemd systemd-devel python3-docutils liburing liburing-devel
 ```
 
 in order to get the necessary dependencies.

diff --git a/README.md b/README.md
@@ -35,7 +35,7 @@ See [Performance](./doc/PERFORMANCE.md) for a performance run.
 
 * Process model
 * Shared memory model across processes
-* [libev](http://software.schmorp.de/pkg/libev.html) for fast network interactions
+* [liburing](https://github.com/axboe/liburing) for fast network interactions
 * [Atomic operations](https://en.cppreference.com/w/c/atomic) are used to keep track of state
 * The [PostgreSQL](https://www.postgresql.org) native protocol
   [v3](https://www.postgresql.org/docs/11/protocol-message-formats.html) for its communication
@@ -61,7 +61,7 @@ See [Architecture](./doc/ARCHITECTURE.md) for the architecture of [**pgagroal**]
 * [gcc 8+](https://gcc.gnu.org) (C17)
 * [cmake](https://cmake.org)
 * [make](https://www.gnu.org/software/make/)
-* [libev](http://software.schmorp.de/pkg/libev.html)
+* [liburing](https://github.com/axboe/liburing)
 * [OpenSSL](http://www.openssl.org/)
 * [systemd](https://www.freedesktop.org/wiki/Software/systemd/)
 * [rst2man](https://docutils.sourceforge.io/)
@@ -72,13 +72,13 @@ On Rocky Linux (and similar) operating systems, the dependencies
 can be installed via `dnf(8)` as follows:
 
 ```sh
-dnf install git gcc cmake make    \
-            libev libev-devel     \
-            openssl openssl-devel \
-            systemd systemd-devel \
-            python3-docutils      \
-            libatomic             \
-            cjson cjson-devel
+dnf install git gcc cmake make      \
+            openssl openssl-devel   \
+            systemd systemd-devel   \
+            python3-docutils        \
+            libatomic               \
+            cjson cjson-devel       \
+            liburing liburing-devel
 ```
 
 Please note that, on Rocky Linux, in order to install the `python3-docutils`

diff --git a/cmake/FindLibev.cmake b/cmake/FindLibev.cmake
diff --git a/cmake/FindLiburing.cmake b/cmake/FindLiburing.cmake
@@ -0,0 +1,18 @@
+# - Try to find liburing
+# Once done this will define
+#  LIBURING_FOUND   - System has liburing
+#  LIBURING_LIBRARY - The library needed to use liburing
+
+FIND_LIBRARY(LIBURING_LIBRARY NAMES liburing liburing.a liburing.so liburing.so.2
+   HINTS
+   /usr/lib64
+   /usr/lib
+   /lib64
+   /lib
+)
+
+IF (LIBURING_LIBRARY)
+   SET(LIBURING_FOUND TRUE)
+ELSE ()
+   SET(LIBURING_FOUND FALSE)
+ENDIF ()
diff --git a/doc/ARCHITECTURE.md b/doc/ARCHITECTURE.md
@@ -100,10 +100,18 @@ AuthenticationSASLFinal and AuthenticationOk. The SSLRequest message is supporte
 
 The remote management interface is defined in [remote.h](../src/include/remote.h) ([remote.c](../src/libpgagroal/remote.c)).
 
-## libev usage
+## I/O layer
 
-[libev](http://software.schmorp.de/pkg/libev.html) is used to handle network interactions, which is "activated"
-upon an `EV_READ` event.
+The I/O layer interface is primarily defined in [ev.h](../src/include/ev.h) (and implemented in [ev.c](../src/libpgagroal/ev.c)).
+
+These files contain the definition and implementation of the event loop for the three supported backends:
+io_uring, epoll, and kqueue.
+
+The backend is defined during runtime and can be set with the configuration option `ev_backend`. 
+Default is `auto`, which will select the first supported backend, considering the following order: 
+io_uring, epoll, kqueue.
+
+[liburing](https://github.com/axboe/liburing) was used for setup and usage io_uring instances.
 
 Each process has its own event loop, such that the process only gets notified when data related only to that process
 is ready. The main loop handles the system wide "services" such as idle timeout checks and so on.
@@ -147,7 +155,7 @@ The functions `start`, `client`, `server` and `stop` has access to the following
 ```C
 struct worker_io
 {
-   struct ev_io io;      /* The libev base type */
+   struct ev_io io;      /* The base type for io operations */
    int client_fd;        /* The client descriptor */
    int server_fd;        /* The server descriptor */
    int slot;             /* The slot */
@@ -235,7 +243,7 @@ The `SIGHUP` signal will trigger a reload of the configuration.
 However, some configuration settings requires a full restart of [**pgagroal**](https://github.com/agroal/pgagroal) in order to take effect. These are
 
 * `hugepage`
-* `libev`
+* `ev_backend`
 * `log_path`
 * `log_type`
 * `max_connections`

diff --git a/doc/CONFIGURATION.md b/doc/CONFIGURATION.md
@@ -78,7 +78,7 @@ The available keys and their accepted values are reported in the table below.
 | tls_cert_file | | String | No | Certificate file for TLS. This file must be owned by either the user running pgagroal or root. |
 | tls_key_file | | String | No | Private key file for TLS. This file must be owned by either the user running pgagroal or root. Additionally permissions must be at least `0640` when owned by root or `0600` otherwise. |
 | tls_ca_file | | String | No | Certificate Authority (CA) file for TLS. This file must be owned by either the user running pgagroal or root.  |
-| libev | `auto` | String | No | Select the [libev](http://software.schmorp.de/pkg/libev.html) backend to use. Valid options: `auto`, `select`, `poll`, `epoll`, `iouring`, `devpoll` and `port` |
+| ev_backend | `auto` | String | No | Select the event handling backend to use (`auto`, `io_uring`, `epoll`, and `kqueue`) |
 | keep_alive | on | Bool | No | Have `SO_KEEPALIVE` on sockets |
 | nodelay | on | Bool | No | Have `TCP_NODELAY` on sockets |
 | non_blocking | off | Bool | No | Have `O_NONBLOCK` on sockets |

diff --git a/doc/DEVELOPERS.md b/doc/DEVELOPERS.md
@@ -17,7 +17,7 @@ dnf install postgresql-server
 #### Basic dependencies
 
 ``` sh
-dnf install git gcc cmake make libev libev-devel openssl openssl-devel systemd systemd-devel python3-docutils libatomic cjson cjson-devel
+dnf install git gcc cmake make openssl openssl-devel systemd systemd-devel python3-docutils libatomic cjson cjson-devel liburing liburing-devel
 ```
 
 #### Generate user and developer guide

diff --git a/doc/etc/pgagroal.conf b/doc/etc/pgagroal.conf
@@ -10,6 +10,7 @@ max_connections = 100
 idle_timeout = 600
 validation = off
 unix_socket_dir = /tmp/
+ev_backend = auto
 
 [primary]
 host = localhost

diff --git a/doc/man/pgagroal.conf.5.rst b/doc/man/pgagroal.conf.5.rst
@@ -146,8 +146,8 @@ tls_key_file
 tls_ca_file
   Certificate Authority (CA) file for TLS. Changes require restart in the server section.
 
-libev
-  The libev backend to use. Valid options: auto, select, poll, epoll, iouring, devpoll and port. Default is auto
+ev_backend
+  The event handling backend to use. Valid options are auto, io_uring, epoll, and kqueue. Default is auto
 
 keep_alive
   Have SO_KEEPALIVE on sockets. Default is on

diff --git a/doc/manual/02-installation.md b/doc/manual/02-installation.md
@@ -41,19 +41,20 @@ We recommend using Fedora to test and run [**pgagroal**][pgagroal], but other Li
 * [gcc 8+](https://gcc.gnu.org) (C17)
 * [cmake](https://cmake.org)
 * [make](https://www.gnu.org/software/make/)
-* [libev](http://software.schmorp.de/pkg/libev.html)
+* [liburing](https://github.com/axboe/liburing)
 * [OpenSSL](http://www.openssl.org/)
 * [systemd](https://www.freedesktop.org/wiki/Software/systemd/)
 * [rst2man](https://docutils.sourceforge.io/)
 * [libatomic](https://gcc.gnu.org/wiki/Atomic)
 * [cJSON](https://github.com/DaveGamble/cJSON)
 
 ```sh
-dnf install git gcc cmake make libev libev-devel \
-            openssl openssl-devel \
-            systemd systemd-devel \
-            python3-docutils libatomic \
-            cjson cjson-devel
+dnf install git gcc cmake make          \
+            openssl openssl-devel       \
+            systemd systemd-devel       \
+            python3-docutils libatomic  \
+            cjson cjson-devel           \
+            liburing liburing-devel
 ```
 
 Alternative [clang 8+](https://clang.llvm.org/) can be used.
@@ -101,7 +102,7 @@ On FreeBSD, `pkg` is used instead of `dnf` or `yum`.
 Use `pkg install <package name>` to install the following packages
 
 ``` sh
-git gcc cmake libev openssl libssh py39-docutils libcjson
+git gcc cmake openssl libssh py39-docutils libcjson
 ```
 
 ### Build

diff --git a/doc/manual/99-references.md b/doc/manual/99-references.md
@@ -10,7 +10,7 @@
   [gcc]: https://gcc.gnu.org
   [cmake]: https://cmake.org
   [make]: https://www.gnu.org/software/make/
-  [libev]: http://software.schmorp.de/pkg/libev.html
+  [liburing]: https://github.com/axboe/liburing
   [openssl]: http://www.openssl.org/
   [systemd]: https://www.freedesktop.org/wiki/Software/systemd/
   [rst2man]: https://docutils.sourceforge.io/

diff --git a/doc/manual/dev-02-architecture.md b/doc/manual/dev-02-architecture.md
@@ -102,10 +102,14 @@ AuthenticationSASLFinal and AuthenticationOk. The SSLRequest message is supporte
 
 The remote management interface is defined in [remote.h](../src/include/remote.h) ([remote.c](../src/libpgagroal/remote.c)).
 
-## libev usage
+## I/O Layer
 
-[libev](http://software.schmorp.de/pkg/libev.html) is used to handle network interactions, which is "activated"
-upon an `EV_READ` event.
+The I/O layer interface is primarily defined in [ev.h](../src/include/ev.h) (and implemented in [ev.c](../src/libpgagroal/ev.c)).
+
+These files contain the definition and implementation of the event loop for the three supported backends:
+`io_uring`, `epoll`, and `kqueue`.
+
+[liburing](https://github.com/axboe/liburing) was used for setup and usage io_uring instances.
 
 Each process has its own event loop, such that the process only gets notified when data related only to that process
 is ready. The main loop handles the system wide "services" such as idle timeout checks and so on.
@@ -149,7 +153,7 @@ The functions `start`, `client`, `server` and `stop` has access to the following
 ```C
 struct worker_io
 {
-   struct ev_io io;      /* The libev base type */
+   struct ev_io io;      /* The base type for io operations */
    int client_fd;        /* The client descriptor */
    int server_fd;        /* The server descriptor */
    int slot;             /* The slot */
@@ -237,7 +241,6 @@ The `SIGHUP` signal will trigger a reload of the configuration.
 However, some configuration settings requires a full restart of [**pgagroal**](https://github.com/agroal/pgagroal) in order to take effect. These are
 
 * `hugepage`
-* `libev`
 * `log_path`
 * `log_type`
 * `max_connections`

diff --git a/doc/manual/user-02-configuration.md b/doc/manual/user-02-configuration.md
@@ -60,7 +60,7 @@ The available keys and their accepted values are reported in the table below.
 | tls_cert_file | | String | No | Certificate file for TLS. This file must be owned by either the user running pgagroal or root. |
 | tls_key_file | | String | No | Private key file for TLS. This file must be owned by either the user running pgagroal or root. Additionally permissions must be at least `0640` when owned by root or `0600` otherwise. |
 | tls_ca_file | | String | No | Certificate Authority (CA) file for TLS. This file must be owned by either the user running pgagroal or root.  |
-| libev | `auto` | String | No | Select the [libev](http://software.schmorp.de/pkg/libev.html) backend to use. Valid options: `auto`, `select`, `poll`, `epoll`, `iouring`, `devpoll` and `port` |
+| ev_backend | `auto` | String | No | Select the event handling backend to use (`auto`, `io_uring`, `epoll`, and `kqueue`) |
 | keep_alive | on | Bool | No | Have `SO_KEEPALIVE` on sockets |
 | nodelay | on | Bool | No | Have `TCP_NODELAY` on sockets |
 | non_blocking | off | Bool | No | Have `O_NONBLOCK` on sockets |

diff --git a/pgagroal.spec b/pgagroal.spec
@@ -7,8 +7,8 @@ URL:           https://github.com/agroal/pgagroal
 Source0:       https://github.com/agroal/pgagroal/archive/%{version}.tar.gz
 
 BuildRequires: gcc cmake make python3-docutils
-BuildRequires: libev libev-devel openssl openssl-devel systemd systemd-devel libatomic cjson cjson-devel
-Requires:      libev openssl systemd libatomic cjson
+BuildRequires: liburing liburing-devel openssl openssl-devel systemd systemd-devel libatomic cjson cjson-devel
+Requires:      liburing openssl systemd libatomic cjson
 
 %description
 pgagroal is a high-performance connection pool for PostgreSQL.