kmods-via-containers or KVC

kmods-via-containers is a framework for building and delivering kernel modules via containers. This implementation for this framework was inspired by the work done by Joe Doss on atomic-wireguard.

This framework relies on 3 independently developed pieces.

1. The kmods-via-containers code/config (this repo)

Delivers the stencil code and configuration files for building and delivering kmods via containers. It also delivers a service kmods-via-containers@.service that can be instantiated for each instance of the KVC framework.

2. The kernel module code that needs to be compiled

This repo represents the kernel module code that contains the source code for building the kernel module. This repo can be delivered by vendors and generally nothing about containers. Most importantly, if someone wanted to deliver this kernel module via the KVC framework, the owners of the code don't need to be consulted.

3. A KVC framework repo for the kernel module to be delivered

This repo defines a container build configuration as well as a library, userspace tools, and config files that need to be created on the host system. This repo does not have to be developed by the owner of the kernel module that is wanted to be delivered.

It must define a few functions in the bash library:

• build_kmods()
  • Performs the kernel module container build
• load_kmods()
  • Loads the kernel module(s)
• unload_kmods()
  • Unloads the kernel module(s)
• wrapper()
  • A wrapper function for userspace utilities

Example

Code Repositories

To give a full illustration of how to use this framework, a full example is worth a thousand words. In this example I will use

1. The kmods-via-containers software (this repo)
2. simple-kmod
   • A simple kmod source code repo
   • Contains the source code for two modules (simple-kmod and simple-procfs-kmod)
   • Contains the source code for one userspace tool (spkut)
     • Compiled from simple-procfs-kmod-userspace-tool.c
3. kvc-simple-kmod
   • An instance of a KVC framework repo that shows how to build and deliver the modules from the simple-kmod source code repository

So we'll build the modules (simple-kmod and simple-procfs-kmod) and the userspace tool (spkut) inside of a container by using the build_kmods() function from the library provided by the kvc-simple-kmod repo.

In this case build_kmods() calls out to the CONTAINER_RUNTIME defined by the kmods-via-containers.conf file (default of podman) to perform a container build using the KMOD_CONTAINER_BUILD_CONTEXT and the container build file specified by KMOD_CONTAINER_BUILD_FILE. Both of these vars are defined in the kvc-simple-kmod.conf file.

For kvc-simple-kmod the config file build content defaults to git://github.com/kmods-via-containers/kvc-simple-kmod.git and the build file defaults to Dockerfile.fedora.

There are a few other values defined in the config file of the kvc-simple-kmod example. Here are all of them:

• KMOD_CONTAINER_BUILD_CONTEXT="git://github.com/kmods-via-containers/kvc-simple-kmod.git"
• KMOD_CONTAINER_BUILD_FILE=Dockerfile.fedora
• KMOD_SOFTWARE_VERSION=dd1a7d4
• KMOD_NAMES="simple-kmod simple-procfs-kmod"

The KMOD_SOFTWARE_VERSION gives a clue to the library about what version of the simple_kmod softwre to use. This can be changed by the end user to test out different versions.

The KMOD_NAMES define the list of kernel modules that the user would like to be loaded/unloaded when the library is called.

Testing it out on a booted Classic Host

Install the kmods-via-containers files on your system by running make install. This will place the executable config file and service on your system.

git clone https://github.com/kmods-via-containers/kmods-via-containers
cd kmods-via-containers
sudo make install

Now reload systemd to read the systemd unit we just installed:

sudo systemctl daemon-reload

Install the kvc-simple-kmod KVC framework instance files on your system by running make install. This will install the KVC framework instance library and config file as well as the userspace wrapper onto the system.

git clone https://github.com/kmods-via-containers/kvc-simple-kmod
cd kvc-simple-kmod
sudo make install

Now instantiate an instance of kmods-via-containers@.service for simple-kmod:

sudo systemctl enable kmods-via-containers@simple-kmod.service

We can now either call the service to build and insert the kernel module(s) or we can wait until the next reboot when the service will detect there is no built module container and execute the build then. We can also call the kmods-via-containers script directly to view all of the output:

sudo kmods-via-containers build simple-kmod $(uname -r)

After building we can load and unload:

sudo kmods-via-containers load simple-kmod $(uname -r)
sudo kmods-via-containers unload simple-kmod $(uname -r)
lsmod | grep simple.*kmod
simple_procfs_kmod     16384  0
simple_kmod            16384  0

Which is roughly equivalent to start and stop:

sudo systemctl start kmods-via-containers@simple-kmod.service
sudo systemctl stop kmods-via-containers@simple-kmod.service

Once the modules are loaded we can view that they are loaded by looking at lsmod, dmesg and also interacting with the procfs file:

$ lsmod | grep simple.*kmod
simple_procfs_kmod     16384  0
simple_kmod            16384  0

$ dmesg | grep 'Hello world'
[ 6420.761332] Hello world from simple_kmod.

$ sudo cat /proc/simple-procfs-kmod 
simple-procfs-kmod number = 0

We can also use the spkut userspace utility to interact with the simple-procfs-kmod:

$ sudo spkut 44
KVC: wrapper simple-kmod for 5.3.7-301.fc31.x86_64
Running userspace wrapper using the kernel module container...
+ podman run -i --rm --privileged simple-kmod-dd1a7d4:5.3.7-301.fc31.x86_64 spkut 44
simple-procfs-kmod number = 0

simple-procfs-kmod number = 44

Testing it out on an OSTree based host provisioned via Ignition

First create a base ignition config that you'd like to use. It will contain the ssh pub key to add to the authorized keys file for the core user and also a systemd unit (require-simple-kmod.service that requires kmods-via-containers@simple-kmod.service. The systemd unit is a workaround for a an upstream bug and makes sure the kmods-via-containers@simple-kmod.service gets started on boot.

cat <<EOF > ./baseconfig.ign
{
  "ignition": { "version": "3.0.0" },
  "passwd": {
    "users": [
      {
        "name": "core",
        "groups": ["sudo"],
        "sshAuthorizedKeys": [
          "ssh-rsa AAAA"
        ]
      }
    ]
  },
  "systemd": {
    "units": [{
      "name": "require-kvc-simple-kmod.service",
      "enabled": true,
      "contents": "[Unit]\nRequires=kmods-via-containers@simple-kmod.service\n[Service]\nType=oneshot\nExecStart=/usr/bin/true\n\n[Install]\nWantedBy=multi-user.target"
    }]
  }
}
EOF

NOTE You'll need to add your public SSH key to that baseconfig.ign.

Next we'll create a fakeroot directory and populate it with files that we want to deliver via Ignition:

FAKEROOT=$(mktemp -d)
git clone https://github.com/kmods-via-containers/kmods-via-containers
cd kmods-via-containers
make install DESTDIR=${FAKEROOT}/usr/local CONFDIR=${FAKEROOT}/etc/
cd ..
git clone https://github.com/kmods-via-containers/kvc-simple-kmod
cd kvc-simple-kmod
make install DESTDIR=${FAKEROOT}/usr/local CONFDIR=${FAKEROOT}/etc/
cd ..

Now we'll use a tool call the filetranspiler to generate a final ignition config given the base ignition config and the fakeroot directory with files we'd like to deliver:

git clone https://github.com/ashcrow/filetranspiler
./filetranspiler/filetranspile -i ./baseconfig.ign -f ${FAKEROOT} -p -o config.ign

Now we can use this ignition config to start a Fedora CoreOS or RHEL CoreOS node and see the kmods-via-containers@simple-kmod.service and the kernel modules associated with simple-kmods get loaded.

You can check the modules are loaded with:

$ lsmod | grep simple
simple_procfs_kmod     16384  0
simple_kmod            16384  0

And run a userspace utility:

NOTE: sudo -i is necessary because otherwise /usr/local/bin won't be in the $PATH.

$ sudo -i /usr/local/bin/spkut 44
KVC: wrapper simple-kmod for 5.3.11-300.fc31.x86_64
Running userspace wrapper using the kernel module container...
+ podman run -i --rm --privileged simple-kmod-dd1a7d4:5.3.11-300.fc31.x86_64 spkut 44
simple-procfs-kmod number = 0

simple-procfs-kmod number = 44

Steps for OpenShift (RHCOS) via the MCO

Start with a base MCO yaml snippet that looks like:

cat <<EOF > mc-base.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 10-kvc-simple-kmod
spec:
  config:
EOF

Start with a base ignition config snippet that looks like the following. Note that we don't need an SSH key here like we did before because our openshift install already has configs for the SSH key defined.

cat <<EOF > ./baseconfig.ign
{
  "ignition": { "version": "2.2.0" },
  "systemd": {
    "units": [{
      "name": "require-kvc-simple-kmod.service",
      "enabled": true,
      "contents": "[Unit]\nRequires=kmods-via-containers@simple-kmod.service\n[Service]\nType=oneshot\nExecStart=/usr/bin/true\n\n[Install]\nWantedBy=multi-user.target"
    }]
  }
}
EOF

Next we'll create a fakeroot directory and populate it with files that we want to deliver via Ignition:

FAKEROOT=$(mktemp -d)
git clone https://github.com/kmods-via-containers/kmods-via-containers
cd kmods-via-containers
make install DESTDIR=${FAKEROOT}/usr/local CONFDIR=${FAKEROOT}/etc/
cd ..
git clone https://github.com/kmods-via-containers/kvc-simple-kmod
cd kvc-simple-kmod
make install DESTDIR=${FAKEROOT}/usr/local CONFDIR=${FAKEROOT}/etc/
cd ..

For RHCOS we'll need to switch the container build file to Dockerfile.rhel:

sed -i s/KMOD_CONTAINER_BUILD_FILE=Dockerfile.fedora/KMOD_CONTAINER_BUILD_FILE=Dockerfile.rhel/ $FAKEROOT/etc/kvc/simple-kmod.conf

And we'll also need to get entitlements onto our system so we can install protected content for the kernel module build. We can do this by pulling the entitlements from a RHEL 8 system. On a RHEL 8 system that has already been attached to subscription/entitlement you can package up the necessary data like:

[rhel8.host] # tar -czf subs.tar.gz /etc/pki/entitlement/ /etc/rhsm/ /etc/yum.repos.d/redhat.repo

Then copy subs.tar.gz from the RHEL 8 system and extract them into the fakeroot:

tar -x -C ${FAKEROOT} -f subs.tar.gz

Now we'll use a tool call the filetranspiler to generate a final ignition config given the base ignition config and the fakeroot directory with files we'd like to deliver. For this call to the filetranspiler we'll add two new arguments:

• --format=yaml to output yaml for the machineconfig
• --dereference-symlinks to workaround missing symlink support in the MCO

We'll also pipe that output into a sed command to indent the text by the appropriate amount so that we can append it to the mc-base.yaml. The appended file will be written to mc.yaml

git clone https://github.com/ashcrow/filetranspiler
./filetranspiler/filetranspile -i ./baseconfig.ign -f ${FAKEROOT} --format=yaml --dereference-symlinks | sed 's/^/     /' | (cat mc-base.yaml -) > mc.yaml

Now we can create a new machine config for the cluster:

oc create -f mc.yaml

After a period of time your nodes should have started the kmods-via-containers@simple-kmod.service service and the kernel modules should be loaded.