can we use systemd services instead of rc.local to run ansible-pull-script.sh once on boot ?

[martbhell]

It would be good if ansible runs at the end of the systemd boot sequence, not at the rc.local (which runs before many services like ypbind, slurm and fs mounts, see "systemd-analyze plot > plot.svg" output).

[martbhell]

Related: fgci-org/ansible-role-cuda#5

[A1ve5]

See fgci-org/ansible-role-cuda#5

[martbhell]

Reopened and renaming as the cuda part is now done. Guess I shouldn't have two issues in the same issue..

[martbhell]

fgci-org/ansible-role-cuda@f93fd8a cuda_init.sh now uses a systemd service oneshot and that seems to work.

• If we were to run ansible-pull as a systemd service - what point in boot do we want it to start at?
• Right now rc.local and its scripts are started after network.target and it's type is forking. (/usr/lib/systemd/system/rc-local.service)
• Making it start after ypbind, network-online, maybe more? But we should make sure that the service starts even if those services are not yet installed.
• Do we want to keep Type=forking or do we want to start some services after ansible-pull? This will increase time before the node is available in slurm.

[martbhell]

An idea: maybe if the ansible-pull-script.sh creates for example /etc/FGCI (once, after first success of the ansible-pull command) and then if that file exists then the systemd service does not wait and if /etc/FGCI does not exist then it waits at that point in boot?

[martbhell]

To add this we could:
A) add the service in fgci-install and remove the line added to rc.local by pxe_config during kickstart
B) change pxe_config to install the service during kickstart

One pro for B is that there won't be a confusing rc.local entry sometimes
One con for B is that we'd have to modify the kickstart template :)

Thoughts? Is A OK?

[martbhell]

/etc/FGCI added in fgci-org/ansible-role-fgci-install@ac6e092

[jabl]

Here's a blog post that provides a systemd service to run ansible-pull on boot:

https://calgaryrhce.ca/blog/2016/02/03/ansible-pull-and-kickstart-for-one-touch-server-provisioning/

[martbhell]

The fgci-install role can now install the ansible-pull script. Some thought is still required to think about when exactly to run it during boot and how to make it non-blocking when the node has already been configured once with ansible (because on some sites it takes ~an hour for ansible-pull to run)

[martbhell]

We now install a watermark file https://github.com/CSCfi/fgci-ansible/tree/master/roles/watermark-release so that could/should be used when doing this.