SaltStack Administration

Salt

We use SaltStack to configure our build master and the slaves. The sources are at: https://github.com/servo/saltfs/

Resources

Note that we're currently on the Salt 2015.5 release branch, so make sure to look at docs for the right version of Salt.

• the Salt docs

Deploying changes

Examples:

• salt '*' state.highstate - Apply all configs to all hosts
• salt 'servo-linux*' state.highstate test=True - See what would happen if we applied all configs to all servo-linux hosts

Master Setup

Minions find out where their master is by reading /etc/salt/minion. They attempt to connect to all masters listed in that file.

This issue tracks how we moved the build/salt master from Linode to AWS.

File configuration notes on servo-master1:

file_roots:
  base:
    - /srv/salt
    
pillar_roots:
  base:
    - /srv/pillar

Upgrading Salt

We're using a manual update process for now since there are so few machines: first on the salt master, then on the minions. Things to be aware of:

• It's necessary to restart the salt-master and salt-minion services to apply config changes, but restarting a master or minion service will interrupt an ongoing highstate. Make sure to run the highstate more than once to fully converge on changes.
• Masters need to be updated before minions, but salt '*' state.highstate cannot enforce ordering - make sure to update just the master first with salt 'servo-master1' state.highstate.

For the discussion about whether to use Salt to manage Salt upgrades and files, see this issue.

Adding Minions

Linux

⚠️ Setting up a master requires additional steps, these instructions only set up a minion.

Install the Salt minion:

# curl https://raw.githubusercontent.com/servo/saltfs/master/.travis/install_salt.sh | sh -s linux

Configure and start the Salt minion:

# sudo bash -c 'echo master: > /etc/salt/minion'
# sudo bash -c 'echo " - servo-master1.servo.org" >> /etc/salt/minion'
# sudo bash -c 'echo servo-linux10 > /etc/salt/minion_id' # replace servo-linux10 
# sudo service salt-minion start

OSX

Install latest Xcode and commandline tools from the Appstore, then verify the license:

sudo xcodebuild -license

If the app store version of xcode is incompatible with the OSX version, manually install it and run xcodebuild --install when connected via VNC viewer. The installation command creates pop-ups which must be clicked to successfully install xcode.

Install brew and git and pip:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install git
echo export PATH='/usr/local/bin:$PATH' >> ~/.bash_profile
source ~/.bash_profile
sudo easy_install -s /usr/bin pip

Download salt files:

curl https://raw.githubusercontent.com/servo/saltfs/master/.travis/install_salt.sh | sh -s osx
curl -O https://raw.githubusercontent.com/saltstack/salt/develop/pkg/darwin/com.saltstack.salt.minion.plist

Add the following to the .plist file just after the Label section:

     <key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
        <key>HOME</key>
        <string>/Users/administrator</string>
    </dict>

sudo cp com.saltstack.salt.minion.plist /Library/LaunchDaemons/
sudo bash -c "echo '52.37.76.55 salt' >> /etc/hosts"
sudo mkdir /etc/salt
sudo mkdir /var/log/salt
sudo salt-run launchd.write_launchd_plist salt-minion
printf 'id: servo-mac10' | sudo tee /etc/salt/minion # replace servo-linux10 with the builder name 
sudo launchctl load -w /Library/LaunchDaemons/com.saltstack.salt.minion.plist

Set screen resolution to 1280x960 or else reftests will fail randomly Create a standard user named 'Tom Servo', account name servo, give him a password, and log in. Set up to autologin on reboot. LEAVE LOGGED IN (else reftests will fail)

Do the following to stop coreaudiod from using 100% of a cpu:

sudo launchctl load -w /system/library/launchdaemons/com.apple.blued.plist

And do this to stop mds/mdworker from using another 100% of a cpu:

sudo launchctl unload -w /system/library/launchdaemons/com.apple.metadata.mds.plist 

Enabling a minion for the first time

On the master, use salt-key -L to list pending minon keys. Then use salt-key -a KEY to accept new minion keys.

Verify by logging in to the master and doing salt '*' test.ping. Should see results from all minons.

Run salt servo-NEWMACHINE state.highstate to install of the dependencies and set up the new minion.