Blog Work Stories Contact

Installing Salt on Debian servers

As I start to manage more and more servers, both at work and at home, I wished there was a way to send all of the machines commands at once.  I started thinking of a way to queue up commands to be cast out to all of the servers at once, but then I realized someone else must have thought of this before.

After talking to some friends with more experience than me, I was pointed towards SaltStack, or just “Salt.”  Salt is a free, open-source remote execution engine.  I have been using it for a month now, and it has made my life so much easier.    There are other tools out there that do similar things, but Salt has been working for me and seems to have a large user base, so I have stuck with it.

Why is Salt awesome?

  • It saves you time.  You don’t need to SSH in to all of your servers one at a time and then run whatever commands you want to run.
  • It ensures consistency.  Commands you issue will be run on all servers without fail.  No servers will be left behind, so your environment will be consistent.
  • It’s easy to understand.  The command syntax is straightforward, easy to learn, and well documented.
  • It has many built-in useful utilities.  For instance, to add a hostfile entry, you don’t have to bother with silly ‘sed’ commands.  There are built-in commands to add, remove, and modify entries in config files like hostfiles, SSH allowed keys, and many more.  Some of the utilities can even check if entries exist and will return a true/false answer, which can be then used in conditional logic to perform additional commands.

Is there a catch?

The things that make Salt so useful can also make it very dangerous.

  • Be careful.  The commands you enter will be run at once on all machines.  There’s no way to recall commands once they are issued, so ensure your commands are tested before issuing them.
  • Think of your servers as a single collective entity.  Since the commands you send out will take place across all machines, make sure that the commands and changes make sense for your entire server collection.  For instance, don’t run a command to install Apache when some of them are database-only machines that shouldn’t have Apache installed on them.  Targeting scope can help isolate certain machines in your server collection.
  • Protect the master.  Preferably, the master should be on a separate, hardened machine.  If an attacker can gain access to your Salt master, he can execute commands on all of your machines.  The potential impact here is particularly devastating.

As long as you keep those cautions in mind, you should be OK.

System architecture

Salt follows a very standard master/slave relationship, with multiple slaves taking orders from a single master.  Although in the Salt world they use the terms master and minion.

Here’s what a typical Salt environment will look like:

                +--------+
                | Master |
                +--------+
                     |
      ------------------------------
      |              |             |
+----------+   +----------+   +----------+
| Minion A |   | Minion B |   | Minion C |
+----------+   +----------+   +----------+

Installing Salt

Step 1: Add the Salt repository

The commands in this step will add the repository and install Salt’s GPG key to allow packages to be installed on your machine.

echo "" >> /etc/apt/sources.list
echo "# SaltStack" >> /etc/apt/sources.list
echo "deb http://repo.saltstack.com/apt/debian/8/amd64/latest jessie main" >> /etc/apt/sources.list
wget -O - https://repo.saltstack.com/apt/debian/8/amd64/latest/SALTSTACK-GPG-KEY.pub | apt-key add -
apt-get update

Step 2: Install the master

On the machine you want to the the master, preferably a very secure, locked down machine built just for this sole purpose, do the following:

Install the Salt master package

sudo apt-get install salt-master

Generate the master’s public key.  Save this result later, you’ll be adding it to each minion machine.

sudo salt-key -F master
master.pub: xx:xx...xx:xx

Note: Ensure that iptables is configured to allow incoming traffic from the Salt minions on ports 4505 and 4506.

Step 3: Install the minions

Repeat these steps for each computer you want to be a minion. You can also install the minion software on the master, so that the commands you apply to the server collection are reflexively applied to the master as well.

Install the salt minion package and then immediately stop the service.

sudo apt-get install salt-minion && sudo service salt-minion stop

Add a host file entry called “salt” pointing to the IP address of the master.

Edit the minion’s configuration file (/etc/salt/minion) and make the following changes:

  1. Uncomment master_finger and add the master’s key (generated in the above steps).
  2. Uncomment id and set a manual minion ID (otherwise the machine’s hostname will be used).

Start the salt minion service.

sudo service salt-minion start

At this point the minion will connect to the master and send a request to join the server collection.  Minions must be accepted by the administrator, which will be covered in the next step.

Step 4: Validate and add the minions to the master

Newly created minions will connect to the master, but they will not become part of the server collection until you approve their application to join.

On the master:

List all minions waiting to be approved:

sudo salt-key -L

Approve all minions in the list:

sudo salt-key -A

Approve a single minion with ID XXX:

sudo salt-key -a xxx

Reject a particular minion with ID xxx in the list:

sudo salt-key -r xxx

Once you have approved your minions, you’re ready to send out your first command to them!

Basic commands

A good first command is to ping the minions and see if they are alive and responding.

sudo salt '*' test.ping

You should see a response from each server of “True” indicating the minions responded to the test ping.

Here are some other good basic commands you can try:

sudo salt '*' status.loadavg
sudo salt '*' status.uptime
sudo salt '*' status.version

Package management commands

Let’s move on to some more advanced stuff.  How about doing this on all of your machines?

sudo apt-get update
sudo apt-get upgrade
sudo apt-get autoremove

Salt can issue commands directly to the servers, so we could technically have it run each of those three commands as we normally would.  But what if you’re running a server collection with some Debian machines (using apt) and some Redhat machines (using yum)?

Salt has built-in utilities for many similar scenarios that are a huge help.  These utilities determine which OS is running on the machine and translate your requests into commands specifically for that OS.

So, to run those commands via Salt, you would do:

sudo salt '*' pkg.refresh_db   // does "apt-get update"
sudo salt '*' pkg.upgrade      // does "apt-get upgrade"
sudo salt '*' pkg.autoremove   // does "apt-get autoremove"

You’re now up and running with Salt.  There’s so much more you can do with the tool, this is just barely scratching the surface.  I hope you found this guide useful!