CheckMK

CheckMK is a great monitoring solution that can be used for free if you want to know what is going on with e.g. your Puppet server or your own Mail- and Cloud server.

If you want to use it for free, there are two different options. Either you use CheckMK Raw, which does not offer all features of the Enterprise Edition, or you use CheckMK Free which offers all features but is limited to a certain number of hosts/services. I prefer Open Source Software whenever possible, so I use CheckMK Raw for my needs. For a comparison of the available editions you should check out the official site.

The documentation of CheckMK is superb and I really urge you to use it. Here I just want to describe some specialties which are not included in the standard CheckMK documentation or distribution.

In older versions of CheckMK it was not possible to encrypt the traffic between the CheckMK server and the agents. Nowadays that is possible by registering agents to the server and factually running the agent over SSH is no longer necessary.

However, there might be situations when that is not feasible—e.g. when your network infrastructure allows only one-way traffic. That is why I try to document here how to still run the agent over SSH, obviously that has been removed in the official documentation.

First of all, log in the the CLI of your CheckMK server and switch to the user of the monitoring site you created or want to run over SSH. There, you should create an SSH key like this:

ssh-keygen -t ed25519

While you are still logged in as your monitoring user, adjust the SSH configuration to accept host keys of new hosts only. This provides some level of security for MITM attacks, but also will break monitoring on hosts which change their SSH host key for whatever reason, please keep that in mind. To do so, edit the ~/.ssh/config file and add this line (or if it is already there, adjust it accordingly):

StrictHostKeyChecking accept-new

Have a look at the public SSH key you just created with cat ~/.ssh/id_ed25519.pub and copy it to the clipboard.

Now we are done on the CLI of the monitoring server and we need to configure all the servers which need to be monitored this way. Note that I do this using Puppet and you can use any means you wish (e.g. manually or using another state configuration tool than Puppet or use Ansible, the possibilities are endless.) but this description is about configuring one exemplary server manually.

On the server to be monitored, we first need to make sure that root can not login to the server, but at least run a restricted command over SSH. Also we do not want to allow passwords as a means of authentication. To accomplish this, at least these lines need to be configured in /etc/ssh/sshd_config:

PermitRootLogin forced-commands-only
PasswordAuthentication no

After adjusting the SSH server settings, don't forget to restart it using systemctl restart sshd. Now we take the public SSH key of the monitoring user we created earlier and compose a line like this:

command="/usr/bin/check_mk_agent" ssh-ed25519 <your-key> <your-idenitfier>

That line needs to be added to root's ~/.ssh/authorized_keys file. If that file does not exist (which it shouldn't), please create it like this (as root):

mkdir -m 0700 ~/.ssh
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

After creating the file enter the line we composed above.

Because we do not want the CheckMK agent to respond to network requests at all, you should stop and disable the cmk-agent-ctl-daemon.service:

systemctl stop cmk-agent-ctl-daemon.service
systemctl disable cmk-agent-ctl-daemon.service

At last, we are ready to tell CheckMK to run the Agent over SSH. To accomplish this, go to Setup → Agents → Other Integrations.

Here, create a rule using the Add rule button. Fill in the values as follows.

• Description - A meaningful description, e.g. “CheckMK via SSH”
• Comment - A comment with further explanations, e.g. “CheckMK via SSH and key-based authentication”

• Command line to execute - Fill in ssh -T root@$HOSTADDRESS$ here.

• Condition type - “Explicit Conditions”
• Folder - “Main” or any folder that you want to apply this to.

You could also apply this rule to hosts with certain tags or labels, but in general this should be all. Save and activate the rule and do a service discovery on an arbitrary host to check if it works as designed.

There used to be a great CheckMK plugin to monitor the Puppet Agent and its status on a system. Usually this is automatically done by the Puppet Enterprise Console, but if you are using the OpenSource version of Puppet you are out of luck and have to find a way to monitor that yourself.

On the CheckMK Exchange there are three Plugins available. Two seem to be dead, just one of the plugins seems to be still alive. And the one which is still alive is based on the Bakery—i.e. the commercial CheckMK version.

If I see this correctly, writing a plugin for CheckMK is not overly complicated so I might be inclined to write my own check plugin in future. As soon as I find the time for that I will of course update this website, so stay tuned!