Using nagios to monitor print configuration

Now this isn't a copy a previous post, which was about monitoring print queues. This is about monitoring our quite complex printing configuration system.

Again, a bit of background. We keep as much configuration info as we can in an LDAP directory. And we install printers on our windows desktops ( about 1,200 spread over several sites ) during either the startup scripts (for locally attached printer) or the login script (for network printers). We use the concept of a 'nearest printer', assigned to the workstation dependent on physical location.

So, sticking to network printers, we have:

• A CUPS queue per printer, served by Samba to windows desktops.
• A 'nearest printer' attribute in LDAP, attached to the workstation entry. This 'points to'..
• A printer entry per printer, with an 'installcommand' attribute, which gives an appropriate command line to install that printer on a workstation. This gets run during the login script.

Perhaps it would be clearer to describe how a workstation gets a printer. I'm focusing on network printing, so this happens in the login script.

• Look up my (the workstation's) LDAP entry.
• From my LDAP entry, get the nearest printers.
• For each nearestprinter, get the installcommand from the printer's LDAP entry.
• And run it.

Now we like this. We can have several printers associated with one workstation, and we could (though we don't) associate nearestprinters with user accounts as well as workstations. It's real easy to change a workstation's printer(s), and users need know nothing about it. When it works, it just works.

But there's no referential integrity going on here. We can have orphans anywhere. A CUPS queue with no install commands. An installcommand referring to a non-existent CUPS queue. A workstation with a nearestprinter that doesn't exist. etc. Basically, config rot, caused by human failure to attend to detail.

And here we get to the point of it all. We have a nagios check called 'check_print_config', which checks all this, and creates a warning state if something's out of whack. It's posted below. As with most code posted here, it's finished to the point where it works. It's not great code. It does, I'd posit, do something interesting.

#!/usr/bin/perl -w
use strict;

my @nagiosCupsQueues;
my @nearestPrinterQueues;
my @installCommandQueues;
my @output = ();

#       print "Getting monitored queues from nagios...\n";
       @nagiosCupsQueues = (`grep check_cups_queue /etc/nagios2/conf.d/allPrintQueues.cfg | cut -f2 -d'!'`);
       chop @nagiosCupsQueues;

#       print "Getting installCommand queues from LDAP...\n";
       @installCommandQueues = (`ldapsearch -LLL -x -b "ou=hosts,dc=example,dc=com" '(installcommand=*con2prt*)' installcommand | grep '/cd ' | grep -iv idcard | grep -iv tmu220 | grep -iv null | sort | uniq | cut -f4 -d" " | cut -f4
-d"\\\\"`);
       chop @installCommandQueues;

#       print "Getting nearestprinter queues from LDAP...\n";
       @nearestPrinterQueues = (`ldapsearch -LLL -x -b "ou=hosts,dc=example,dc=com" '(&(objectclass=computer)(nearestprinter=*))' nearestprinter | grep nearestprinter | grep -iv lpt | grep -iv archicad | grep -iv tmu220 | grep -iv idcard | grep -iv null | sort | uniq | cut -f1 -d"," | cut -f2 -d"="`);
       chop @nearestPrinterQueues;

foreach my $icq ( @installCommandQueues ) {
   next if $icq =~ /^$/;
   push(@output, "ICQ: $icq ") if ! grep(/^$icq$/i, @nagiosCupsQueues);
}
foreach my $npq ( @nearestPrinterQueues ) {
   next if $npq =~ /^$/;
   my $npqNotInCups = 0;
   my $npqNoInstallCommand = 0;
   $npqNotInCups = 1 if ! grep(/^$npq$/i, @nagiosCupsQueues);
   $npqNoInstallCommand = 1 if ! grep(/^$npq$/i, @installCommandQueues);
#    push(@output, "NPQ:$npq:") if ( $npqNoInstallCommand && $npqNotInCups );
   my @duffClients = `ldapsearch -LLL -x -b "ou=hosts,dc=example,dc=com" "nearestprinter=cn=$npq,ou=hosts,dc=example,dc=com" dn | grep dn: | cut -f1 -d"," | cut -f2 -d"="`;
   chop @duffClients;
   push(@output, "NPQ:$npq: " . join(",", @duffClients) . " ") if ( $npqNoInstallCommand && $npqNotInCups );
}

#print Dumper([ \@output, ]);
if ( @output > 0 ) {
   print "WARNING: " . join(" ",@output) . "\n";
   exit 1;
} else {
   print "OK\n";
   exit 0;
}

Automating nagios configurations.

At the last count, we run something like 140 print queues, and as offices move, and printers get replaced, and 'stuff changes', queues are created and deleted and renamed. This post is about how I've addressed ensuring that nagios is monitoring all our queues, and minimising the opportunity for operator error.

A little background. We use CUPS to queue print jobs, and our technicians are free to create and delete queues as need be. They do not have access to the nagios configs.

So, the basic idea is that we periodically run a script on the nagios server that:

• Queries each of our print servers for a list of existing queues
• Creates a nagios config file for all print queues in the list
• signals nagios to restart, and re-read it's configuration

So we get a monitoring configuration that doesn't miss print queues out, nor alarms about print queues that no longer exist. And no-one has to remember.

Which is nice.

So, ( and I apologise in advance for the code. I'm a sysadmin. Whaddya expect. ). The following is a perl script called from cron, once for each CUPS server. We pass the server address, and a human-readable site name, and we get nagios code out on stdout, which is piped into the appropriate nagios config directory. It depends on lpstat, which queries the CUPS server.

#!/usr/bin/perl

$cupsServer = $ARGV[0];
$site = $ARGV[1];

@queues = `lpstat -h $cupsServer -p | grep printer | grep -iv "sent" | grep -iv "off-line" | grep -iv "unable" | grep -iv "attempt" | cut -f2 -d" "`;
chop @queues;

foreach $queue ( @queues ) {
   print "define service{\n";
   print "\tuse                        generic-service\n";
   print "\thost_name                  $cupsServer\n";
   print "\tservice_description        CUPS_" . $queue . "\n";
   print "\tservicegroups              " . $site . "PrintQueues\n";
   print "\tcontact_groups             " . $site . "-printer-admins\n";
   print "\tcheck_command              check_cups_queue!" . $queue . "\n";
   print "\tregister                   1\n}\n\n";

   print "define serviceextinfo{\n";
   print "        host_name       " . $cupsServer . "\n";
   print "        service_description     CUPS_" . $queue . "\n";
   print "        notes_url       http://wiki.example.com/wiki/index.php?title=Nagios/" . $queue . "&action=edit&preload=Nagios/NewServiceTemplate\n";
   print "        action_url      http://" . $cupsServer . ".example.com:631/printers/" . $queue . "\n";
   print "        icon_image      HPlj4550p.gif\n}\n\n";
}

Coupla notes - the nagios action_url shows a clickable icon taking the user to the CUPS queue in question. The notes_url points to a wiki page. We use this to keep notes about the service.

This is all very well, but nagios won't pick up the changes without a restart. So once cron has built the config file, it does this:

export now=$( /bin/date "+\%s" ); #get the current time into a format nagios understands
export commandfile='/var/lib/nagios2/rw/nagios.cmd'; #identify the file nagios reads for external commands
/usr/bin/printf "[\%lu] RESTART_PROGRAM\n" $(( now + 30 )) > $commandfile #tell nagios to restart in 30 seconds

And Bob's yer uncle. Monitoring our CUPS queues with nagios means we become aware of problems quicker, and respond quicker. And automating the config makes this practical.

More monitoring with twitter

Or, the little twitterbot that could.

As I've mentioned before, we've got two identical nagios boxes running, one notifies us of problems via email, one via a special private twitter account that the systems team follow. So if email service or one of the nagios boxes goes down, we'll still get notified.

This is an improvement, and we're already getting to problems quicker. Great smashing super. But sometimes we trip over each other. I'll log in to fix something to find that P is already working on it. This hasn't bit us yet, but rest assured, if we don't deal with it, it will bite us one day. So.

The little protocol we're working with now is as follows: when you take on a problem, you IM the others that you're working on it. But that's n-1 messages before you start working on the fix. A pain and a waste of time.

So I'm working on a little bot. It watches the direct messages feed for the monitoring twitter account ( let's call it skaffen ), and when it gets a new direct message, sends it back as an update to the skaffen account, with the original sender prepended. Like this:

skaffen: WARNING -- stuff is borken
mawhin: d skaffen fixing stuff
... up to a minute, because of twitter rate limiting
skaffen: mawhin is fixing stuff

So to pick up a problem you direct message the monitor. I think that's sweet.

Nagios and Twitter

Following on from this post, I've got a second nagios server set up now, monitoring all the same stuff.
It's running in a VM on hardware connected to a different UPS, so that's one weakness mitigated.The other improvement is using twitter as a notification channel, as opposed to mail on the primary monitor. So if our mail service goes down, we'll know about it.
We weren't finding out before, cos the monitor was mailing us, but the mail wasn't getting through.
Reminds me of what my partner pointed out ( and that I'd not considered ) when I outlined VoIP. "But doesn't that mean that when the network is down, no-one will be able to call you to complain?". Every cloud's got a silver lining.