Planet Munin

DebianAccepted 2.0.26-4 in unstable (medium) (Holger Levsen)

DebianAccepted 2.0.26-3 in unstable (medium) (Holger Levsen)

Debianmunin 2.0.26-2 MIGRATED to testing (Britney)

DebianAccepted 2.999.3-1 in experimental (medium) (Holger Levsen)

Debianmunin 2.0.26-1 MIGRATED to testing (Britney)

DebianAccepted 2.0.26-2 in unstable (medium) (Holger Levsen)

DebianAccepted 2.0.26-1 in unstable (medium) (Holger Levsen)

Stig Sandbeck MathisenMunin 3 packaging

The Munin project is moving slowly closer to a Munin 3 release. In parallel, the Debian packaging is changing, too.

The new web interface is looking much better than the traditional web-1.0 interface normally associated with munin.

New package layout

perl libraries

All the Munin perl libraries are placed in “libmunin-*-perl”, and split into separate packages, where the split is decided mostly on dependencies.

If you don’t want to monitor samba, or SNMP, or MySQL, there should be no need to have those libraries installed. That does mean more binary packages, on the other hand.

Munin master

Munin now runs as a standalone HTTPD, it no longer graphs from cron, nor does it run as CGI or FastCGI scripts.

The user “munin” grants read-write access, while the group “munin” grants read only access. The new web interface runs as the “munin-httpd” user, which is member of the “munin” group.

There is a “munin” service. For now, it runs rrdcached for the munin user and RRD directory.

munin node

The perl “munin-node” and the compiled “munin-node-c” should be interchangeable, and be able to run the same plugins.

Munin node, and Munin async node, should be wholly separate from the munin master. It should be possible to use the perl “munin-node” package, and the

munin plugins

The munin plugins are placed separate packages named “munin-plugins-*”. The split is based on monitoring subject, or dependencies. They depend on appropriate “libmunin-plugin-*-perl” packages

The “munin-plugins-c” package, which is is from the “munin-node-c” source, contains a number of compiled plugins which should use less resources than their shell, perl or python equivalents.

Plugins from other sources than “munin” must work similar to the ones from “munin”. More work on this is needed.


Late December 2015, I set up Jenkins, with jenkins-debian-glue to build packages, test with autopkgtest and and update my development apt repository on each commit. That helped developing and testing the new Munin packages.

The packages are not quite ready to upload to experimental, but they are continuously deployed to weed out bugs. They can be found in my packaging apt repo. (The usual non-guarantees apply, handle with care, keep away from small children, etc…)


Munin developers, packagers and users hang out on “#munin” on the OFTC network. Please drop by if you have questions or comments.

Stig Sandbeck MathisenFree software activities in September 2015

What did I do in September 2015?


Working on making the munin master fit inside Mojolicious.

The existing code is not written to make this trivial, but all the pieces are there. Most of the pieces need breaking up into smaller pieces to fit.



New version of puppet-module-puppetlabs-apache which closes:

  • #788124 By default, produces a configuration incompatible with Apache 2.4
  • #788125 Missing directory referenced in SSL configuration
  • #788127 SSLv3 is not disabled

I like it when a new upstream version closes all bugs left in the bts for a package.

A new package, the TLS proxy hitch currently waiting in the queue.

(Update 2016-01-02: It’s not in NEW anymore)


Lots of work on a new ceph puppet module.

Debianmunin 2.0.25-2 MIGRATED to testing (Britney)

DebianAccepted 2.0.25-2 in unstable (medium) (Stig Sandbeck Mathisen)

Debianmunin 2.0.25-1 MIGRATED to testing (Britney)

DebianAccepted 2.0.25-1 in unstable (medium) (Holger Levsen)

Debianmunin 2.0.24-1 MIGRATED to testing (Britney)

DebianAccepted 2.0.24-1 in unstable (medium) (Stig Sandbeck Mathisen)

Debianmunin 2.0.23-1 MIGRATED to testing (Britney)

DebianAccepted 2.0.23-1 in unstable (medium) (Holger Levsen)

DebianAccepted 2.0.22-1 in unstable (low) (Holger Levsen)

DebianAccepted 1.4.5-3+deb6u1 in squeeze-lts (low) (Holger Levsen)

DebianAccepted 2.1.9-1 in experimental (medium) (Stig Sandbeck Mathisen)

Debianmunin 2.0.21-2 MIGRATED to testing (Britney)

DebianAccepted 2.0.21-2 in unstable (medium) (Stig Sandbeck Mathisen)

DebianAccepted 2.0.21-1 in unstable (low) (Stig Sandbeck Mathisen)

DebianAccepted in experimental (medium) (Stig Sandbeck Mathisen)

Stig Sandbeck MathisenMunin helps showing performance changes

Using btrfs on a networked backup server looked like a good idea, what with the data integrity checksumming and all. Problem was, we experienced massive performance issues.

Reformatting it to ext4 gave a decent increase in write performance, and will hopefully give fewer server crashes per week (from “many” to “none” is the goal) Just before this wipe-and-reinstall, “umount” had been hanging for a few hours, and the admin got a tad annoyed.

This is on Ubuntu 12.04.4 LTS (GNU/Linux 3.12.8-031208-generic x86_64). The “disk” in question is a single 28T device on a nearby disk shelf.

Now, why this performance difference? We have Munin installed, and showing graphs from before and after the change gives us a few clues.

Network througput

Network throughput increased to the limit. Looks like it is time to move to active/active bonding, instead of active/passive.


Storage graphs

The number of operations went down, while the request size increased massively. This allowed much more data to be written to the device.

iops latency throughput utilization

DebianAccepted 2.1.5-1 in experimental (medium) (Holger Levsen)

Debianmunin 2.0.19-3 MIGRATED to testing (Britney)

DebianAccepted 2.0.19-3 in unstable (medium) (Holger Levsen)

DebianAccepted 2.1.4-1 in experimental (medium) (Holger Levsen)

Steve SchneppTutorial - Setup a dev environment for Munin

I discovered some time ago the marvelous dev_scripts/ directory in the munin source code. So, as its usage is very easy, I'll just write a tutorial about how to use it


To use it, one has to install all the packages needed for munin, and to grab a copy of the source code. Easiest is to use either a tarball, or to clone the git repository.

Note that the guidelines on contributing back are specified directly in the git repo.

Now, I just assume you want to contribute back, otherwise you would not care much about the said dev environment. That means using the git way of doing it.

Download the source code

First step is to clone the git repository. We will use $HOME/src/munin as the development directory.

mkdir -p $HOME/src
cd $HOME/src
git clone munin
cd munin

Compile munin

Now, we have to compile the source code. I know that it sounds strange as the code is mostly Perl, but there are some templates that need to be filled with the environment specifics, such as the Perl interpreter path, a POSIX compatible shell, ...

dev_scripts/install 1

Now all munin (and munin-node) should be compiled and installed in $HOME/src/munin/sandbox.

Note that the 1 at the end is explained below.

Using the dev tools

There are some different tools in dev_scripts/ :


This is the one you used already. You have to use it every time you want to recompile & deploy the package.

The 1 argument, does a full re-install (wipe & install), so you don't usually want to do that.


This is a tool to start the development node. Note that it listens on the port 4948, so you can use it alongside a normal munin-node.


The run command inside is used to launch all the executable parts of munin, such as munin-update, munin-html or munin-limits. It can also be used to launch munin-run and munin-node-configure.

The usage is very simple, just prefix the command to launch with dev_scripts/run, every environment variable and command line argument will be forwarded to the said command.

# launch munin-cron

# launch manually some cron parts

# debug a plugin
dev_scripts/munin-run --debug cpu config


This is the same as run, only for CGI. It sets up the whole environment vars that emulates a CGI call. Usage is very easy :

dev_scripts/cgi munin-cgi-graph /localnet/localhost/cpu-day.png > out.dat

The out.dat will contain the whole HTTP output, with the HTTP headers and the PNG content. Everything that is sent to STDERR won't be catched, so you can liberally use it while debugging.


The query_munin_node is used to send commands to the node in a very simple way. Node commands are just args of the tool.

dev_scripts/query_munin_node list
dev_scripts/query_munin_node config cpu
dev_scripts/query_munin_node fetch cpu

Real CGI usage with your web browser

That's the holy grail. You will have a development version that behaves the same as a real munin install.

First, let's assume you have a working user cgi configuration (ie ~user/cgi/whatever is working). If not you should refer yourself to the local documentation of your preferred webserver. Note that nginx will _not_ work, as it does not support CGI.

I wrote a very simple cgi wrapper script. The home dir is hard coded in the script.

#! /bin/sh

eval "$(perl -V:version)"

#export DBI_TRACE=2=/tmp/dbitrace.log

exec perl -T -I $PERL5LIB $ROOT/sandbox/opt/munin/www/cgi/$CGI_NAME

Debianmunin 2.0.19-2 MIGRATED to testing (Britney)

DebianAccepted 2.0.6-4+deb7u2 in wheezy-security (high) (Stig Sandbeck Mathisen)

DebianAccepted 2.0.19-2 in unstable (low) (Holger Levsen)

DebianAccepted 2.0.19-1 in unstable (low) (Holger Levsen)

Debianmunin 2.0.18-1 MIGRATED to testing (Britney)

Steve SchneppExperimenting with a C munin node

Core plugins are designed for simplicity...

As I wrote about it earlier, Helmut rewrote some core plugins in C. It was maintly done with efficiency in mind.

As those plugins are only parsing one /proc file, there seemed no need to endure the many forks inherent with even trivial shell programming. It also acknowledges the fact that the measuring system shall be as light as possible

Munin plugin are highly driven towards simplicity. Therefore having shell plugins is quite logical. It conveys the educational sample purpose for users to write their own, while being quite easy to code/debug for the developpers. Since their impact on current systems is very small, there are not much incentive to change.

... but efficiency is coming !

Nonetheless, now monitored systems are becoming quite small.

It is mostly thanks to embedded systems like the RaspberryPi. This means that processing power available is much lower than on normal nodes[1].

Now the embedded C approach for plugins has a new rationale.


[1] Usually datacenter nodes are more in the high end of the spectrum than the low-end.

DebianAccepted 2.0.18-1 in unstable (high) (Stig Sandbeck Mathisen)

Debianmunin 2.0.17-3 MIGRATED to testing (Britney)

DebianAccepted 2.0.17-3 in unstable (low) (Stig Sandbeck Mathisen)

Debianmunin 2.0.17-2 MIGRATED to testing (Britney)

DebianAccepted 2.0.17-2 in unstable (low) (Holger Levsen)

Debianmunin 2.0.17-1 MIGRATED to testing (Britney)

DebianAccepted 2.0.17-1 in unstable (low) (Holger Levsen)

Debianmunin 2.0.16-3 MIGRATED to testing (Britney)

DebianAccepted 2.0.16-3 in unstable (low) (Stig Sandbeck Mathisen)

Debianmunin 2.0.16-2 MIGRATED to testing (Britney)

DebianAccepted 2.0.6-4+deb7u1 in wheezy (low) (Holger Levsen)

DebianAccepted 2.0.16-2 in unstable (low) (Holger Levsen)

DebianAccepted 2.1.2-1 in experimental (low) (Stig Sandbeck Mathisen)

DebianAccepted 2.0.16-1 in unstable (low) (Stig Sandbeck Mathisen)

DebianAccepted 2.0.15-1 in unstable (low) (Stig Sandbeck Mathisen)

DebianAccepted 2.1.1-1 in experimental (low) (Holger Levsen)

Debianmunin 2.0.14-1 MIGRATED to testing (Britney)

Steve SchneppSpinoffs in the munin ecosystem

KISS is the core design of Munin

Munin's greatest strength is its very KISS architecture. It therefore gets many things right, such as a huge modularity.

Each component (master/node/plugin) has a simple API to communicate with the others.

Spin-offs ...

I admit that the master, even the node, have convoluted code. In fact some rewrites already do exist.

... are welcomed ...

And they are a really good thing, as it enables rapid prototyping on things that the stock munin has (currently) trouble to do.

The stock munin is a piece of software that many depend upon, so it has to move at a much slower pace than one does want, even me. As much as I really want to add many many features to it, I still have to take extra care that it doesn't break stuff, even the least known features.

So I take munin off-springs very seriously and even offer as much help as I can in order for them to succeed.

... because they are very valuable in the long term

In my opinion competition is only short bad in the short term, and in the long term they usually add significant value to the whole ecosystem. That said, there's always a risk to become slowly irrelevant, but I think that's the real power of open-source's evolutionary paradigm : embrace them or become obsolete and get replaced.

Since, if someone takes the time to author a competitor that has a real threat potential, it mostly means that there's a real itch to scratch and that many things are to be learnt.

Different layers of spin-offs

The munin ecosystem is divided in 3 main categories, obviously related to the 3 main components of munin : master, node & plugin.


That's the most obvious part as custom plugins are the real bread and butter of munin.

Stock plugins are mostly written in Perl or POSIX shell, as Perl is munin's own language and POSIX shell is ubiquitous. That fact is acknowledged by the fact that core munin provides 2 libraries (Perl & Shell) to help plugin authoring.

So, it's quite natural that each mainstream language has grown its own plugin library. Some language even have two of them.


Some plugins got even rewritten in plain C, as it was shown that shell plugins do have a significant impact on very under-powered nodes, such as embedded routers.


This component is very simple. Yet, it has to be run on all the nodes that one wants to monitor. It is currently written in Perl, and while that's not an issue on UNIX-like systems, it can be quite problematic on embedded ones

Simple munin

The official package comes with a POSIX shell rewrite that has to be run from inetd. It is quite useful for embedded routers like OpenWRT, but still suffers from an hard dep on POSIX shell and inetd.


SNMP is another way to monitor nodes. While it works really well, it mostly suffers the fact that its configuration is quite different of the usual way, so I guess some things will change on that side.

Win32 ports

Win32 has long been a very difficult OS to monitor, as it doesn't offer much of the UNIX-esque features. Yet the number of win32 nodes that one wants to monitor is quite high, as it makes munin one the few systems that can easily monitor heterogeneous systems.

Therefore, while you can install the stock munin-node, several projects emerged. We decided to adopt munin-node-win32.


There's also a dedicated node for Android. It makes sense, given that the Android is yet Linux-derived, but lacks Perl, and is a Java mostly platform. This node also has some basic capabilities of pushing data to the master instead of the usual polling.

This is specially interesting given the fact that Android nodes are usually loosely connected, so the node spools values itself and pushes them when it recovers connectivity.

Note that this is specifically an aspect that is currently lacking in munin, and I'm planning to address it in the 2.1 series. So thanks to its author for showing a relevant use-case.


That's my last experiment. It started with a simple question : how difficult would it be to code a fairly portable version of the node ?

It turned out that it wasn't that difficult. I'm even asking myself about eventually replacing the win32 specific port with this one, as the code is much simpler. The win32 node has several plugin built-in mostly due to platform specifics. I still have to find a way to work my way around it, but it's in quite good shape.

This post was originally done to promote it, but while writing it I noticed that the ecosystem deserved a post on its own. So I'll write another one, specific to the C port of munin-node and plugins.


The master is the most complex component. So rewrites of it won't happen as-is. They usually take the form of a bridge between the munin protocol and another graphing system, such as Graphite.


There are also client libraries that are able to directly query munin nodes, to be able to reuse the vast ecosystem. Languages are various, from the obvious Python to Ruby, along with a quite modern node.js one.

Steve SchneppWhen having good relationships with package maintainers can also be a curse

I advise every user to only use the packaged version of munin. Here's a short article to explain the background of my reluctance to ask for users to directly use the official tarball.

I have become upstream of munin a while ago now. As such, I'm in contact with package maintainers. They take the official releases and cram it into their own distribution of choice[1].

I have to admit that the various epic war stories read throughout the web about upstream vs packagers are very far from the truth here. They are a charm to work with. Often challenging and demanding, but always because there's a real need. And that's quite a good thing, as I'm still a rookie in term of open source software management. Therefore I'm quite grateful when they gently pinpoint my mistakes[2].

Yet, this nice team comes with a price. Since we mostly hang out on IRC together, there is way much inter-distro communication than on other software. But I'm the sole owner of the tarball distro .

Yet, as I don't like to build everything from source, I obviously use a distro. There, since the packaging is very nicely done, I don't feel to take the hassle of using my own "tarball" to test them. I just build a package for my distro out of the release code.

That's also a curse, as I admit that I although I test the code, I only seldom test the packaging. This means that I cannot really advise someone on using the tarball, nor directly git code as even I don't do it.

But, that said, I still think I'm the luckiest upstream around. Thanks guys !


[1] Be it linux-based like Gentoo, Redhat..., BSD-based as FreeBSD, OpenBSD..., or even multi-kernel based as Debian

[2] Defaulting to CGI graphics was a move that was way too premature, end-user wise. So thanks to them, it defaults to cron again

Diego Elio PettenòBook review: Instant Munin Plugin Starter

This is going to be a bit of a different review than usual, if anything because I actually I already reviewed the book, in the work-in-progress sense. So bear with me.

Today, Packt published Bart ten Brinkle’s Instant Munin Plugin Starter which I reviewed early this year. Bart has done an outstanding job in expanding from the sparsely-available documentation to a comprehensive and, especially coherent book.

If you happen to use Munin, or are interested to use it, I would say it’s a read well worth the \$8 that it’s priced at!

Steve SchneppAvoid those milli-hits in Munin

A recurring question on IRC is : why do I have 500 million hit/s in my graph ?.

Turns out that they are really seeing 500m hit/s, and that lower-case m means milli, and not Mega as specified in the Metric system. This is automatically done by RRD.

To avoid this you should just specify graph_scale no as specified.

Diego Elio PettenòThe unsolved problem of the init scripts

One of probably the biggest problems with maintaining software in Gentoo where a daemon is involved, is dealing with init scripts. And it’s not really that much of a problem with just Gentoo, as almost every distribution or operating system has its own to handle init scripts. I guess this is one of the nice ideas behind systemd: having a single standard for daemons to start, stop and reload is definitely a positive goal.

Even if I’m not sure myself whether I want the whole init system to be collapsed into a single one for every single operating system out there, there at least is a chance that upstream developers will provide a standard command-line for daemons so that init scripts no longer have to write a hundred lines of pre-start setup code commands. Unfortunately I don’t have much faith that this is going to change any time soon.

Anyway, leaving the daemons themselves alone, as that’s a topic for a post of its own and I don care about writing it now. What remains is the init script itself. Now, while it seems quite a few people didn’t know about this before, OpenRC has been supporting since almost ever a more declarative approach to init scripts by setting just a few variables, such as command, pidfile and similar, so that the script works, as long as the daemon follows the most generic approach. A whole documentation for this kind of scripts is present in the runscript man page and I won’t bore you with the details of it here.

Beside the declaration of what to start, there are a few more issues that are now mostly handled to different degrees depending on the init script, rather than in a more comprehensive and seamless fashion. Unfortunately, I’m afraid that this is likely going to stay the same way for a long time, as I’m sure that some of my fellow developers won’t care to implement the trickiest parts that can implemented, but at least i can try to give a few ideas of what I found out while spending time on said init scripts.

So the number one issue is of course the need to create the directories the daemon will use beforehand, if they are to be stored on temporary filesystems. What happens is that one of the first changes that came with the whole systemd movements was to create /run and use that to store pidfiles, locks and other runtime stateless files, mounting it as tmpfs at runtime. This was something I was very interested in to begin with because I was doing something similar before, on the router with a CF card (through an EIDE adapter) as harddisk, to avoid writing to it at runtime. Unfortunately, more than an year later, we still have lots of ebuilds out there that expects /var/run paths to be maintained from the merge to the start of the daemon. At least now there’s enough consensus about it that I can easily open bugs for them instead of just ignore them.

For daemons that need /var/run it’s relatively easy to deal with the missing path; while a few scripts do use mkdir, chown and chmod to handle the creation of the missing directories , there is a real neat helper to take care of it, checkpath — which is also documented in the aforementioned man page for runscript. But there has been many other places where the two directories are used, which are not initiated by an init script at all. One of these happens to be my dear Munin’s cron script used by the Master — what to do then?

This has actually been among the biggest issues regarding the transition. It was the original reason why screen was changed to save its sockets in the users’ home instead of the previous /var/run/screen path — with relatively bad results all over, including me deciding to just move to tmux. In Munin, I decided to solve the issue by installing a script in /etc/local.d so that on start the /var/run/munin directory would be created … but this is far from a decent standard way to handle things. Luckily, there actually is a way to solve this that has been standardised, to some extents — it’s called tmpfiles.d and was also introduced by systemd. While OpenRC implements the same basics, because of the differences in the two init systems, not all of the features are implemented, in particular the automatic cleanup of the files on a running system —- on the other hand, that feature is not fundamental for the needs of either Munin or screen.

There is an issue with the way these files should be installed, though. For most packages, the correct path to install to would be /usr/lib/tmpfiles.d, but the problem with this is that on a multilib system you’d end up easily with having both /usr/lib and /usr/lib64 as directories, causing Portage’s symlink protection to kick in. I’d like to have a good solution to this, but honestly, right now I don’t.

So we have the tools at our disposal, what remains to be done then? Well, there’s still one issue: which path should we use? Should we keep /var/run to be compatible, or should we just decide that /run is a good idea and run with it? My guts say the latter at this point, but it means that we have to migrate quite a few things over time. I actually started now on porting my packages to use /run directly, starting from pcsc-lite (since I had to bump it to 1.8.8 yesterday anyway) — Munin will come with support for tmpfiles.d in 2.0.11 (unfortunately, it’s unlikely I’ll be able to add support for it upstream in that release, but in Gentoo it’ll be). Some more of my daemons will be updated as I bump them, as I already spent quite a lot of time on those init scripts to hone them down on some more issues that I’ll delineate in a moment.

For some, but not all!, of the daemons it’s actually possible to decide the pidfile location on the command line — for those, the solution to handle the move to the new path is dead easy, as you just make sure to pass something equivalent to -p ${pidfile} in the script, and then change the pidfile variable, and done. Unfortunately that’s not always an option, as the pidfile can be either hardcoded into the compiled program, or read from a configuration file (the latter is the case for Munin). In the first case, no big deal: you change the configuration of the package, or worse case you patch the software, and make it use the new path, update the init script and you’re done… in the latter case though, we have trouble at hand.

If the location of the pidfile is to be found in a configuration file, even if you change the configuration file that gets installed, you can’t count on the user actually updating the configuration file, which means your init script might get out of sync with the configuration file easily. Of course there’s a way to work around this, and that is to actually get the pidfile path from the configuration file itself, which is what I do in the munin-node script. To do so, you need to see what the syntax of the configuration file is. In the case of Munin, the file is just a set of key-value pairs separated by whitespace, which means a simple awk call can give you the data you need. In some other cases, the configuration file syntax is so messed up, that getting the data out of it is impossible without writing a full-blown parser (which is not worth it). In that case you have to rely on the user to actually tell you where the pidfile is stored, and that’s quite unreliable, but okay.

There is of course one thing now that needs to be said: what happens when the pidfile changes in the configuration between one start and the stop? If you’re reading the pidfile out of a configuration file it is possible that the user, or the ebuild, changed it in between causing quite big headaches trying to restart the service. Unfortunately my users experienced this when I changed Munin’s default from /var/run/munin/ to /var/run/ — the change was possible because the node itself runs as root, and then drops privileges when running the plugins, so there is no reason to wait for the subdirectory, and since most nodes will not have the master running, /var/run/munin wouldn’t be useful there at all. As I said, though, it would cause the started node to use a pidfile path, and the init script another, failing to stop the service before starting it new.

Luckily, William corrected it, although it’s still not out — the next OpenRC release will save some of the variables used at start time, allowing for this kind of problems to be nipped in the bud without having to add tons of workarounds in the init scripts. It will require some changes in the functions for graceful reloading, but that’s in retrospective a minor detail.

There are a few more niceties that you could do with init scripts in Gentoo to make them more fool proof and more reliable, but I suppose this would cover the main points that we’re hitting nowadays. I suppose for me it’s just going to be time to list and review all the init scripts I maintain, which are quite a few.

Diego Elio PettenòMunin and IPv6

Okay here it comes another post about Munin for those who are using this awesome monitoring solution (okay I think I’ve been involved in upstream development more than I expected when Jeremy pointed me at it). While the main topic of this post is going to be IPv6 support, I’d like first to spend a few words for context of what’s going on.

Munin in Gentoo has been slightly patched in the 2.0 series — most of the patches were sent upstream the moment when they were introduced, and most of them have been merged in for the following release. Some of them though, including the one bringing my FreeIPMI plugin to replace the OpenIPMI plugins, or at least the first version of it, and those dealing with changes that wouldn’t have been kosher for other distributions (namely, Debian) at this point, were also not merged in the 2.0 branch upstream.

But now Steve opened a new branch for 2.0, which means that the development branch (Munin does not use the master branch, for a simple logistic reason of having a master/ directory in GIT I suppose) is directed toward the 2.1 series instead. This meant not only that I can finally push some of my recent plugin rewrites but also that I could make some more deep changes to it, including rewriting the seven asterisk plugins into a single one, and work hard on the HTTP-based plugins (for web servers and web services) so that they use a shared backend, like SNMP. This actually completely solved an issue that, in Gentoo, we solved only partially before — my ModSecurity ruleset blacklists the default libwww-perl user agent, so with the partial and complete fix, Munin advertises itself in the request; with the new code it includes also the plugin that is currently making the request so that it’s possible to know which requests belongs to what).

Speaking of Asterisk, by the way, I have to thank Sysadminman for lending me a test server for working on said plugins — this not only got us the current new Asterisk plugin (7-in-1!) but also let me modify just a tad said seven plugins, so that instead of using Net::Telnet, I could just use IO::Socket::INET. This has been merged for 2.0, which in turn means that the next ebuild will have one less dependency, and one less USE flag — the asterisk flag for said ebuild only added the Net::Telnet dependency.

To the main topic — how did I get to IPv6 in Munin? Well, I was looking at which other plugins need to be converted to “modernity” – which to me means re-using as much code possible, collapse multiple plugins in one through multigraph, and support virtual-nodes – and I found the squid plugins. This was interesting to me because I actually have one squid instance running, on the tinderbox host to avoid direct connection to the network from the tinderboxes themselves. These plugins do not use libwww-perl like the other HTTP plugins, I suppose (but I can’t be sure, for what I’m going to explain in a moment) because the cache://objects request that has to be done might or might not work with the noted library. Since as I said I have a squid instance, and these (multiple) plugins look exactly like the kind of target that I was looking for to rewrite, I started looking into them.

But once I started, I had a nasty surprise: my Squid instance only replies over IPv6, and that’s intended (the tinderboxes are only assigned IPv6 addresses, which makes it easier for me to access them, and have no NAT to the outside as I want to make sure that all network access is filtered through said proxy). Unfortunately, by default, libwww-perl does not support accessing IPv6. And indeed, neither do most of the other plugins, including the Asterisk I just rewrote, since they use IO::Socket::INET (instead of IO::Socket::INET6). A quick searching around, and this article turned up — although then this also turned up that relates to IPv6 support in Perl core itself.

Unfortunately, even with the core itself supporting IPv6, libwww-perl seems to be of different ideas, and that is a showstopper for me I’m afraid. At least, I need to find a way to get libwww-perl to play nicely if I want to use it over IPv6 (yes I’m going to work this around for the moment and just write the new squid plugins against the IPv4). On the other hand, using IO::Socket::IP would probably solve the issue for the remaining parts of the node and that will for sure at least give us some better support. Even better, it might be possible to abstract and have a Munin::Plugin::Socket that will fall-back to whatever we need. As it is, right now it’s a big question mark of what we can do there.

So what can be said about the current status of IPv6 support in Munin? Well, the Node uses Net::Server, and that in turn is not using IO::Socket::IP, but rather IO::Socket::INET or INET6 if installed — that basically means that the node itself will support IPv6 as long as INET6 is installed, and would call for using it as well, instead of using IO::Socket::IP — but the latter is the future and, for most people, will be part of the system anyway… The async support, in 2.0, will always use IPv4 to connect to the local node. This is not much of a problem, as Steve is working on merging the node and the async daemon in a single entity, which makes the most sense. Basically it means that in 2.1, all nodes will be spooled, instead of what we have right now.

The master, of course, also uses IPv6 — via IO::Socket::INET6 – yet another nail in the coffin of IO::Socket::IP? Maybe. – this covers all the communication between the two main components of Munin, and could be enough to declare it fully IPv6 compatible — and that’s what 2.0 is saying. But alas, this is not the case yet. On an interesting note, the fact that right now Munin supports arbitrary commands as transports, as long as they provide an I/O interface to the socket, make the fact that it supports IPv6 quite moot. Not only you just need an IPv6-capable SSH to handle it, but you can probably use SCTP instead of TCP simply by using a hacked up netcat! I’m not sure if monitoring would get any improvement of using SCTP, although I guess it might overcome some of the overhead related to establishing the connection, but.. well it’s a different story.

Of course, Munin’s own framework is only half of what has to support IPv6 for it to be properly supported; the heart of Munin is the plugins, which means that if they don’t support IPv6, we’re dead in the water. Perl plugins, as noted above, have quite a few issues with finding the right combination of modules for supporting IPv6. Bash plugins, and indeed any other language that could be used, would support IPv6 as good as the underlying tools — indeed, even though libwww-perl does not work with IPv6, plugins written with wget would work out of the box, on an IPv6-capable wget… but of course, the gains we have by using Perl are major enough that you don’t want to go that route.

All in all, I think what’s going to happen is that as soon as I’m done with the weekend’s work (which is quite a bit since the Friday was filled with a couple of server failures, and me finding out that one of my backups was not working as intended) I’ll prepare a branch and see how much of IO::Socket::IP we can leverage, and whether wrapping around that would help us with the new plugins. So we’ll see where this is going to lead us, maybe 2.1 will really be 100% IPv6 compatible…

Diego Elio PettenòWhy my Munin plugins are now written in Perl

This post is an interlude between Gentoo-related posts. The reason is that I have one in drafts that requires me to produce some results that I have not yet, so it’ll have to wait for the weekend or so.

You might remember that my original IPMI plugin was written in POSIX sh and awk, rather than bash and gawk as the original one. Since then, the new plugin (that as it turns out might become part of the 2.1 series but not to replace both the old ones, since RHEL and Fedora don’t package a new enough version of Freeipmi) has been rewritten in Perl, so using neither sh nor awk. Similarly, I’ve written a new plugin for sensors which I also wrote in Perl (although in this case the original one also used it).

So why did I learn a new language (since I never programmed in Perl before six months ago) just to get these plugins running? Well, as I said in the other post, the problem was calling the same command so many times, which is why I wanted to go multigraph — but when dealing with variables, sticking to POSIX sh is a huge headache. One of the common ways to handle this is to save to a temporary directory the output of a command and parse that multiple times, but that’s quite a pain, as it might require I/O to disk, and it also means that you have to execute more and more commands. Doing the processing in Perl means that you can save things in variables, or even just parse it once and split it into multiple objects, to be later used for output, which is what I’ve been doing for parsing FreeIPMI’s output.

But why Perl? Well, Munin itself is written in Perl, so while my usual language of choice is Ruby, the plugins are much more usable if doing it in Perl. Yes, there are some alternative nodes written in C and shell, but in general it’s a safe bet that these plugins will be executed on a system that at least supports Perl — the only system I can think of that wouldn’t be able to do so would be OpenWRT, but that’s a whole different story.

There are a number of plugins written in Python and Ruby, some in the official package, but most in the contrib repository and they could use some rewriting. Especially those that use net-snmp or other SNMP libraries, instead of Munin’s Net::SNMP wrapper.

But while the language is of slight concern, some of the plugins could use some rewriting simply to improve their behaviour. As I’ve said, using multigraphs it’s possible to reduce the number of times that the plugin is executed, and thus the number of calls to the backend, whatever that is (a program, or access to /sys), so in many cases plugins that support multiple “modes” or targets through wildcarding can be improved by making them a single plugin. In some cases, it’s even possible to reduce multiple plugins into one, as I did to the various apache_* plugins shipping with Munin itself, replaced on my system with apache_status as provided by the contrib repository, that fetches the server status page only once and then parses it to produce the three graphs that were, before that, created by three different plugins with three different fetches.

Another important trick up our sleeves while working on Munin plugins is dirty config which basically means that (under indication from the node itself), you can make the plugin output the values as well as the configuration itself during the config execution — this saves you one full trip to the node (to fetch the data), and usually that also means it saves you from having to send one more call to the backend. In particular with these changes my IPMI plugin went from requiring six calls to ipmi-sensors per update, for the three graphs, to just one. And since it’s either IPMI on the local bus (which might require some time to access) or over LAN (which takes more time), the difference is definitely visible both in timing, and in traffic — in particular one of the servers at my day job is monitoring another seven servers (which can’t be monitored through the plugin locally), which means that we went from 42 to 7 calls per update cycle.

So if you use Munin, and either have had timeout issues in the past or recently, or you have some time at hand to improve some plugins, you might want to follow what I’ve been doing, and start improving or re-writing plugins to support multigraph or dirtyconfig, and thus improve its performance.