I've now switched my stats collection for the third time in the past year, and I'm hoping this will be the last time.
I'm not going to cover configuration or installation here, as their wiki is excellent, I will however post a couple of things which got me, and a couple of changes I made.
Five minute demo video (well worth watching)
I really liked collectd, nice plugins collection - there was a plugin for everything I wanted/cared about.
Written in C, push-based, fast, and regular polling intervals for HD stats (and who doesn't want HD these days :)
The catch is the frontend, they all seem to be severely lacking.
Various areas where CGP lacks was pointed out at my new job, mainly the inability to compare multiple graphs - at a specific point in time, combine values into one graph, and be able to search/combine graphs easily (regex or similar). After much searching for an alternative frontend, I finally came across Jart, which is the frontend for Yaketystats.
Yaketystats 2.1 has been around for over two years (Yaketystats itself has been around for approximately five altogether internally at USG), but it doesn't seem to have gotten the same community attention as munin or collectd, which is a shame.
Yaketystats is all bash/perl/php, and it's all quite simply put together.
That's not to say it's simple -- but it's very easy to modify and add onto.
They have put thought into the backend system, stats are collected on each host, stored (plain text), and a cron runs which posts the stats to a php page on the server. If it cannot contact the server it just keeps queueing up the stats till it can.
Once the stats are submitted, a couple of jobs are run on them, eventually outputting them as RRDs.
I noticed less load from this system than collectd/rrdcached for the same number of hosts.
This is where the real magic starts, Jart is their frontend.
It can draw graphs in three different ways:
1) You draw 'all' the graphs for cpu for a host - this draws a graph containing idle, iowait, user, sys for a system (combined for all CPUs).
2) You select which stats you'd like, specifically, from any number of hosts, which it will then put in one graph
3) Regex! You can build graphs using Perl's regex -- very, very cool.
You can modify any graph you create, on the fly -- change from using lines, to stacks to area, negative or positive for different values (e.g. received traffic is given in negative, transmitted given in positive values on the graph), change colour etc - you have full control.
You can then save whichever graphs you have open as a playlist - meaning you can re-open these graphs easily, with all your settings.
So far been using this two or three days, and converted a couple other people to it too.
Couple of changes I made for my setups:
patch to add limits tab to graphs.
Remove the info at end of each graph (generation time, most recent rrds etc), modify index.php:
$tmp = array_merge($this->args,$this->defs,$this->lines,$this->events,$this->comments);
$tmp = array_merge($this->args,$this->defs,$this->lines,$this->events);
Also, we wanted to sort/break up our server names by the fourth character, and not the first, again requires modifying index.php:
var nl = me.charAt(0);
var nl = me.charAt(3);
Note, found a bug where it breaks completely if you have a username with less characters than it looks for above (4 in this case).
The last change is to remove the capitalisation of the hostnames - comment out the two lines above the charAt line:
mefirst = me.replace(/(.).*/,'$1').toUpperCase();
me = me.replace(/.(.*)/,mefirst + '$1');