Sunday, November 21, 2010

Hetzner, upgrading lenny to squeeze

Late last week ended buying a hetzner VM (vServer vq7) mainly out of curiosity.
I'd heard a fair bit of good things about hetzner, and wanted to try them.
Fairly impressed, 30ms - 50ms latency (stable, varies from site-to-site), and they seem to be sanely installed VMs (no custom kernels etc).

Went for Debian 64-bit VM which turned out to be 5.0 (Lenny), again, out of curiosity did the upgrade process to 6.0 (Squeeze - unstable):
Modify sources.list to squeeze
apt-get update
apt-get remove linux-image-2.6-amd64
apt-get install linux-image-2.6-amd64
aptitude full-upgrade

(Note: The above must be done in that order, otherwise you will hit udev/kernel bug).

Sure enough, it came back up, no problems.

Spent last couple days moving all my stats and monitoring over to the hetzner VM, pleased to take some load/work off my atom servers; and have a fourth location for DNS/backups.

Thursday, November 18, 2010

Yaketystats & Jart

I've now switched my stats collection for the third time in the past year, and I'm hoping this will be the last time.

I'm not going to cover configuration or installation here, as their wiki is excellent, I will however post a couple of things which got me, and a couple of changes I made.

Yaketystats Homepage
Five minute demo video (well worth watching)

I really liked collectd,  nice plugins collection - there was a plugin for everything I wanted/cared about.
Written in C, push-based, fast, and regular polling intervals for HD stats (and who doesn't want HD these days :)

The catch is the frontend, they all seem to be severely lacking.

Various areas where CGP lacks was pointed out at my new job, mainly the inability to compare multiple graphs - at a specific point in time, combine values into one graph, and be able to search/combine graphs easily (regex or similar). After much searching for an alternative frontend, I finally came across Jart, which is the frontend for Yaketystats.

Yaketystats 2.1 has been around for over two years (Yaketystats itself has been around for approximately five altogether internally at USG), but it doesn't seem to have gotten the same community attention as munin or collectd, which is a shame.

Yaketystats is all bash/perl/php, and it's all quite simply put together.
That's not to say it's simple -- but it's very easy to modify and add onto.
They have put thought into the backend system, stats are collected on each host, stored (plain text), and a cron runs which posts the stats to a php page on the server. If it cannot contact the server it just keeps queueing up the stats till it can.

Once the stats are submitted, a couple of jobs are run on them, eventually outputting them as RRDs.
I noticed less load from this system than collectd/rrdcached for the same number of hosts.

This is where the real magic starts, Jart is their frontend.
It can draw graphs in three different ways:
1) You draw 'all' the graphs for cpu for a host - this draws a graph containing idle, iowait, user, sys for a system (combined for all CPUs).
2) You select which stats you'd like, specifically, from any number of hosts, which it will then put in one graph
3) Regex!  You can build graphs using Perl's regex -- very, very cool.

You can modify any graph you create, on the fly -- change from using lines, to stacks to area, negative or positive for different values (e.g. received traffic is given in negative, transmitted given in positive values on the graph), change colour etc - you have full control.
You can then save whichever graphs you have open as a playlist - meaning you can re-open these graphs easily, with all your settings.

So far been using this two or three days, and converted a couple other people to it too.

Couple of changes I made for my setups:
patch to add limits tab to graphs.

Remove the info at end of each graph (generation time, most recent rrds etc), modify index.php:
$tmp  = array_merge($this->args,$this->defs,$this->lines,$this->events,$this->comments);
to read
$tmp  = array_merge($this->args,$this->defs,$this->lines,$this->events);

Also, we wanted to sort/break up our server names by the fourth character, and not the first, again requires modifying index.php:
var nl = me.charAt(0);
var nl = me.charAt(3);
Note, found a bug where it breaks completely if you have a username with less characters than it looks for above (4 in this case).

The last change is to remove the capitalisation of the hostnames - comment out the two lines above the charAt line:
mefirst = me.replace(/(.).*/,'$1').toUpperCase();
me      = me.replace(/.(.*)/,mefirst + '$1');

Sunday, November 7, 2010

Collectd - Frontent & performance modifications

Been using collectd a fair while now, one of the largest drawbacks of it the lack of a good frontend.

I recently found Collectd Graph Panel (CGP) which is by far the best frontend I've used for it yet (oddly enough the handiest to setup also!)

It doesn't seem to be under overly active development, but lot of it works, and support for more plugins is getting added to the git repository (which I'd suggest using as opposed to the 0.3 release).

Another modification to collectd I made recently was moving from using the rrd plugin to the rrdcached plugin, which lets the rrdcached service handle the rrd data.
Caching seems to be done better using this service as opposed to within collectd.

By default rrd storage location is different, so to migrate just rsync (assuming defaults) from /var/lib/collectd/ to /var/lib/rrdcached/db/collectd/ and restart collectd (after disabling rrd plugin & enabling rrdcached plugin).

Migrating to nginx from apache (with PHP + CGI support)

Was always aware of nginx, and several people mentioned it, eventually it hit home (especially after reading some interesting LOPSA posts on their mailing list.

For this example, I'll detail my setup for  PHP, CGI, nginx status page and restricting by address (IPv4 & IPv6).

The following is done on Debian Squeeze, I have tested the config with nginx version 0.7 & 0.8.

The following dependencies are required: php-fastcgi fcgiwrap spawn-fcgi php5-cgi

Although not in the Debian repository, I found php-fastcgi packaged, and a friend ended up packaging nginx 0.8 (0.7 is available in the repository).

I went with version 0.8 of nginx as it allows you to restrict access to sites using IPv6 addresses, as opposed to IPv4 only.

Firstly, the /etc/init.d/php-fastcgi wrapper must be modified - by default it listens on a port, and only starts one process - I want to use socket-based communication between nginx & start multiple processes (for me, 3 is enough).


The following is my nginx configs from /etc/nginx/sites-enabled/

server {
        listen; ## listen for ipv4
        listen   [::1]:80;
        server_name  localhost;
        location /nginx_status {
                stub_status on;
                access_log   off;
                allow ::1;
                deny all;
server {
        listen   80;
        listen   [::]:80;
        root   /var/www/stats;
        access_log  /var/log/nginx/;
        location / {
                index  index.html index.php;
                allow ::1;
                deny all;
        location ~ \.php$ {
                gzip off;
                fastcgi_pass unix:/var/run/php-fastcgi.socket;
                fastcgi_index index.php;
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        location /smokeping/ {
                index smokeping.cgi;
                gzip off;
                if ($uri ~ "/smokeping/smokeping.cgi") {
                        fastcgi_pass unix:/var/run/fcgiwrap.socket;
 I symlinked /usr/share/smokeping/www to /var/www/smokeping & also symlinked /usr/share/smokeping/cgi-bin/smokeping.cgi to /usr/share/smokeping/www/