Archive for April, 2011

Implement a high available Cobbler provisioning system

Sunday, April 24th, 2011

A not so well known feature of Cobbler is its replication facility. It allows you to create a high available system provisioning system. The whole set up is straight forward.

Background
Today people tend to NOT backup systems, only data is being backed up. In the case of a system failure, they just re-provisioning the system and automatically configure it with configuration management tools such as Puppet, CFengine or RHN Satellite.

You not only need to have your configuration management system(s) to be redundant, you also need to replicate your Cobbler provisioning systems.

It even works in a set up where you have multiple datacenters for disaster recovery, having one or two Cobbler servers at each site.

Luckily, Cobbler comes with that feature out-of-the-box.

Assumptions

  • Both Cobbler servers are in the same network
  • Your master cobbler has a DNS-entry cobbler-master, your slave is cobbler-slave

Lets do it

Unfortunately, Cobbler replication, at the moment, does not work when SELinux is running in enforcing mode. You need to turn it off on both Cobbler servers.

setenforce 0

Hopefully this will change soon.

For a initial replication just run:

cobbler replicate --master=cobbler-master --sync-all

On the slave server. It is recommended to run this command nightly by a cronjob to ensure you have the latest RPM-packages of your distros in sync. Feel free to run this command manually at every time if you think it is needed to immediately sync the whole Cobbler system.

Update on-the-fly
You can use a Cobbler trigger script that automatically connects to the slave and triggers a replication with the master whenever a system is added or deleted.

I’m using the following script to do so:

#!/bin/bash
#
# This is a Cobbler trigger script that triggers a replication on the slave
# server. You need to create a private/public key-pair on the master and 
# add the public key to ~/.ssh/authorized_keys on the slave

SLAVE=cobbler-slave

ssh $SLAVE "cobbler replicate --master=cobbler-master --systems=* --prune"
ssh $SLAVE "cobbler sync"

Put this script in /var/lib/cobbler/triggers/add/system/post and create a symlink in /var/lib/cobbler/triggers/delete/system/post on your master server.

Security
Cobbler handles the file /etc/rsyncd.conf. So far so good. You need to be aware that EVERY host can rsync your distros, kickstarts, snippets etc unless you take some security measures with some firewall and/or other restrictions.

Possibly the simplest way is configuring xinetd.

A sample /etc/xinetd.d/rsync, where 192.0.2.2 is the IP of your slave, would be:

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
#       allows crc checksumming etc.
service rsync
{
        disable         = no
        only_from       = 192.0.2.2
        flags           = IPv6
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

DHCP redundancy
You also need to operate a dhcp server on both Cobbler servers. If you have both Cobbler servers on the same subnet, you may have a look how to configure ISC dhcpd. A good guide (untestet) is at http://www.madboa.com/geek/dhcp-failover/. The lazy ones just turn off dhcp on the slave and activate it as needed.

Conclusion
As I wrote before, Cobbler is just great. Not only “batteries are included”, high availability, disaster safety and other cool stuff is included as well and can be quickly implemented.

Having fun? I rarely needed to re-provision systems because they went foobar and hopefully I will never experience a disaster such a burned-down, exploded or whatever datacenter. Cobbler does not protect you from such disasters, it helps you to recover.

I got employed by Red Hat

Thursday, April 21st, 2011

This is pretty cool: End of March I signed a contract with Red Hat as a senior Linux consultant. It is not just “another new job”. It is cool for (at least) two reasons: First reason is that Red Hat is not “just another company”, it is Red Hat which is not very comparable to other employers, it is THE Linux and open source company, for me as a open source guy, this is perfect. The second reason is: I’m moving from Zurich in Switzerland to Berlin in Germany.

So, two major changes in my life at the same time. I’m looking forward to the challenges that are waiting for me.

I’ll continue to work at Siemens IT Solutions and Services AG until approx. mid of June and start working at Red Hat at 1st of July.

From May, 09 to May 15. I’ll be the first time in Berlin to have a look at the city and its different suburbs. I’ll also be there to organize some stuff required to settle in Berlin. In the same time, Europe’s biggest Linux conference will be held in Berlin, the “Linux Tag”. I guess I’ll have a lot of fun, and maybe meet some of my future workmates.

It is hard for me to leave my country, I have a lot of friends here. On the other hand, Berlin is just about 1.5h away by plane. As a consultant, I’m travelling a lot. Because of that, it would not be that easy to build up a social network (I mean real-life-stuff, not Facebook) in Berlin.

It also is not easy for me to leave Siemens, I’m involved in a very cool project with the Swiss government (all Systems will be RHEL6) and I also have friends and nice workmates at work which I’m going to leave.

I already know quite some people at Red Hat, they are all nice and I guess some of them will get good friends over the time.

Having fun?

Absolutely guaranteed!

I voted for beefy miracle

Thursday, April 7th, 2011

Beefy miracle

 

There is a open poll on voting for a name for Fedora 16. I gave my vote to Beefy Miracle. Why I voted for Beefy Miracle? Because it is cool, geeky, freaky, I’m loving hot dogs and it is something new.

The Fedora distribution is geeky, freaky and open to new stuff.

Having fun? Of course!