A not so well known feature of Cobbler is its replication facility. It allows you to create a high available system provisioning system. The whole set up is straight forward.
Background
Today people tend to NOT backup systems, only data is being backed up. In the case of a system failure, they just re-provisioning the system and automatically configure it with configuration management tools such as Puppet, CFengine or RHN Satellite.
You not only need to have your configuration management system(s) to be redundant, you also need to replicate your Cobbler provisioning systems.
It even works in a set up where you have multiple datacenters for disaster recovery, having one or two Cobbler servers at each site.
Luckily, Cobbler comes with that feature out-of-the-box.
Assumptions
- Both Cobbler servers are in the same network
- Your master cobbler has a DNS-entry cobbler-master, your slave is cobbler-slave
Lets do it
Unfortunately, Cobbler replication, at the moment, does not work when SELinux is running in enforcing mode. You need to turn it off on both Cobbler servers.
setenforce 0
Hopefully this will change soon.
For a initial replication just run:
cobbler replicate --master=cobbler-master --sync-all
On the slave server. It is recommended to run this command nightly by a cronjob to ensure you have the latest RPM-packages of your distros in sync. Feel free to run this command manually at every time if you think it is needed to immediately sync the whole Cobbler system.
Update on-the-fly
You can use a Cobbler trigger script that automatically connects to the slave and triggers a replication with the master whenever a system is added or deleted.
I’m using the following script to do so:
#!/bin/bash # # This is a Cobbler trigger script that triggers a replication on the slave # server. You need to create a private/public key-pair on the master and # add the public key to ~/.ssh/authorized_keys on the slave SLAVE=cobbler-slave ssh $SLAVE "cobbler replicate --master=cobbler-master --systems=* --prune" ssh $SLAVE "cobbler sync"
Put this script in /var/lib/cobbler/triggers/add/system/post and create a symlink in /var/lib/cobbler/triggers/delete/system/post on your master server.
Security
Cobbler handles the file /etc/rsyncd.conf. So far so good. You need to be aware that EVERY host can rsync your distros, kickstarts, snippets etc unless you take some security measures with some firewall and/or other restrictions.
Possibly the simplest way is configuring xinetd.
A sample /etc/xinetd.d/rsync, where 192.0.2.2 is the IP of your slave, would be:
# default: off
# description: The rsync server is a good addition to an ftp server, as it \
# allows crc checksumming etc.
service rsync
{
disable = no
only_from = 192.0.2.2
flags = IPv6
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = --daemon
log_on_failure += USERID
}
DHCP redundancy
You also need to operate a dhcp server on both Cobbler servers. If you have both Cobbler servers on the same subnet, you may have a look how to configure ISC dhcpd. A good guide (untestet) is at http://www.madboa.com/geek/dhcp-failover/. The lazy ones just turn off dhcp on the slave and activate it as needed.
Conclusion
As I wrote before, Cobbler is just great. Not only “batteries are included”, high availability, disaster safety and other cool stuff is included as well and can be quickly implemented.
Having fun? I rarely needed to re-provision systems because they went foobar and hopefully I will never experience a disaster such a burned-down, exploded or whatever datacenter. Cobbler does not protect you from such disasters, it helps you to recover.