Upgrading RHN Satellite 5.5 to 5.6

Redhat released version 5.6 of the Redhat Satellite. Time to have a closer look to it and how to upgrade from version 5.5.

New features

  • Finally PostgreSQL support is mature enough for Enterprise usage. No need of a closed source data base anymore. This also brings a lot of new capabilities such as online backups which before was only available using an external Oracle Database which needs the availability of a DBA.

    PostgreSQL also brings some performance benefits over the embedded Oracle database as delivered with 5.5 and earlier. Disclaimer: I did not made any benchmarks, but it “feels” much faster.

  • If you are using the multi-org feature, you may be happy about enhancements for Inter-Satellite-Sync (ISS). Now you can define access rights for different software channels for different organizations.
  • It is not a new feature, but now it is supported: cobbler buildiso. It is a handy solution if you can not use PXE boot in your environment. cobbler buildiso generates a small boot image which allows you to select the installation of a system from a boot menu.
  • Intergrated System Asset Manager (SAM) which is based on Candlepin and allows you assess your system landscape for subscription compliance.
  • Upgrading from RHN Satellite 5.5
    The first thing that you probably would ask: Is it possible and supported to migrate from the Embedded Oracle Database to PostgreSQL? Is it hassle free and bullet-proof? Yes it is.

    Keep in mind

  • As always: Have a look to the product documentation before doing anything on a production Satellite.
  • Create a new RHN Satellite Certificate at access.redhat.com
  • Download the ISO image for 5.6
  • ensure having a recent database backup
  • ensure having a recent backup of your /etc/rhn directory as well as /var/lib/cobbler
  • Update your existing Satellite 5.5 with the latest available patches
  • Delete unnecessary software channels from the Satellite for faster DB migration
  • Delete old Snapshots to minimize database data to be migrated
  • Make enough storage available to migrate from embedded Oracle to PostgreSQL. It takes roughly about the same amount of storage for the data. The PostgreSQL database stores its data in /var/lib/pgsql.
  • Install the latest available package rhn-upgrade: yum install rhn-upgrade

    Lets do it, Perparation work

    First of all, create a database backup of your embedded Oracle Database:

    [root@rhnsat ~]# rhn-satellite stop
    [root@rhnsat ~]# su - oracle -c "db-control backup /path/to/your/backup/directory"
    [root@rhnsat ~]# su - oracle -c "db-control verify /path/to/your/backup/directory"
    [root@rhnsat ~]# rhn-satellite start

    Backup the rest of your Satellite:

    [root@rhnsat ~]# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")
    [root@rhnsat ~]# cp -rp /var/lib/cobbler /var/lib/cobbler-$(date +"%F")
    [root@rhnsat ~]# cp -rp /etc/cobbler /etc/cobbler-$(date +"%F")

    Update your RHN Satellite 5.5 with the latest available patches and reboot:

    [root@rhnsat ~]# yum -y update && reboot

    Ensure the latest schema updates have been applied. The output should read as follow:

    [root@rhnsat ~]# spacewalk-schema-upgrade 
    You are about to perform upgrade of your satellite-schema.
    For general instructions on Red Hat Satellite schema upgrade, please consult
    the following article:
    Hit Enter to continue or Ctrl+C to interrupt: 
    Schema upgrade: [satellite-schema-] -> [satellite-schema-]
    Your database schema already matches the schema package version [satellite-schema-].
    [root@rhnsat ~]#

    It is always a good idea to restart a software and check if all is working as expected *before* doing an upgrade. So you can pinpoint problems better if there are some.

    [root@rhnsat ~]# rhn-satellite restart

    Review your list of software channels and delete unused ones. This example will delete the channel rhel-i386-rhev-agent-6-server:

    [root@rhnsat ~]# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
    Deleting package metadata (20):
    Removing:         ######################################## - complete
    [root@rhnsat ~]#  

    Delete old system snapshots not used anymore. The following example deletes all snapshots which are older than one month:

    [root@rhnsat ~]# sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000")

    Update the rhn-update package to the latest available:

    yum install rhn-upgrade

    After installing the the rhn-upgrade package, the SQL scripts needed for the DB migration are installed as well as some documentation you should read. They are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

    Upgrade Procedure

    Mount the downloaded ISO image:

    [root@rhnsat ~]# mount satellite-5.6.0-20130927-rhel-6-x86_64.iso /mnt -o loop && cd /mnt
    [root@rhnsat mnt]# 

    If you operate your Satellite behind a proxy, you need to upgrade it in disconnected mode, if not, ignore the –disconneded parameter.

    [root@rhnsat mnt]# ./install.pl --disconnected --upgrade
    * Starting the Spacewalk installer.
    * Performing pre-install checks.
    * Pre-install checks complete.  Beginning installation.
    * RHN Registration.
    ** Registration: Disconnected mode.  Not registering with RHN.
    * Upgrade flag passed.  Stopping necessary services.
    * Purging conflicting packages.
    * Checking for uninstalled prerequisites.
    ** Checking if yum is available ...
    There are some packages from Red Hat Enterprise Linux that are not part
    of the @base group that Satellite will require to be installed on this
    system. The installer will try resolve the dependencies automatically.
    However, you may want to install these prerequisites manually.
    Do you want the installer to resolve dependencies [y/N]? y
    * Installing RHN packages.
    * Now running spacewalk-setup.
    * Setting up Selinux..
    ** Database: Setting up database connection for PostgreSQL backend.
    ** Database: Installing the database:
    ** Database: This is a long process that is logged in:
    ** Database:   /var/log/rhn/install_db.log
    *** Progress: #
    ** Database: Installation complete.
    ** Database: Populating database.
    *** Progress: ###################################
    * Database: Starting Oracle to PostgreSQL database migration.
    ** Database: Starting embedded Oracle database.
    ** Database: Trying to connect to Oracle database: succeded.
    ** Database: Migrating data.
    *** Database: Migration process logged at: /var/log/rhn/rhn_db_migration.log
    ** Database: Data migration successfully completed.
    ** Database: Stoping embedded Oracle database.
    * Setting up users and groups.
    ** GPG: Initializing GPG and importing key.
    * Performing initial configuration.
    * Activating Red Hat Satellite.
    ** Certificate not activated.
    ** Upgrade process requires the certificate to be activated after the schema is upgraded.
    * Enabling Monitoring.
    * Configuring apache SSL virtual host.
    Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? y
    * Configuring tomcat.
    ** /etc/sysconfig//tomcat6 has been backed up to tomcat6-swsave
    ** /etc/tomcat6//tomcat6.conf has been backed up to tomcat6.conf-swsave
    Reversed (or previously applied) patch detected!  Skipping patch.
    1 out of 1 hunk ignored -- saving rejects to file web.xml.rej
    * Configuring jabberd.
    * Creating SSL certificates.
    ** Skipping SSL certificate generation.
    * Deploying configuration files.
    * Update configuration in database.
    * Setting up Cobbler..
    cobblerd does not appear to be running/accessible
    Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? 
    This portion of the Red Hat Satellite upgrade process has successfully completed.
    Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
    for any remaining steps in the process.
    [root@rhnsat mnt]# 

    Depending on the size of your database and the speed of your disks, the upgrade procedure can take many hours.

    The next step is having a look at diff /etc/rhn/rhn.conf /etc/rhn-$(date +”%F”)/rhn.conf
    and edit /etc/rhn/rhn.conf accordingly. You will probably see missing things such as proxy, server.satellite.rhn_parent etc. Also change the setting disconnected to 0.

    After checking and correcting the config file you can activate the Satellite:

    [root@rhnsat ~]# rhn-satellite-activate --rhn-cert=/root/rhns-cert56.cert --ignore-version-mismatch

    After the activation the System is subscribed to the Softwarechannel “redhat-rhn-satellite-5.6-server-x86_64-6”, now bring the Satellite to latest available patchlevel:

    [root@rhnsat ~]# yum -y update 

    Stop and disable Oracle
    Bofore doing any Database related actions its better to stop the old Oracle Database to be sure all is now running on PostgreSQL.

    [root@rhnsat ~]# service oracle stop
    Shutting down Oracle Net Listener ...                      [  OK  ]
    Shutting down Oracle DB instance "rhnsat" ...              [  OK  ]
    [root@rhnsat ~]# chkconfig oracle off
    [root@rhnsat ~]# rhn-satellite restart


    Check if your database schema is up-to-date:

    root@rhnsat ~]# spacewalk-schema-upgrade 
    You are about to perform upgrade of your satellite-schema.
    For general instructions on Red Hat Satellite schema upgrade, please consult
    the following article:
    Hit Enter to continue or Ctrl+C to interrupt: 
    Schema upgrade: [satellite-schema-] -> [satellite-schema-]
    Your database schema already matches the schema package version [satellite-schema-].
    [root@rhnsat ~]# 

    Rebuild the search index:

    [root@rhnsat ~]# service rhn-search cleanindex
    Stopping rhn-search...
    Stopped rhn-search.
    Starting rhn-search...
    [root@rhnsat ~]# 

    Recreate the software channel meta data:

    [root@rhnsat doc]# /etc/sysconfig/rhn/satellite-upgrade/scripts/regenerate-repodata -a
    Scheduling repodata creation for 'rhel-x86_64-server-supplementary-6'
    Scheduling repodata creation for 'rhel-x86_64-server-6'
    Scheduling repodata creation for 'rhn-tools-rhel-x86_64-server-6'
    [root@rhnsat doc]# 

    Check functionality
    Before removing the Oracle Database, run your tests to validate the Satellites functionality. Please proceed as stated in /etc/sysconfig/rhn/satellite-upgrade/doc/verification.txt

    This is an important point, as we are getting rid of the Oracle database later on. To be sure all is working as expected, do a complete functionality test for the important things.

    To be on the safe side, let the Satellite run for a few days with Oracle still installed.

    Getting rid of Oracle

    Please read /etc/sysconfig/rhn/satellite-upgrade/doc/satellite-upgrade-postgresql.txt first!

    [root@rhnsat ~]# yum remove *oracle*

    Getting rid of the last Oracle bits:

    [root@rhnsat ~]# rm -rf /rhnsat /opt/apps/oracle /usr/lib/oracle/

    Having fun with a faster Satellite with an open source database 🙂

    I take no responsibility about damaged Satellites, lost data etc. in doubt, stick on the official product documentation at http://access.redhat.com

Intercepting proxies and spacewalk-repo-sync

More and more companies are using intercepting proxies to scan for malware. Those malware scanners can be problematic due to added latency.

If you using spacewalk-repo-sync to synchronize external yum repositories to your custom software channels and experience the famous message [Errno 256] No more mirrors to try in your log files, then you need to configure spacewalk-repo-sync.

Unfortunately the documentation for that is a bit hidden in the man page. You need to create a directory and create a file.

mkdir /etc/rhn/spacewalk-repo-sync/

Create the configuration item:

echo "[main]" >> /etc/rhn/spacewalk-repo-sync/yum.conf
echo timeout=300 >> /etc/rhn/spacewalk-repo-sync/yum.conf

You need to experiment a bit with the value of the timeout setting, 5min should be good enough for most environments.

/etc/rhn/spacewalk-repo-sync/yum.conf has the same options like yum.conf, have a look for more information in the man page.

Have fun 🙂

Automated disk partitioning on virtual machines with Cobbler

The default Cobbler Snippets just do simple auto partitioning. For a more sophisticated partition layout you need to know what kind of VM you are going to install. KVMs and RHEVs device name is /dev/vda, Xen uses /dev/xvda and ESX /dev/sda.

Luckily this can be figured out automatically, those different virtualization vendors are using its own MAC prefixes. So we can add two nice small Cobbler snippets to do the job. In this example, I call them hw-detect and partitioning.


#set $mac = $getVar('$mac_address_eth0')
#if $mac
#set $mac_prefix = $mac[0:8]
#if $mac_prefix == "00:1a:4a"
# This is a RHEV virtual machine
#set global $machinetype = 'kvm'

#else if $mac_prefix == "52:54:00"
# This is a KVM/Qemu virtual machine
#set global $machinetype='kvm'

#else if $mac_prefix == "00:16:3e"
# This is a XEN virtual machine
#set global $machinetype='xen'
#else if $mac_prefix == "00:50:56"
# This is a ESX virtual machine
#set global $machinetype = 'esx'

# #This is a physical machine
#set global $machinetype = 'physical'
#end if
#end if


#if $machinetype == 'kvm'
#set $disk='vda'
#else if $machinetype == 'xen'
#set $disk = 'xvda'
#set $disk = 'sda'
#end if
# Lets install the system on /dev/$disk
part /boot      --fstype ext2 --size=250 --ondisk=$disk
part pv.0       --size=1 --grow --ondisk=$disk

volgroup vg_${name} pv.0

logvol /        --fstype ext4 --name=lv_root    --vgname=vg_${name} --size=4096
logvol /home    --fstype ext4 --name=lv_home    --vgname=vg_${name} --size=512 --fsoption=nosuid,nodev,noexec
logvol /tmp     --fstype ext4 --name=lv_tmp    --vgname=vg_${name} --size=1024 --fsoption=nosuid,nodev,noexec
logvol /var     --fstype ext4 --name=lv_var    --vgname=vg_${name} --size=2048 --fsoption=nosuid,nodev,noexec
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_${name} --size=2048

An additional “feature” of the partitioning Snippet is: It sets up the Volume Group name according to your systems name. This is the unofficial standard since quite some time. It also sets some more secure mount options. Review them carefully if they make sense for you and edit them as needed.

The next step is to configure your kickstart template.

Standalone Cobbler
On a standalone Cobbler server edit /var/lib/cobbler/kickstart/your-kick-start-template.ks

# Detect the used hardware type
# Set up default partitioning

Bundled Cobbler
When using cobbler bundled with Spacewalk or Red Hat Satellite, you need to edit the Kickstart profile in the WebUI.

Navigate to Systems -> Kickstart -> Profile. Select the Kickstart profile to be modified -> System Details -> Partitioning.

Copy the two Snippets in /var/lib/cobbler/spacewalk/1, where 1 is representing your OrgId.

Alternatively you can edit them in the WebUI as well.

To check if all is working as expected, add a system to Cobbler using the Command Line Interface and have a look to the rendered Kickstart file. This can be easily done with cobbler system getks --name=blah.

Happy System installing….

Have fun 🙂

Cross distribution system management with Spacewalk

In a perfect world, all systems in a data centre are running the same Linux operating system, a homogeneous system landscape. In real life things are working differently. Windows systems are out of focus in this post, lets concentrate on Linux systems.

Most companies with a large Linux base are either RHEL shops or using SLES. A lot of RHEL users have some SLES systems running and so are SLES users running some RHEL systems. Some companies have additional systems running Debian.

How to handle those heterogeneous system landscapes? Those real world scenarios? Lets assume a company runs 500 RHEL systems, 20 SLES systems and some 10 Debian systems.

At the moment, for the base software management subscription such Linux users are spending a lot of money for RHN Satellite and SUSE Manager. Additionally there are per-system costs for management, provisioning and other modules. The Debian systems are handled manually. A lot of additional costs for a few out-of-strategy systems.

The solution is Spacewalk, the upstream project of the RHN Satellite which is at the same time the upstream for the recently released SUSE Manager. While SUSE offers support for RHEL systems, Red Hat does not (yet) offer support for SLES systems for RHN Satellite.

In Spacewalk Version 1.4 code contributions from SUSE are included and a student at Brno University of Technology contributed Debian support for Spacewalk as part of his master thesis.

While the support for SUSE is already quite stable, the Debian related code still have some rough edges. No wonder, SUSE is using RPM for its packaging wile Debian has its own packaging system. This makes it much easier for SUSE to get Spacewalk ready for its distribution.

At the moment, one can call the Debian support still as experimental, but the goal for the Spacewalk project is to have it fully functional in future releases.

The goal should be that both of the management system from the major enterprise Linux vendors, Red Hat and SUSE should support each others distribution for its Spacewalk based products. Debian is a niche player in the enterprise Linux environment and should also be supported by both products, RHN Satellite and SUSE manager. Nobody does expected to get system support for those distributions by the competing distribution, but having support for the management of it.

Further readings:
Registering Clients
Deb support in Spacewalk

Have fun!