Posts Tagged ‘RHN Satellite’

Upgrading RHN Satellite 5.6 to 5.7

Sunday, February 8th, 2015

This post guides you trough the upgrade procedure for a Satellite 5.6 using the embedded database on RHEL6-x86_64. Further it guides you to setup of Kerberos authentication of Satellite users with IPA.

Recently Redhat released Satellite Server 5.7. Despite Satellite 5.x will be outphased in the next few years, there are plenty of new features. The most significant new features are:

  • Upgraded PostgreSQL to 9.2
  • Authentication via IPA/SSSD/Kerberos
  • IPMI support
  • Renewed WebUI
  • Readonly API users

And finally… drum roll…. formal support for spacecmd :-)

As always when you plan to upgrade your Satellite server to the latest version, you need to do some preparations first.

Download the ISO
As usual, visit the Download site and make sure you select 5.7 and the architecture fitting you system (x86_64 or S390)

Get a new Satellite Certificate
Satellite 5.7 needs a new certificate to get it activated. You can create it by your own at the Subscription Management Application site, ensure you attach enough subscriptions to your Satellite server(s). Alternatively open a support case.

Backup
Usually an upgrade runs smooth, but just in case… it is recommended practice to have a recent backup ready. If your Satellite is running on a virtual machine, power off, snapshot and power on to have a consistent backup ready. For physical systems, db-control and the choice of your backup software need to be visited.

Backup the rest of your Satellite:

Create a copy of your rhn configuration directory as we need some information from the old files after the upgrade.

[root@rhnsat ~]# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")

Update your OS and Satellite
First step is to update the operating system and the Satellite 5.6 and apply the latest database schema updates as well.

yum -y update && reboot

To update the database schema, run the following command. Ideally it looks as follows:

root@rhnsat ~]# spacewalk-schema-upgrade 

You are about to perform upgrade of your satellite-schema.

For general instructions on Red Hat Satellite schema upgrade, please consult
the following article:

https://access.redhat.com/knowledge/articles/273633

Hit Enter to continue or Ctrl+C to interrupt: 
Schema upgrade: [satellite-schema-5.6.0.21-1.el6sat] -> [satellite-schema-5.6.0.21-1.el6sat]
Your database schema already matches the schema package version [satellite-schema-5.6.0.21-1.el6sat].
[root@rhnsat ~]# 

Functionality Check
It is recommended to restart and check a softwares functionality before upgrading to be able to pinpoint problems if there are some.

[root@rhnsat ~]# rhn-satellite restart

Cleanup
Its a good idea to review the software channels in use and delete unused channels as this can free up quite some diskspace and reduces the size of the database significantly.

[root@rhnsat ~]# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
Deleting package metadata (20):
                  ________________________________________
Removing:         ######################################## - complete
[root@rhnsat ~]#

Delete old system snapshots not used anymore. The following example deletes all snapshots which are older than one month:

[root@rhnsat ~]# sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000" 

Remove spacecmd from EPEL
Most Satellite users have spacecmd installed from EPEL. Its a good idea to remove it to avoid conflicts. It is also important to disable the EPEL repositories on Satellite servers as a simple yum update can bring your Satellite server into trouble.

RTFM
If not done yet, install the rhn-upgrade package which contains the instructions how to proceed.

yum -y install rhn-upgrade

The package contains not only SQL- and other useful scripts needed for the upgrade but also important documents to read. The are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

For most users, the document satellite-upgrade-postgresql.txt applies.

Do not forget to read the updated product documentation as well:

Changing your file system layout
As there will be an updated PostgreSQL version needed which is part of the Software Collection and not installable from the base channel, you need to add a new file system in /opt/rh.
The new database is about the same size as before. Check your used disk space at /var/lib/pgsql

[root@rhnsat ~]# lvcreate /dev/vg_data -n lv_opt_rh -L 17G 
[root@rhnsat ~]# mkfs.ext4 /dev/vg_data/lv_opt_rh
[root@rhnsat ~]# tune2fs -c0 -i0  /dev/vg_data/lv_opt_rh

Exit your /etc/fstab accordingly and mount the file system with mount -a to check if it working as expected.

Lets do it
Mount the ISO image and run the installer.

[root@rhnsat ~]# mount satellite-5.7.0-20150108-rhel-6-x86_64.iso /mnt -o loop
[root@rhnsat ~]# cd /mnt
[root@rhnsat mnt]# 

If you are using a proxy to sync your satellite, provide the --diconnected flag.

[root@rhnsat mnt]# ./install.pl --upgrade --disconnected
* Starting Red Hat Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHN Registration.
** Registration: Disconnected mode.  Not registering with RHN.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Installing RHN packages.
* Now running spacewalk-setup.
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
*** Upgrading embedded database.
** Database: Populating database.
** Database: Skipping database population.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating Red Hat Satellite.
** Certificate not activated.
** Upgrade process requires the certificate to be activated after the schema is upgraded.
* Enabling Monitoring.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? y
* Configuring tomcat.
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
task started: 2015-02-08_154708_sync
task started (id=Sync, time=Sun Feb  8 15:47:08 2015)
running pre-sync triggers
cleaning trees
removing: /var/www/cobbler/images/ks-rhel-x86_64-es-4-u6
[..]
running shell triggers from /var/lib/cobbler/triggers/change/*
*** TASK COMPLETE ***
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? y
This portion of the Red Hat Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.
[root@rhnsat mnt]#

The next step is having a look at diff /etc/rhn/rhn.conf /etc/rhn-$(date +”%F”)/rhn.conf
and edit /etc/rhn/rhn.conf accordingly. You will probably see missing things such as proxy, server.satellite.rhn_parent etc. Also change the setting disconnected to 0.

Activate the updated Satellite server
To subscribe the Satellite server to the appropriate software channels, it must be activated. Since it was activated before, the --ignore-version-mismatch parameter must be provided.

[root@rhnsat ~]# rhn-satellite-activate --rhn-cert=rhn-satellite57-2015-02-08.xml --ignore-version-mismatch

Initial Update of Software and database schema
There is a good chance that there are updates available for the Satellite Server as the ISO image will not be updated that often.

[root@rhnsat ~]# yum -y update

Even if no update was installed, there is a schema update available:

[root@rhnsat ~]# spacewalk-schema-upgrade 
Schema upgrade: [satellite-schema-5.6.0.21-1.el6sat] -> [satellite-schema-5.7.0.11-1.el6sat]
Searching for upgrade path: [satellite-schema-5.6.0.21-1] -> [satellite-schema-5.7.0.11-1]
Searching for upgrade path: [satellite-schema-5.6.0.21] -> [satellite-schema-5.7.0.11]
Searching for upgrade path: [satellite-schema-5.6.0] -> [satellite-schema-5.7.0]
Searching for upgrade path: [satellite-schema-5.6] -> [satellite-schema-5.7]
The path: [satellite-schema-5.6] -> [satellite-schema-5.7]
Planning to run spacewalk-sql with [/var/log/spacewalk/schema-upgrade/20150208-155657-script.sql]

Plase make sure you have a valid backup of your database before continuing.

Hit Enter to continue or Ctrl+C to interrupt: 
Executing spacewalk-sql, the log is in [/var/log/spacewalk/schema-upgrade/20150208-155657-to-satellite-schema-5.7.log].
The database schema was upgraded to version [satellite-schema-5.7.0.11-1.el6sat].
[root@rhnsat ~]# 

Cleanup
After startarting the Satellite Server, the package meta data should be automatically recreated. If not, run
/etc/sysconfig/rhn/satellite-upgrade/scriptsregenerate-repodata manually.

Rebuild the search index:

[root@rhnsat ~]# service rhn-search cleanindex

You don’t need to remove the old PostgreSQL version, this is done automatically.

Using IPA and Kerberos for authentication
Before configure the Satellite Server to use IPA, make sure it is enrolled and the HTTP service principal exists. If not, add it with the following command:

[root@ipa1 ~]# ipa-addservice HTTP/rhnsat.example.com@EXAMPLE.COM

Next will be getting a Kerbros Ticket of a user allowed to create Keytabs. In this example it is the user admin.

[root@rhnsat ~]# kinit admin
Password for admin@EXAMPLE.COM: 
[root@rhnsat ~]# 

Afterwards, run the setup script:

[root@rhnsat ~]# spacewalk-setup-ipa-authentication
Enabling authentication against [ipa2.example.com].
Retrieving HTTP/ service keytab into [/etc/httpd/conf/http.keytab] ...
Keytab successfully retrieved and stored in: /etc/httpd/conf/http.keytab
changed ownership of `/etc/httpd/conf/http.keytab' to apache
Configuring PAM service [spacewalk].
Will install additional packages ...
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package mod_auth_kerb.x86_64 0:5.4-13.el6 will be installed
---> Package mod_authnz_pam.x86_64 0:0.9.2-1.el6 will be installed
---> Package mod_intercept_form_submit.x86_64 0:0.9.7-1.el6 will be installed
---> Package mod_lookup_identity.x86_64 0:0.9.2-1.el6 will be installed
---> Package sssd-dbus.x86_64 0:1.11.6-30.el6_6.3 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================
 Package                                      Arch                      Version                               Repository                               Size
============================================================================================================================================================
Installing:
 mod_auth_kerb                                x86_64                    5.4-13.el6                            rhel-x86_64-server-6                     30 k
 mod_authnz_pam                               x86_64                    0.9.2-1.el6                           rhel-x86_64-server-6                     13 k
 mod_intercept_form_submit                    x86_64                    0.9.7-1.el6                           rhel-x86_64-server-6                     17 k
 mod_lookup_identity                          x86_64                    0.9.2-1.el6                           rhel-x86_64-server-6                     19 k
 sssd-dbus                                    x86_64                    1.11.6-30.el6_6.3                     rhel-x86_64-server-6                    122 k

Transaction Summary
============================================================================================================================================================
Install       5 Package(s)

Total download size: 201 k
Installed size: 0  
Downloading Packages:
(1/5): mod_auth_kerb-5.4-13.el6.x86_64.rpm                                                                                           |  30 kB     00:00     
(2/5): mod_authnz_pam-0.9.2-1.el6.x86_64.rpm                                                                                         |  13 kB     00:00     
(3/5): mod_intercept_form_submit-0.9.7-1.el6.x86_64.rpm                                                                              |  17 kB     00:00     
(4/5): mod_lookup_identity-0.9.2-1.el6.x86_64.rpm                                                                                    |  19 kB     00:00     
(5/5): sssd-dbus-1.11.6-30.el6_6.3.x86_64.rpm                                                                                        | 122 kB     00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                        41 kB/s | 201 kB     00:04     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : mod_authnz_pam-0.9.2-1.el6.x86_64                                                                                                        1/5 
  Installing : mod_intercept_form_submit-0.9.7-1.el6.x86_64                                                                                             2/5 
  Installing : mod_auth_kerb-5.4-13.el6.x86_64                                                                                                          3/5 
  Installing : mod_lookup_identity-0.9.2-1.el6.x86_64                                                                                                   4/5 
  Installing : sssd-dbus-1.11.6-30.el6_6.3.x86_64                                                                                                       5/5 
  Verifying  : mod_intercept_form_submit-0.9.7-1.el6.x86_64                                                                                             1/5 
  Verifying  : sssd-dbus-1.11.6-30.el6_6.3.x86_64                                                                                                       2/5 
  Verifying  : mod_lookup_identity-0.9.2-1.el6.x86_64                                                                                                   3/5 
  Verifying  : mod_authnz_pam-0.9.2-1.el6.x86_64                                                                                                        4/5 
  Verifying  : mod_auth_kerb-5.4-13.el6.x86_64                                                                                                          5/5 

Installed:
  mod_auth_kerb.x86_64 0:5.4-13.el6                  mod_authnz_pam.x86_64 0:0.9.2-1.el6            mod_intercept_form_submit.x86_64 0:0.9.7-1.el6          
  mod_lookup_identity.x86_64 0:0.9.2-1.el6           sssd-dbus.x86_64 0:1.11.6-30.el6_6.3          

Complete!
** /etc/sssd/sssd.conf has been backed up to sssd.conf-swsave
Updated sssd configuration.
Turning SELinux boolean [httpd_dbus_sssd] on ...
        ... done.
Turning SELinux boolean [allow_httpd_mod_auth_pam] on ...

        ... done.
Configuring Apache modules.
** /etc/tomcat6/server.xml has been backed up to server.xml-swsave.ipa
Stopping sssd:                                             [  OK  ]
Starting sssd:                                             [  OK  ]
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
Waiting for tomcat to be ready ...
Authentication against [ipa2.example.com] sucessfully enabled.
As admin, at Admin > Users > External Authentication, select
          Default organization to autopopulate new users into.
[root@rhnsat ~]# 

Next, point your browser to https://rhnsat.example.com/rhn/admin/multiorg/ExternalAuthentication.do to finalize the setup.

Configure your browser for Kerberos
If you did not yet configured your browser to use Kerberos authentication, do so. Assuming you are using an IPA invironment, follow the instructions provided on the IPA servers. http://ipa2.example.com/ipa/config/browserconfig.html

Disclaimer
I take no responsibility about damaged Satellites, lost of data etc. in doubt, stick on the official product documentation at http://access.redhat.com

Upgrading RHN Satellite 5.5 to 5.6

Saturday, October 12th, 2013

Redhat released version 5.6 of the Redhat Satellite. Time to have a closer look to it and how to upgrade from version 5.5.

New features

  • Finally PostgreSQL support is mature enough for Enterprise usage. No need of a closed source data base anymore. This also brings a lot of new capabilities such as online backups which before was only available using an external Oracle Database which needs the availability of a DBA.

    PostgreSQL also brings some performance benefits over the embedded Oracle database as delivered with 5.5 and earlier. Disclaimer: I did not made any benchmarks, but it “feels” much faster.

  • If you are using the multi-org feature, you may be happy about enhancements for Inter-Satellite-Sync (ISS). Now you can define access rights for different software channels for different organizations.
  • It is not a new feature, but now it is supported: cobbler buildiso. It is a handy solution if you can not use PXE boot in your environment. cobbler buildiso generates a small boot image which allows you to select the installation of a system from a boot menu.
  • Intergrated System Asset Manager (SAM) which is based on Candlepin and allows you assess your system landscape for subscription compliance.
  • Upgrading from RHN Satellite 5.5
    The first thing that you probably would ask: Is it possible and supported to migrate from the Embedded Oracle Database to PostgreSQL? Is it hassle free and bullet-proof? Yes it is.

    Keep in mind

  • As always: Have a look to the product documentation before doing anything on a production Satellite.
  • Create a new RHN Satellite Certificate at access.redhat.com
  • Download the ISO image for 5.6
  • ensure having a recent database backup
  • ensure having a recent backup of your /etc/rhn directory as well as /var/lib/cobbler
  • Update your existing Satellite 5.5 with the latest available patches
  • Delete unnecessary software channels from the Satellite for faster DB migration
  • Delete old Snapshots to minimize database data to be migrated
  • Make enough storage available to migrate from embedded Oracle to PostgreSQL. It takes roughly about the same amount of storage for the data. The PostgreSQL database stores its data in /var/lib/pgsql.
  • Install the latest available package rhn-upgrade: yum install rhn-upgrade

    Lets do it, Perparation work

    First of all, create a database backup of your embedded Oracle Database:

    [root@rhnsat ~]# rhn-satellite stop
    [root@rhnsat ~]# su - oracle -c "db-control backup /path/to/your/backup/directory"
    [root@rhnsat ~]# su - oracle -c "db-control verify /path/to/your/backup/directory"
    [root@rhnsat ~]# rhn-satellite start
    

    Backup the rest of your Satellite:

    [root@rhnsat ~]# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")
    [root@rhnsat ~]# cp -rp /var/lib/cobbler /var/lib/cobbler-$(date +"%F")
    [root@rhnsat ~]# cp -rp /etc/cobbler /etc/cobbler-$(date +"%F")
    

    Update your RHN Satellite 5.5 with the latest available patches and reboot:

    [root@rhnsat ~]# yum -y update && reboot
    

    Ensure the latest schema updates have been applied. The output should read as follow:

    [root@rhnsat ~]# spacewalk-schema-upgrade 
    
    You are about to perform upgrade of your satellite-schema.
    
    For general instructions on Red Hat Satellite schema upgrade, please consult
    the following article:
    
    https://access.redhat.com/knowledge/articles/273633
    
    Hit Enter to continue or Ctrl+C to interrupt: 
    Schema upgrade: [satellite-schema-5.6.0.10-1.el6sat] -> [satellite-schema-5.6.0.10-1.el6sat]
    Your database schema already matches the schema package version [satellite-schema-5.6.0.10-1.el6sat].
    [root@rhnsat ~]#
    

    It is always a good idea to restart a software and check if all is working as expected *before* doing an upgrade. So you can pinpoint problems better if there are some.

    [root@rhnsat ~]# rhn-satellite restart
    

    Review your list of software channels and delete unused ones. This example will delete the channel rhel-i386-rhev-agent-6-server:

    [root@rhnsat ~]# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
    Deleting package metadata (20):
                      ________________________________________
    Removing:         ######################################## - complete
    [root@rhnsat ~]#  
    

    Delete old system snapshots not used anymore. The following example deletes all snapshots which are older than one month:

    [root@rhnsat ~]# sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000")
    

    Update the rhn-update package to the latest available:

    yum install rhn-upgrade
    

    After installing the the rhn-upgrade package, the SQL scripts needed for the DB migration are installed as well as some documentation you should read. They are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

    Upgrade Procedure

    Mount the downloaded ISO image:

    [root@rhnsat ~]# mount satellite-5.6.0-20130927-rhel-6-x86_64.iso /mnt -o loop && cd /mnt
    [root@rhnsat mnt]# 
    

    If you operate your Satellite behind a proxy, you need to upgrade it in disconnected mode, if not, ignore the –disconneded parameter.

    [root@rhnsat mnt]# ./install.pl --disconnected --upgrade
    * Starting the Spacewalk installer.
    * Performing pre-install checks.
    * Pre-install checks complete.  Beginning installation.
    * RHN Registration.
    ** Registration: Disconnected mode.  Not registering with RHN.
    * Upgrade flag passed.  Stopping necessary services.
    * Purging conflicting packages.
    * Checking for uninstalled prerequisites.
    ** Checking if yum is available ...
    There are some packages from Red Hat Enterprise Linux that are not part
    of the @base group that Satellite will require to be installed on this
    system. The installer will try resolve the dependencies automatically.
    However, you may want to install these prerequisites manually.
    Do you want the installer to resolve dependencies [y/N]? y
    * Installing RHN packages.
    * Now running spacewalk-setup.
    * Setting up Selinux..
    ** Database: Setting up database connection for PostgreSQL backend.
    ** Database: Installing the database:
    ** Database: This is a long process that is logged in:
    ** Database:   /var/log/rhn/install_db.log
    *** Progress: #
    ** Database: Installation complete.
    ** Database: Populating database.
    *** Progress: ###################################
    * Database: Starting Oracle to PostgreSQL database migration.
    ** Database: Starting embedded Oracle database.
    ** Database: Trying to connect to Oracle database: succeded.
    ** Database: Migrating data.
    *** Database: Migration process logged at: /var/log/rhn/rhn_db_migration.log
    ** Database: Data migration successfully completed.
    ** Database: Stoping embedded Oracle database.
    * Setting up users and groups.
    ** GPG: Initializing GPG and importing key.
    * Performing initial configuration.
    * Activating Red Hat Satellite.
    ** Certificate not activated.
    ** Upgrade process requires the certificate to be activated after the schema is upgraded.
    * Enabling Monitoring.
    * Configuring apache SSL virtual host.
    Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? y
    * Configuring tomcat.
    ** /etc/sysconfig//tomcat6 has been backed up to tomcat6-swsave
    ** /etc/tomcat6//tomcat6.conf has been backed up to tomcat6.conf-swsave
    Reversed (or previously applied) patch detected!  Skipping patch.
    1 out of 1 hunk ignored -- saving rejects to file web.xml.rej
    * Configuring jabberd.
    * Creating SSL certificates.
    ** Skipping SSL certificate generation.
    * Deploying configuration files.
    * Update configuration in database.
    * Setting up Cobbler..
    cobblerd does not appear to be running/accessible
    Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? 
    This portion of the Red Hat Satellite upgrade process has successfully completed.
    Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
    for any remaining steps in the process.
    [root@rhnsat mnt]# 
    

    Depending on the size of your database and the speed of your disks, the upgrade procedure can take many hours.

    The next step is having a look at diff /etc/rhn/rhn.conf /etc/rhn-$(date +”%F”)/rhn.conf
    and edit /etc/rhn/rhn.conf accordingly. You will probably see missing things such as proxy, server.satellite.rhn_parent etc. Also change the setting disconnected to 0.

    After checking and correcting the config file you can activate the Satellite:

    [root@rhnsat ~]# rhn-satellite-activate --rhn-cert=/root/rhns-cert56.cert --ignore-version-mismatch
    

    After the activation the System is subscribed to the Softwarechannel “redhat-rhn-satellite-5.6-server-x86_64-6″, now bring the Satellite to latest available patchlevel:

    [root@rhnsat ~]# yum -y update 
    

    Stop and disable Oracle
    Bofore doing any Database related actions its better to stop the old Oracle Database to be sure all is now running on PostgreSQL.

    [root@rhnsat ~]# service oracle stop
    Shutting down Oracle Net Listener ...                      [  OK  ]
    Shutting down Oracle DB instance "rhnsat" ...              [  OK  ]
    [root@rhnsat ~]# chkconfig oracle off
    [root@rhnsat ~]# rhn-satellite restart
    

    Aftermath

    Check if your database schema is up-to-date:

    root@rhnsat ~]# spacewalk-schema-upgrade 
    
    You are about to perform upgrade of your satellite-schema.
    
    For general instructions on Red Hat Satellite schema upgrade, please consult
    the following article:
    
    https://access.redhat.com/knowledge/articles/273633
    
    Hit Enter to continue or Ctrl+C to interrupt: 
    Schema upgrade: [satellite-schema-5.6.0.10-1.el6sat] -> [satellite-schema-5.6.0.10-1.el6sat]
    Your database schema already matches the schema package version [satellite-schema-5.6.0.10-1.el6sat].
    [root@rhnsat ~]# 
    

    Rebuild the search index:

    [root@rhnsat ~]# service rhn-search cleanindex
    Stopping rhn-search...
    Stopped rhn-search.
    Starting rhn-search...
    [root@rhnsat ~]# 
    

    Recreate the software channel meta data:

    [root@rhnsat doc]# /etc/sysconfig/rhn/satellite-upgrade/scripts/regenerate-repodata -a
    Scheduling repodata creation for 'rhel-x86_64-server-supplementary-6'
    Scheduling repodata creation for 'rhel-x86_64-server-6'
    Scheduling repodata creation for 'rhn-tools-rhel-x86_64-server-6'
    [root@rhnsat doc]# 
    

    Check functionality
    Before removing the Oracle Database, run your tests to validate the Satellites functionality. Please proceed as stated in /etc/sysconfig/rhn/satellite-upgrade/doc/verification.txt

    This is an important point, as we are getting rid of the Oracle database later on. To be sure all is working as expected, do a complete functionality test for the important things.

    To be on the safe side, let the Satellite run for a few days with Oracle still installed.

    Getting rid of Oracle

    Please read /etc/sysconfig/rhn/satellite-upgrade/doc/satellite-upgrade-postgresql.txt first!

    [root@rhnsat ~]# yum remove *oracle*
    

    Getting rid of the last Oracle bits:

    [root@rhnsat ~]# rm -rf /rhnsat /opt/apps/oracle /usr/lib/oracle/
    

    Result:
    Having fun with a faster Satellite with an open source database :-)

    Disclaimer
    I take no responsibility about damaged Satellites, lost data etc. in doubt, stick on the official product documentation at http://access.redhat.com

Intercepting proxies and spacewalk-repo-sync

Saturday, September 14th, 2013

More and more companies are using intercepting proxies to scan for malware. Those malware scanners can be problematic due to added latency.

If you using spacewalk-repo-sync to synchronize external yum repositories to your custom software channels and experience the famous message [Errno 256] No more mirrors to try in your log files, then you need to configure spacewalk-repo-sync.

Unfortunately the documentation for that is a bit hidden in the man page. You need to create a directory and create a file.

mkdir /etc/rhn/spacewalk-repo-sync/

Create the configuration item:

echo "[main]" >> /etc/rhn/spacewalk-repo-sync/yum.conf
echo timeout=300 >> /etc/rhn/spacewalk-repo-sync/yum.conf

You need to experiment a bit with the value of the timeout setting, 5min should be good enough for most environments.

/etc/rhn/spacewalk-repo-sync/yum.conf has the same options like yum.conf, have a look for more information in the man page.

Have fun :-)

Automated disk partitioning on virtual machines with Cobbler

Saturday, December 15th, 2012

The default Cobbler Snippets just do simple auto partitioning. For a more sophisticated partition layout you need to know what kind of VM you are going to install. KVMs and RHEVs device name is /dev/vda, Xen uses /dev/xvda and ESX /dev/sda.

Luckily this can be figured out automatically, those different virtualization vendors are using its own MAC prefixes. So we can add two nice small Cobbler snippets to do the job. In this example, I call them hw-detect and partitioning.

hw-detect

#set $mac = $getVar('$mac_address_eth0')
#if $mac
#set $mac_prefix = $mac[0:8]
#if $mac_prefix == "00:1a:4a"
# This is a RHEV virtual machine
#set global $machinetype = 'kvm'

#else if $mac_prefix == "52:54:00"
# This is a KVM/Qemu virtual machine
#set global $machinetype='kvm'

#else if $mac_prefix == "00:16:3e"
# This is a XEN virtual machine
#set global $machinetype='xen'
#
#else if $mac_prefix == "00:50:56"
# This is a ESX virtual machine
#set global $machinetype = 'esx'

#else
# #This is a physical machine
#set global $machinetype = 'physical'
#end if
#end if

partitioning

#if $machinetype == 'kvm'
#set $disk='vda'
#else if $machinetype == 'xen'
#set $disk = 'xvda'
#else
#set $disk = 'sda'
#end if
# Lets install the system on /dev/$disk
part /boot      --fstype ext2 --size=250 --ondisk=$disk
part pv.0       --size=1 --grow --ondisk=$disk

volgroup vg_${name} pv.0

logvol /        --fstype ext4 --name=lv_root    --vgname=vg_${name} --size=4096
logvol /home    --fstype ext4 --name=lv_home    --vgname=vg_${name} --size=512 --fsoption=nosuid,nodev,noexec
logvol /tmp     --fstype ext4 --name=lv_tmp    --vgname=vg_${name} --size=1024 --fsoption=nosuid,nodev,noexec
logvol /var     --fstype ext4 --name=lv_var    --vgname=vg_${name} --size=2048 --fsoption=nosuid,nodev,noexec
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_${name} --size=2048

An additional “feature” of the partitioning Snippet is: It sets up the Volume Group name according to your systems name. This is the unofficial standard since quite some time. It also sets some more secure mount options. Review them carefully if they make sense for you and edit them as needed.

The next step is to configure your kickstart template.

Standalone Cobbler
On a standalone Cobbler server edit /var/lib/cobbler/kickstart/your-kick-start-template.ks

# Detect the used hardware type
$SNIPPET('hw-detect')
# Set up default partitioning
$SNIPPET('partitioning')

Bundled Cobbler
When using cobbler bundled with Spacewalk or Red Hat Satellite, you need to edit the Kickstart profile in the WebUI.


Navigate to Systems -> Kickstart -> Profile. Select the Kickstart profile to be modified -> System Details -> Partitioning.

Copy the two Snippets in /var/lib/cobbler/spacewalk/1, where 1 is representing your OrgId.

Alternatively you can edit them in the WebUI as well.

To check if all is working as expected, add a system to Cobbler using the Command Line Interface and have a look to the rendered Kickstart file. This can be easily done with cobbler system getks --name=blah.

Happy System installing….

Have fun :-)

Upgrading RHN Satellite 5.4.1 to 5.5

Sunday, September 23rd, 2012

Red Hat has released RHN Satellite version 5.5. It is a release that is mainly a bug-fix release, but has some interesting new features as well. Here comes a brief guide how to update your RHN Satellite to the latest version. It is not a official guide, so if you trash your Satellite, it is not my fault…

Preparation
As always, before you upgrade the RHN Satellite, you need to order a new certificate. Open a Support case at Red Hat and tell them you need a new certificate for Version .5.5.

You also need to download the ISO file for the upgrade as the packages are only available in the software channel after the upgrade and activation. You can download the ISO at Red Hats download site. Of course you need to choose the architecture that matches your environment. Note that there is only one ISO available for each architecture, not two as it was before. The ISO comes with the embedded database. If you need to use an external database, use the --external-db parameter with install.pl

Ensure you have a working backup of your database before starting with the upgrade. Do this as follows:

su - oracle
db-control backup /your/back/up/directory
db-control verify /your/back/up/directory

A backup of your /etc/rhn directory is also a good idea, just for the case something is going wrong: cp -rp /etc/rhn /etc/rhn-$(date +"%F")

Ensure your database has enough free table space left. For the DATA_TBS and the UNDO_TBS it should be at least 1Gbyte, better are 2Gbyte. The following example shows an example:

[root@rhns ~]# su - oracle
-bash-4.1$ db-control report
Tablespace                  Size    Used   Avail   Use%
DATA_TBS                   16.1G   12.6G    3.5G    78%
SYSAUX                      500M  182.6M  317.3M    37%
SYSTEM                      400M  254.1M  145.8M    64%
TEMP_TBS                   1000M      0B   1000M     0%
UNDO_TBS                    3.9G  474.7M    3.4G    12%
USERS                       128M     64K  127.9M     0%
-bash-4.1$ 

You can grow the table spaces if needed by fire db-control extend UNDO_TBS.

It is also very important to have enoght free space in the /rhnsat filesystem, db-control gather-stats needs some extra space. At least 2 Gbyte to be on the safe side.

Having a look to the official upgrade guide is strongly recommended.

First you need to loop-back mount the ISO image and cd into the mountpoint:

[root@rhns ~]# mount satellite-5.5.0-20120911-rhel-6-x86_64.iso /mnt -o loop
[root@rhns ~]# cd /mnt
[root@rhns mnt]# 

Next step is to install the rhn-upgrade package.

[root@rhns mnt]# yum -y install rhn-upgrade
Loaded plugins: product-id, rhnplugin, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package rhn-upgrade.noarch 0:5.5.0.16-1.el6sat will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================
 Package         Arch       Version               Repository                                    Size
=====================================================================================================
Installing:
 rhn-upgrade     noarch     5.5.0.16-1.el6sat     redhat-rhn-satellite-5.4-server-x86_64-6      38 k

Transaction Summary
=====================================================================================================
Install       1 Package(s)

Total download size: 38 k
Installed size: 0  
Downloading Packages:
rhn-upgrade-5.5.0.16-1.el6sat.noarch.rpm                                      |  38 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : rhn-upgrade-5.5.0.16-1.el6sat.noarch                                              1/1 
Installed products updated.
  Verifying  : rhn-upgrade-5.5.0.16-1.el6sat.noarch                                              1/1 

Installed:
  rhn-upgrade.noarch 0:5.5.0.16-1.el6sat                                                             

Complete!
[root@rhns mnt]# 

The package contains documents and scripts to help you with the upgrade. They are located in the directory /etc/sysconfig/rhn/satellite-upgrade. Read those documents carefully before proceeding with the upgrade.

Upgrading
Lets do it… run the installer script with the --upgrade parameter, bold red letters are interactive input.

[root@rhns mnt]# ./install.pl --upgrade
* Starting the Red Hat Network Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHN Registration.
** Registration: System is already registered with RHN.  Not re-registering.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Applying updates.
* Installing RHN packages.
Warning: yum did not install the following packages:
	geronimo-specs-compat
* Now running spacewalk-setup.
* Setting up Oracle environment.
* Setting up database.
** Database: Upgrading the database server to latest Oracle 10g:
** Database: This is a long process that is logged in:
** Database: /var/log/rhn/upgrade_db.log
*** Progress: ##############################################################
** Database: Setting up database connection for Oracle backend.
** Database: Testing database connection.
** Database: Populating database.
** Database: Skipping database population.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating RHN Satellite.
** Certificate not activated.
** Upgrade process requires the certificate to be activated after the schema is upgraded.
* Enabling Monitoring.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? y
* Configuring tomcat.
** /etc/tomcat6/tomcat6.conf has been backed up to tomcat6.conf-swsave
** /etc/tomcat6/server.xml has been backed up to server.xml-swsave
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file web.xml.rej
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
cobblerd does not appear to be running/accessible
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? y
cobblerd does not appear to be running/accessible
This portion of the RHN Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.
[root@rhns mnt]# 

Now some database actions are needed. Make sure your Satellite is stopped and only the database is running:

rhn-satellite stop
service oracle start

You need to create schema statistics:

su - oracle
-bash-4.1$ db-control gather-stats
Gathering statistics...
WARNING: this may be a very slow process.
done.
-bash-4.1$ 

Now it is time to upgrade the database schema

[root@rhns mnt]# spacewalk-schema-upgrade
Schema upgrade: [satellite-schema-5.4.0.19-1.el6sat] -> [satellite-schema-5.5.0.13-1.el6sat]
Searching for upgrade path: [satellite-schema-5.4.0.19-1] -> [satellite-schema-5.5.0.13-1]
Searching for upgrade path: [satellite-schema-5.4.0.19] -> [satellite-schema-5.5.0.13]
Searching for upgrade path: [satellite-schema-5.4.0] -> [satellite-schema-5.5.0]
Searching for upgrade path: [satellite-schema-5.4] -> [satellite-schema-5.5]
The path: [satellite-schema-5.4] -> [satellite-schema-5.5]
Planning to run spacewalk-sql with [/var/log/spacewalk/schema-upgrade/20120922-132500-script.sql]
Hit Enter to continue or Ctrl+C to interrupt: Enter
Executing spacewalk-sql, the log is in [/var/log/spacewalk/schema-upgrade/20120922-132500-to-satellite-schema-5.5.log].
The database schema was upgraded to version [satellite-schema-5.5.0.13-1.el6sat].
[root@rhns mnt]# 

Now it is time to activate your RHN Satellite to be able to receive updates for the Satellite and running satellite-sync

[root@rhns ~]# rhn-satellite-activate --ignore-version-mismatch --rhn-cert=/root/rhns-cert55.cert 
RHN_PARENT: satellite.rhn.redhat.com
[root@rhns ~]# 

To rebuild the search index please run service rhn-search cleanindex

[root@rhns ~]# service rhn-search cleanindex
Stopping rhn-search...
rhn-search was not running.
Starting rhn-search...
[root@rhns ~]# 

Before restarting the RHN Satellite, check if any updates are available for it.
yum -y update

Afterward, please check if there is another database schema update available. If the output looks as following, you are safe.

[root@rhns ~]# spacewalk-schema-upgrade
Schema upgrade: [satellite-schema-5.5.0.13-1.el6sat] -> [satellite-schema-5.5.0.13-1.el6sat]
Your database schema already matches the schema package version [satellite-schema-5.5.0.13-1.el6sat].
[root@rhns ~]# 

As a verification that the upgrade is fine, run a satellite-sync to sync some new content and update a registered server. If you have more than one Satellite, run a ISS (Inter Satellite Sync) for proofing its functionality.

Troubleshooting
If something goes wrong with the database update, before reverting to a backup, first check the Oracle alert file /rhnsat/admin/rhnsat/bdump/alert_rhnsat.log to figure out what went wrong. Another good place to have a look at are the trace files located in /rhnsat/admin/rhnsat/udump