Upgrading Redhat Satellite 5.7 to 5.8

Couple of days ago, Redhat released its latest and last major upgrade for Satellite 5.x. Its a rather important upgrade, you are advised to upgrade soon.

This upgrade contains some major improvements like stated in an earlier article

Disclaimer

I’m not responsible for any damage caused by the procedure provided here. Always create a backup before even thinking about upgrading a Satellite server.

Preparation

As always when you plan to upgrade your Satellite server to the latest version, you need to do some preparations first.

Ensure you will have enough disk space free in /var/opt/rh. The upgrade to 5.8 will also install a new PostgreSQL version located at /var/opt/rh/rh-postgresql95/lib/pgsql. The new version will roughly use the same diskspace as the old version 9.2 in /opt/rh/postgresql92/root/var/lib/pgsql.

Download the ISO

Visit https://access.redhat.com/downloads/content/250/ver=5.8/rhel—6/5.8/x86_64/product-software and make sure you select 5.8 and the architecture fitting you system (x86_64 or S390)

Get a new Satellite Certificate Manifest

Satellite 5.8 switches from Certificates to Manifests like Satellite 6. You need a Manifest to get it activated. You can create it by your own at the Subscription Management Application site, ensure you attach enough subscriptions to your Satellite server(s).

Backup

Usually an upgrade runs smooth, but just in case… it is recommended practice to have a recent backup ready. If your Satellite is running on a virtual machine, power off, snapshot and power on to have a consistent backup ready. For physical systems, db-control and the choice.

Backup the rest of your Satellite:

Create a copy of your rhn configuration directory as we need some information from the old files after the upgrade.

rhnsat:~# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")

Update your OS and Satellite 5.7

First step is to update the operating system and the Satellite 5.7 and apply the latest database schema updates as well.

rhnsat:~# yum -y update && reboot

Update the database schema if needed

To update the database schema, run the following command. Ideally it looks as follows:

rhnsat:~# spacewalk-schema-upgrade 
Schema upgrade: [satellite-schema-5.7.0.27-1.el6sat] -> [satellite-schema-5.7.0.27-1.el6sat]
Your database schema already matches the schema package version [satellite-schema-5.7.0.27-1.el6sat].
rhnsat:~# 

Switch from RHN to Subscription Manager

It is important to ensure you switched from RHN to subscription manager before doing the upgrade. You can check if this is the case with:

rhnsat:~# subscription-manager repos --list-enabled
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   rhel-6-server-rpms
Repo Name: Red Hat Enterprise Linux 6 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/os
Enabled:   1

Repo ID:   rhel-6-server-satellite-5.7-rpms
Repo Name: Red Hat Satellite 5.7 (for RHEL 6 Server) (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/satellite/5.7/os
Enabled:   1

rhnsat:~# 

If this is not yet done, have a look at the knowledge base article located here: https://access.redhat.com/articles/2884191

Functionality Check with the old version 5.7

It is recommended to restart and check a software functionality before upgrading to be able to pinpoint problems if there are some.

rhnsat:~# rhn-satellite restart

Cleanup

Review the software channels in use and delete unused channels as this can free up quite some disk space and reduces the size of the database significantly.

rhnsat:~# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
Deleting package metadata (20):
                  ________________________________________
Removing:         ######################################## - complete
rhnsat:~# 

Delete old system snapshots which are not used anymore. The following example deletes all snapshots which are older than one month:

rhnsat:~#  sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000") 

Check for old MD5 user passwords and certificates

Check if there are still some users with an md5 hashed password. The same applies to certificates.

rhnsat:~# spacewalk-report users-md5
rhnsat:~# spacewalk-report system-md5-certificates

If there are any, please have a look to https://access.redhat.com/documentation/en-us/red_hat_satellite/5.8/html/installation_guide/ch10s03

RTFM

If not done yet, install or update the rhn-upgrade package which contains the instructions how to proceed.

rhnsat:~# yum -y install rhn-upgrade

The package contains not only SQL- and other useful scripts needed for the upgrade but also important documents to read. The are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

For most users, the document satellite-upgrade-postgresql.txt applies.

Do not forget to read the updated product documentation as well:

Performing the upgrade

rhnsat:~# mount satellite-5.8-rhel-6-x86_64-dvd.iso /mnt  -o loop
rhnsat:~#  cd /mnt
rhnsat:/mnt# ./install.pl --upgrade
* Starting Red Hat Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHSM Registration.
** Registration: System is already registered with RHSM.  Not re-registering.
* RHSM Subscriptions.
** Subscriptions: Subscription providing 'Red Hat Satellite' already attached.
** Subscriptions: Subscription providing 'Red Hat Enterprise Linux Server' already attached.
** Subscriptions: Disabling all RHSM repositories (rhel-6-server-rpms, rhel-6-server-satellite-5.7-rpms).
** Subscriptions: All repositories disabled.
** Subscriptions: Enabling RHEL repository.
** Subscriptions: RHEL repository enabled.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Installing Satellite packages.
Warning: more packages were installed by yum than expected:
        python-backports
        python-backports-ssl_match_hostname
        python-chardet
        python-requests
        python-urllib3
* Now running spacewalk-setup.
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
*** Upgrading embedded database.
** Database: Populating database.
** Database: Skipping database population.
* Configuring tomcat.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating Red Hat Satellite.
** Manifest not activated.
** Upgrade process requires the manifest to be activated after the schema is upgraded.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? 
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? 
This portion of the Red Hat Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.
rhnsat:/mnt# 

Active and Updating the Satellite

Since the ISO image is always a bit outdated, you need to activate and update the Satellite after its installation.

Upgrading the Database schema

rhnsat:~# rhn-satellite stop
rhnsat:~# /etc/init.d/rh-postgresql95-postgresql start
rhnsat:~# spacewalk-schema-upgrade
rhnsat:~# rhn-satellite-activate --manifest=/root/manifest.zip --ignore-version-mismatch

Some more work to do

After the upgrade succeed there is some work work to do.

Initial Sync with CDN

Unfortunately cdn-sync does not inherit the history which channels have been synced previously with satellite-sync. You need to once sync each channel again. Only missing data will be downloaded.

rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listbasechannels); do cdn-sync -c $i; done
rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listchildchannels); do cdn-sync -c $i; done

If you have custom channels, this will produce errors as the custom channels are not available in CDN. Just ignore them.

Rebuild the search index

rhnsat:~# service rhn-search cleanindex

Have fun 🙂

Using OTP Tokens and 2FA with FreeIPA 4.0

On 2014-07-08 FreeIPA 4.0 was released. One of the most interesting new features is the support of two factor authentication (2FA). I was curious about how to set it up and get it running. Unfortunately the documentation does not tell much about the OTP setup.

What is OTP and 2FA? An overview
OTP stands for One Time Password and 2FA for two factor authentication. OTP is available since long time, in the beginning usually as a list of passwords printed on paper. It was enhancing security gradually but was an operational nightmare.

RSA then came up with harware tokens somewhere in the 1990this which made it much more usable. Also 2FA was introduced. the two factors are ownership (or possession) and knowledge. One needs to obtain a piece of hardware (Hardware Token or a smart phone with a software token) and knowledge (knowing the password).

Meanwhile a lot of competing tokens are on the market, as well as so called soft-tokens. Most (or all?) of the hardware tokens are proprietary, making system configuration a nightmare (RSA PAM modules and stuff). On the other hand, every proprietary solution comes with the support of Radius. There is a quite new definition of using a Radius proxy to use those tokens with Kerberos and connect them with IPA.

However, hardware tokens and Radius proxies have been out of scope for my initial test. Lets go for the simpler soft token way.

Installing FreeIPA 4.0
It is planed to include FreeIPA 4.0 in Fedora 21 which will be released later this year. For testing you can either use Fedora Rawhide 21 or Fedora 20 with an external Yum repository. I was choosing the later way.

wget https://copr.fedoraproject.org/coprs/pviktori/freeipa/repo/fedora-20-i386/pviktori-freeipa-fedora-20-i386.repo -O /etc/yum.repos.d/pviktori-freeipa-fedora-20-i386.repo

The rest of the installation is the same as with (Free)IPA2 and (Free)IPA3. Please have a look at my earlier Post

Enabling OTP
You can either enable OTP on a global scope or per user. At the moment I recommend it on a per-user base.

ipa user-mod username --user-auth-type=otp

If you want to enable users to authenticate with more than one method, user –user-auth-type={otp,password}

Adding a new user with OTP enabled will probably be possible in the future. There seems to be a bug, according to ipa user-add –help it is supposed to be working.

ipa user-add hwurst --first="Hans" --last="Wurst" --user-auth-type=otp

Adding a token
The best way for a user to add a token is probably the web interface. Lets call it self-service. The user first authenticates with username and the initial password set by the admin to set a new one. The OTP field can be ignored for the moment.

After authentication, the user can navigate to “OTP Tokens” on the top navigation bar and add a new token. This looks as following:

ipa-otpThe ID needs to be unique, this can case problems when users are adding the tokens by themself as people would tend to provide a simple ID by themself. When not providing an ID, one will be generated. The field Unique ID should IMHO not be available for ordinary users.

After adding the token, login via password only is not possible anymore (unless explicitly enabled with the user-auth-type).

After hitting “Add”, a QR code will be shown. This allows users to scan the code with the Smartphone app, such as FreeOTP and Google Authenticator.

The next step users needs to do is to sync the token. This can be done by returning to the login screen and clicking on “Sync OTP Token” right left to the Login button.

ipa-otp2With a generated Unique ID (=Token ID) its quite annoying to enter that ID. However, usually this only needs to be one once 🙂

 

 

 

 

Limitations

The release notes mentions that there are concerns about the scalability when using HOTP, where TOTP has a known issue that tokens can be reused, but only within a short timeframe.

I see another issue which is a kind of a chicken-and-egg problem: After adding a user, this user is able to login with its password only until a token has been added. This ability is needed to log in to the IPA WebUI to add the token at the first place. However, password-only access should be limited to the token add facility.

Conclusion

I’m pretty amazed how well it works as this is a brand new feature for FreeIPA. The involved engineers made a brilliant job! I’m looking forward to see this feature in Redhat IPA/IdM somewhere in the future as 2FA is an often requested killer feature in enterprise environments.

Read more

Have fun! 🙂

Intercepting proxies and spacewalk-repo-sync

More and more companies are using intercepting proxies to scan for malware. Those malware scanners can be problematic due to added latency.

If you using spacewalk-repo-sync to synchronize external yum repositories to your custom software channels and experience the famous message [Errno 256] No more mirrors to try in your log files, then you need to configure spacewalk-repo-sync.

Unfortunately the documentation for that is a bit hidden in the man page. You need to create a directory and create a file.

mkdir /etc/rhn/spacewalk-repo-sync/

Create the configuration item:

echo "[main]" >> /etc/rhn/spacewalk-repo-sync/yum.conf
echo timeout=300 >> /etc/rhn/spacewalk-repo-sync/yum.conf

You need to experiment a bit with the value of the timeout setting, 5min should be good enough for most environments.

/etc/rhn/spacewalk-repo-sync/yum.conf has the same options like yum.conf, have a look for more information in the man page.

Have fun 🙂

PAM and IPA authentication for RHN Satellite

If you have a larger installation on your site, you may wish to have a single source of credentials not only for common system services, but for your RHN Satellite too.

This will show you how to configure your RHN Satellite Server to use PAM with SSSD. SSSD, the System Security Services Daemon is a common framework to provide authentication services. Needless to say that IPA is supported as well.

Assumptions:

  • You have a RHN Satellite running on RHEL6
  • You have an IPA infrastructure running (at least on RHEL 6.2)

Preparations
First you need to install the ipa-client on your satellite:

yum -y install ipa-client

And then join the server to your IPA environment:

ipa-client-install -p admin

Configuring PAM as follows:

cat << EOF > /etc/pam.d/rhn-satellite
auth        required      pam_env.so
auth        sufficient    pam_sss.so 
auth        required      pam_deny.so
account     sufficient    pam_sss.so
account     required      pam_deny.so
EOF

Configure the RHN Satellite
Your Satellite now needs to be aware that there is the possibility to authenticate users with PAM against IPA.

echo "pam_auth_service = rhn-satellite" >> /etc/rhn/rhn.conf

If you have users in your IPA domain with usernames shorter than five characters, you will need to add one more line to be able to create the users in RHN Satellite:

echo "web.min_user_len = 3" >>   /etc/rhn/rhn.conf

After this change, restart your RHN Satellite

rhn-satellite restart

Configuring users
Now you can log in to your RHN Satellite with your already configured admin user and select the checkbox “Pluggable Authentication Modules (PAM)” on existing users and/or new users.

Things to be considered
It is strongly recomended to have at leat one user per organization (ususally a “Organization Administrator”) plus the “RHN Satellite Administrator” not having PAM authentication enabled. Despite of the easy implementation of redundancy with IPA, this is important for a fallback scenario when your IPA environment has some service interruptions due to mainenance or failure.

SSSD caches users credentials on the RHN Satellite system, but this is only true for users logged in at least once. The default value for offline_credentials_expiration is 0, which means no cache time limit. However, depending on your organizations scurity policy this value can vary. Please check the PAM section in /etc/sssd/sssd.conf

Further documents to read