Renew Letsencrypt certificates for Red Hat Satellite 6 and Capsule

Letsencrypt certificates are only valid for just three months. The procedure to renew x509 certificates in Red Hat Satellite 6 is not so straight forward and its even more complex for Capsule servers.

In an earlier post I was writing about how to set up a Satellite 6 and a capsule using Letsencrypt certificates. This post is a follow up on that.

Be aware: You must follow this procedure before the certificate expires or the Satellite simply stops working, all clients will refuse to communicate with the Red Hat Satellite Server. This is a security feature, not a bug.

Obtain the Satellite certificate

This is straight forward, stop the httpd and just use certbot, you must use the -d parameter because for the capsule it will fail.

systemctl stop httpd && certbot renew -d sat6.example.com

Install the renewed certificate into Satellite

You need to run the Satellite installer to make your certificate active:

satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" --certs-server-key "/etc/letsencrypt/live/sat6.example.com/privkey.pem" --certs-server-ca-cert "/root/ca-cert.pem" --certs-update-server --certs-update-server-ca

Keep the output as you need the oauth key and secret for the capsule.

Obtain the new certifcate for the capsule

This step must be done on the Satellite, not on the Caspule.

The only way to obtain a cert for a server different than than the target is to make use of the DNS challenge.

certbot -d capsule.example.com --manual --preferred-challenges dns certonly

It will ask you to create a DNS TXT entry as a challenge to ensure you are in control of the domain. When your DNS entry is ready, hit enter.

Create the tarball with the certifcates for the Capsule

capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar  "~/$CAPSULE-certs.tar" --server-cert "/etc/letsencrypt/live/capsule.example.com/cert.pem" --server-key "/etc/letsencrypt/live/capsule.example.com/privkey.pem" --server-ca-cert "/root/capsule.example.com/bundle-ca-cert.pem" --certs-update-server

The next step is to copy the tarball to your capsule:

scp /root/capsule.example.com-certs.tar capsule.example.com:

Install the new certificate on the Capsule

This step must be done on the Capsule server

satellite-installer --scenario capsule\
                      --foreman-proxy-content-parent-fqdn           "sat6.example.com"\
                      --foreman-proxy-register-in-foreman           "true"\
                      --foreman-proxy-foreman-base-url              "https://sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "capsule.exmple.com"\
                      --foreman-proxy-oauth-consumer-key            "The Key"\
                      --foreman-proxy-oauth-consumer-secret         "The Secret"\
                      --foreman-proxy-content-certs-tar             "/root/capsule.example.com-certs.tar"\
                      --puppet-server-foreman-url                   "https://sat6.example.com"

Feedback welcome…

Have fun 🙂

Installing Red Hat Satellite 6 with Letsencrypt certificates

Red Hat Satellite 6 is a nice tool for system life cycle management. It can get complex and even installation is sometimes tricky. This article is about how to install Satellite, it does not explain the principals and concepts behind it.

Requirements

A valid subscription for the Satellite (and optional for the capsule).

The system requirements are listed here.

There is one important thing the install guide is missing: Satellite 6.4 will not work in IPv6 only environments. There must be an IPv4 address configured, even if it is just an RFC1918 private address. You need to add this IP in /etc/hosts. The reason is that several daemons are listening on IPv4 addresses only. One of them is important: Apache QPID which is used for i.e. errata application via katello-agent. But there are two [update] undocumented installer parameters [/update].

Proxy is mandatory in IPv6 only environments

If your Satellite is only able to connect to the internet via IPv6, you need a IPv4 capable proxy to talk to subscription.rhsm.redhat.com which is not reachable by IPv6. That is the host the subscription manager is talking to.

Install EPEL and certbot

It makes sense to use officially valid certs since they are available for free usage from Letsencrypt. Certbot is available from EPEL and a handy way to request certificates using the ACME protocol.

[root@sat6 ~]# yum -y install http://ftp.tu-chemnitz.de/pub/linux/epel/7Server/x86_64/Packages/e/epel-release-7-11.noarch.rpm

It is important to disable EPEL by default to not get conflicts with RPMs from other repositories. Just enable EPEL when needed and double check.

[root@sat6 ~]# yum-config-manager --disable epel

Install the certbot package

[root@sat6 ~]# yum -y install certbot --enablerepo=epel

Issue the cert

[root@sat6 ~]# certbot certonly -n --standalone --agree-tos --domains sat6.example.com -m user@example.com

Download the CA-Certs

Root-CA

[root@sat6 ~]# wget https://www.identrust.com/node/935 -O trustidrootx3_chain.p7b

Convert the p7b to PEM format.

[root@sat6 ~]# openssl pkcs7 -in trustidrootx3_chain.p7b -inform DER -print_certs -out trustidrootx3_chain.pem

Intermediate CA-Cert

[root@sat6 ~]# wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

Create the CA bundle file

[root@sat6 ~]# cp trustidrootx3_chain.pem bundle-ca-cert.pem
[root@sat6 ~]# cat lets-encrypt-x3-cross-signed.pem.txt >> bundle-ca-cert.pem

Check the certificate

You can check if you made all things as expected, run the check command.

[root@sat6 ~]# katello-certs-check -c "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" -k "/etc/letsencrypt/live/sat6.example.com/privkey.pem" -b "/root/bundle-ca-cert.pem"

Running the installer

Note: The parameters –foreman-proxy-content-qpid-router-hub-addr :: –foreman-proxy-content-qpid-router-agent-addr :: are not documented and only needed if you want to be able that the capsule and/or clients will be able to communicate over IPv6.

[root@sat6 ~]# satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/archive/sat6.example.com/fullchain2.pem" --certs-server-key "/etc/letsencrypt/archive/sat6.example.com/privkey2.pem" --certs-server-ca-cert "/root/bundle-ca-cert.pem" --certs-update-server --certs-update-server-ca --katello-proxy-url=http://proxy.example.com --katello-proxy-port=3128 --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

Installing a Capsule Server

If you want to use a capsule server within an environment with Letsencrypt certificates, its a bit more complex, but however, it works.

Install the software

Needless to say that your capsule needs to have the correct repos enabled. For details, please see here.

[root@capsule ~]# yum install satellite-capsule

Request the Certificate

On the Satellite, request a certificate for the capsule. Note: This only works with the DNS challenge, so you need access to your DNS server.

[root@sat6 ~]# certbot -d capsule.example.com --manual --preferred-challenges dns certonly

Prepare the certificates and key

Create a directory and copy the certificates

[root@sat6 ~]# mkdir /root/capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/privkey.pem capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/cert.pem capsule.example.com
[root@sat6 ~]# cp /root/bundle-ca-cert.pem capsule.example.com

Validate the certificate

[root@sat6 ~]# katello-certs-check -c /root/capsule.example.com/cert.pem -b /root/capsule.example.com/bundle-ca-cert.pem -k /root/capsule.example.com/privkey.pem

If all is fine, run the capsule generator:

[root@sat6 ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar /root/capsule.example.com-certs.tar --server-cert /root/capsule.example.com/cert.pem --server-key /root/capsule.example.com/privkey.pem --server-ca-cert /root/capsule.example.com/bundle-ca-cert.pem

Copy the resulting tarball to the capsule server:

[root@sat6 ~]# scp capsule.example.com-certs.tar capsule.example.com

Running the installer

[root@capsule ~]#  satellite-installer --scenario capsule\
--foreman-proxy-content-parent-fqdn      "sat6.example.com"\
--foreman-proxy-register-in-foreman      "true"\
--foreman-proxy-foreman-base-url         "https://sat6.example.com"\
--foreman-proxy-trusted-hosts            "sat6.example.com"\
--foreman-proxy-trusted-hosts            "capsule.example.com"\
--foreman-proxy-oauth-consumer-key       "the key"\
--foreman-proxy-oauth-consumer-secret    "the secret"\
--foreman-proxy-content-certs-tar        "/root/capsule.example.com-certs.tar"\
--puppet-server-foreman-url              "https://sat6.example.com" \
--foreman-proxy-content-qpid-router-hub-addr :: \
--foreman-proxy-content-qpid-router-agent-addr ::

That’s it 🙂

Do not ask me how certificate renewal works, I’ll let you know in three months 😉

Workaround for IPv6-only Networks

Unfortunately the satellite-installer configures Apache QPID not correctly, it will be set up to use IPv4 only by default. That means, IPv6-only hosts (i.e. including the capsule) are unable to communicate with the Satellite.

There is a workaround: Add two additional listeners on the Satellite and one on the Capsule. Be aware: The Satellite installer overwrites your changes every time when you run it, i.e. for upgrades or adding new features. create a backup of the config file.

There are two undocumented parameters for the satellite installer: –foreman-proxy-content-qpid-router-hub-addr :: and –foreman-proxy-content-qpid-router-agent-addr ::. You can add them during first time run as well as after an initial installation.

It is the same procedure in the Satellite as well as on the Capsule.

satellite-installer --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

This behavior is already fixes upstream as you can see it here: https://github.com/theforeman/puppet-foreman_proxy_content/commit/89b4ea988d18f100b806e7cddc2dca623b68f084″.

Upgrading Redhat Satellite 5.7 to 5.8

Couple of days ago, Redhat released its latest and last major upgrade for Satellite 5.x. Its a rather important upgrade, you are advised to upgrade soon.

This upgrade contains some major improvements like stated in an earlier article

Disclaimer

I’m not responsible for any damage caused by the procedure provided here. Always create a backup before even thinking about upgrading a Satellite server.

Preparation

As always when you plan to upgrade your Satellite server to the latest version, you need to do some preparations first.

Ensure you will have enough disk space free in /var/opt/rh. The upgrade to 5.8 will also install a new PostgreSQL version located at /var/opt/rh/rh-postgresql95/lib/pgsql. The new version will roughly use the same diskspace as the old version 9.2 in /opt/rh/postgresql92/root/var/lib/pgsql.

Download the ISO

Visit https://access.redhat.com/downloads/content/250/ver=5.8/rhel—6/5.8/x86_64/product-software and make sure you select 5.8 and the architecture fitting you system (x86_64 or S390)

Get a new Satellite Certificate Manifest

Satellite 5.8 switches from Certificates to Manifests like Satellite 6. You need a Manifest to get it activated. You can create it by your own at the Subscription Management Application site, ensure you attach enough subscriptions to your Satellite server(s).

Backup

Usually an upgrade runs smooth, but just in case… it is recommended practice to have a recent backup ready. If your Satellite is running on a virtual machine, power off, snapshot and power on to have a consistent backup ready. For physical systems, db-control and the choice.

Backup the rest of your Satellite:

Create a copy of your rhn configuration directory as we need some information from the old files after the upgrade.

rhnsat:~# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")

Update your OS and Satellite 5.7

First step is to update the operating system and the Satellite 5.7 and apply the latest database schema updates as well.

rhnsat:~# yum -y update && reboot

Update the database schema if needed

To update the database schema, run the following command. Ideally it looks as follows:

rhnsat:~# spacewalk-schema-upgrade 
Schema upgrade: [satellite-schema-5.7.0.27-1.el6sat] -> [satellite-schema-5.7.0.27-1.el6sat]
Your database schema already matches the schema package version [satellite-schema-5.7.0.27-1.el6sat].
rhnsat:~# 

Switch from RHN to Subscription Manager

It is important to ensure you switched from RHN to subscription manager before doing the upgrade. You can check if this is the case with:

rhnsat:~# subscription-manager repos --list-enabled
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   rhel-6-server-rpms
Repo Name: Red Hat Enterprise Linux 6 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/os
Enabled:   1

Repo ID:   rhel-6-server-satellite-5.7-rpms
Repo Name: Red Hat Satellite 5.7 (for RHEL 6 Server) (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/satellite/5.7/os
Enabled:   1

rhnsat:~# 

If this is not yet done, have a look at the knowledge base article located here: https://access.redhat.com/articles/2884191

Functionality Check with the old version 5.7

It is recommended to restart and check a software functionality before upgrading to be able to pinpoint problems if there are some.

rhnsat:~# rhn-satellite restart

Cleanup

Review the software channels in use and delete unused channels as this can free up quite some disk space and reduces the size of the database significantly.

rhnsat:~# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
Deleting package metadata (20):
                  ________________________________________
Removing:         ######################################## - complete
rhnsat:~# 

Delete old system snapshots which are not used anymore. The following example deletes all snapshots which are older than one month:

rhnsat:~#  sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000") 

Check for old MD5 user passwords and certificates

Check if there are still some users with an md5 hashed password. The same applies to certificates.

rhnsat:~# spacewalk-report users-md5
rhnsat:~# spacewalk-report system-md5-certificates

If there are any, please have a look to https://access.redhat.com/documentation/en-us/red_hat_satellite/5.8/html/installation_guide/ch10s03

RTFM

If not done yet, install or update the rhn-upgrade package which contains the instructions how to proceed.

rhnsat:~# yum -y install rhn-upgrade

The package contains not only SQL- and other useful scripts needed for the upgrade but also important documents to read. The are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

For most users, the document satellite-upgrade-postgresql.txt applies.

Do not forget to read the updated product documentation as well:

Performing the upgrade

rhnsat:~# mount satellite-5.8-rhel-6-x86_64-dvd.iso /mnt  -o loop
rhnsat:~#  cd /mnt
rhnsat:/mnt# ./install.pl --upgrade
* Starting Red Hat Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHSM Registration.
** Registration: System is already registered with RHSM.  Not re-registering.
* RHSM Subscriptions.
** Subscriptions: Subscription providing 'Red Hat Satellite' already attached.
** Subscriptions: Subscription providing 'Red Hat Enterprise Linux Server' already attached.
** Subscriptions: Disabling all RHSM repositories (rhel-6-server-rpms, rhel-6-server-satellite-5.7-rpms).
** Subscriptions: All repositories disabled.
** Subscriptions: Enabling RHEL repository.
** Subscriptions: RHEL repository enabled.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Installing Satellite packages.
Warning: more packages were installed by yum than expected:
        python-backports
        python-backports-ssl_match_hostname
        python-chardet
        python-requests
        python-urllib3
* Now running spacewalk-setup.
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
*** Upgrading embedded database.
** Database: Populating database.
** Database: Skipping database population.
* Configuring tomcat.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating Red Hat Satellite.
** Manifest not activated.
** Upgrade process requires the manifest to be activated after the schema is upgraded.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? 
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? 
This portion of the Red Hat Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.
rhnsat:/mnt# 

Active and Updating the Satellite

Since the ISO image is always a bit outdated, you need to activate and update the Satellite after its installation.

Upgrading the Database schema

rhnsat:~# rhn-satellite stop
rhnsat:~# /etc/init.d/rh-postgresql95-postgresql start
rhnsat:~# spacewalk-schema-upgrade
rhnsat:~# rhn-satellite-activate --manifest=/root/manifest.zip --ignore-version-mismatch

Some more work to do

After the upgrade succeed there is some work work to do.

Initial Sync with CDN

Unfortunately cdn-sync does not inherit the history which channels have been synced previously with satellite-sync. You need to once sync each channel again. Only missing data will be downloaded.

rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listbasechannels); do cdn-sync -c $i; done
rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listchildchannels); do cdn-sync -c $i; done

If you have custom channels, this will produce errors as the custom channels are not available in CDN. Just ignore them.

Rebuild the search index

rhnsat:~# service rhn-search cleanindex

Have fun 🙂

New features in Satellite 5.8

Redhat Satellite 5 was released in version 5.8. based on Spacewalk 2.5. It will probably be the last upgrade available, Support ends in January 2019.

New features and enhancements

  • The major new feature is the introduction of support for the CDN for both, Satellite activation and content sync. The key benefit is a massively enhanced performance for content sync. It’s now called cdn-sync, not satellite-sync anymore. Be aware that some custom scripts as well as cronjobs must be updated as well.This change also introduces the usage of Satellite Manifests instead of the old Certificates
  • Introduction of the new CLI tool taskotop which allows you to watch the activity of the Taskomatic Daemon.
  • PostgreSQL is upgraded to 9.5 which brings, compared to 9.2, some performance improvements as well.
  • More Perl bits have been rewritten to Java
  • Java JRE is now IBM’s Version 1.8
  • A few new commands in the spacecmd CLI
  • Lots of bugfixes and small enhancements

Removed features

There are some features that have been dropped with this release.

  • Support for Patch management of Solaris Systems. Who was using that? I can not remember that I’ve seen a company using that feature.
  • Monitoring is gone as well, I only know one organization that have used that feature. Most companies are using Icinga or Nagios.

Usange of cdn-sync

Populate Repository Metadata

The listing of available channels is working off-line. To be able to see the number of packages assigned to which channels you need to download the repository metadata first.

[root@sat58 ~]# cdn-sync --count-packages
14:05:25 Number of channels: 1271
14:05:25 Number of repositories: 1456
Downloading repomd:   |##################################################| 100.0% 
Comparing repomd:     |##################################################| 100.0% 
Downloading metadata: |##################################################| 100.0% 
Counting packages:    |##################################################| 100.0% 
14:42:21 Total time: 0:36:56
[root@sat58 ~]# 

Be aware that this will take a while, depending on how many entitlements are defined in the Satellite Manifest.

To keep that data up to date you should add a cronjob to do so.

[root@sat58 ~]# echo '0 1 * * * perl -le "sleep rand 9000" && /usr/bin/cdn-sync --count-packages' >> /etc/cron.d/cdn-sync-populate-metadata

Initial content sync

Similar to the old satellity-sync you provide the parameter -c repeatedly for all repos to be synced.

[root@sat58 ~]# cdn-sync -c rhel-x86_64-server-7              
11:16:20 ======================================
11:16:20 | Channel: rhel-x86_64-server-7
11:16:20 ======================================
11:16:20 Sync of channel started.
11:16:20 Repo URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os
11:16:28 Packages in repo:             14275
11:16:41 Packages already synced:          0
11:16:41 Packages to sync:             14275
11:16:42 New packages to download:     14275
11:16:43 1/14275 : 389-ds-base-1.3.5.10-20.el7_3.x86_64.rpm
11:16:43 2/14275 : 389-ds-base-1.3.4.0-26.el7_2.x86_64.rpm
11:16:43 3/14275 : 389-ds-base-1.3.5.10-11.el7.x86_64.rpm
11:16:43 4/14275 : 389-ds-base-1.3.3.1-16.el7_1.x86_64.rpm
[.. output ommited ..]
11:57:03 14275/14275 : zsh-5.0.2-14.el7_2.2.x86_64.rpm
Importing packages:     |##################################################| 100.0% 
13:10:05 Linking packages to channel.
13:10:19 Repo https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os has comps file 730c62cc7600c7518e4920f800cb9af6b73d75ba-comps.xml.
13:10:20 Repo https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os has 1885 errata.
13:10:50 Kickstartable tree not detected (no valid treeinfo file)
13:10:50 Repo URL: https://cdn.redhat.com/content/dist/rhel/server/7/7.3/x86_64/kickstart
13:10:55 Packages in repo:              4751
13:12:32 No new packages to sync.
13:12:32 Linking packages to channel.
13:12:44 Repo https://cdn.redhat.com/content/dist/rhel/server/7/7.3/x86_64/kickstart has comps file c542e4cf37dd210de68877b53f41d92dc7686c6e1b35ca4b1852f2e62fca2c72-comps-Server.x86_64.xml.gz.
13:12:44 Repo https://cdn.redhat.com/content/dist/rhel/server/7/7.3/x86_64/kickstart has 0 errata.
13:12:44 Added new kickstartable tree ks-rhel-x86_64-server-7-7.3. Downloading content...
13:12:44 Gathering all files in kickstart repository...
Downloading kickstarts: |##################################################| 100.0%
[.. output ommited ..]
13:24:53 Sync of channel completed in 2:08:33.
13:24:54 Total time: 2:08:33
[root@sat58 ~]# 

A subsequent run of cdn-sync without any parameters behaves like satellite-sync, its syncing previously synced channels.

You probably want to schedule a cronjob for daily syncing new content

[root@sat58b ~]# echo '0 1 * * * perl -le "sleep rand 9000" && /usr/bin/cdn-sync' >> /etc/cron.d/cdn-sync

The output of the sync actions are logged to /var/log/rhn/cdnsync.log

Clearing the cache

Remember rm -rf /var/cache/rhn/satsync/* when something went wrong? That’s gone :-). You just use cdn-sync –clear-cache.

cdn-sync --clear-cache

Upgrading from Satellite 5.7

I’ve not found the time yet to test the upgrade, I’ll let you know about my experience and thoughts in a few days.

Conclusion

After approximately 15 years, old school Redhat Satellite 5 will finally be replaced with Satellite 6 which is built on base of completely different technologies such as The Foreman, Pulp, Katello etc.

Satellite 5.8 is a very mature release no major bugs are known.

Satellite users are encouraged to discover Satellite 6 now, to be ready for the transition to be made in 2020.

Have fun 🙂