Renew Letsencrypt certificates for Red Hat Satellite 6 and Capsule

Letsencrypt certificates are only valid for just three months. The procedure to renew x509 certificates in Red Hat Satellite 6 is not so straight forward and its even more complex for Capsule servers.

In an earlier post I was writing about how to set up a Satellite 6 and a capsule using Letsencrypt certificates. This post is a follow up on that.

Be aware: You must follow this procedure before the certificate expires or the Satellite simply stops working, all clients will refuse to communicate with the Red Hat Satellite Server. This is a security feature, not a bug.

Obtain the Satellite certificate

This is straight forward, stop the httpd and just use certbot, you must use the -d parameter because for the capsule it will fail.

systemctl stop httpd && certbot renew -d sat6.example.com

Install the renewed certificate into Satellite

You need to run the Satellite installer to make your certificate active:

satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" --certs-server-key "/etc/letsencrypt/live/sat6.example.com/privkey.pem" --certs-server-ca-cert "/root/ca-cert.pem" --certs-update-server --certs-update-server-ca

Keep the output as you need the oauth key and secret for the capsule.

Obtain the new certifcate for the capsule

This step must be done on the Satellite, not on the Caspule.

The only way to obtain a cert for a server different than than the target is to make use of the DNS challenge.

certbot -d capsule.example.com --manual --preferred-challenges dns certonly

It will ask you to create a DNS TXT entry as a challenge to ensure you are in control of the domain. When your DNS entry is ready, hit enter.

Create the tarball with the certifcates for the Capsule

capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar  "~/$CAPSULE-certs.tar" --server-cert "/etc/letsencrypt/live/capsule.example.com/cert.pem" --server-key "/etc/letsencrypt/live/capsule.example.com/privkey.pem" --server-ca-cert "/root/capsule.example.com/bundle-ca-cert.pem" --certs-update-server

The next step is to copy the tarball to your capsule:

scp /root/capsule.example.com-certs.tar capsule.example.com:

Install the new certificate on the Capsule

This step must be done on the Capsule server

satellite-installer --scenario capsule\
                      --foreman-proxy-content-parent-fqdn           "sat6.example.com"\
                      --foreman-proxy-register-in-foreman           "true"\
                      --foreman-proxy-foreman-base-url              "https://sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "capsule.exmple.com"\
                      --foreman-proxy-oauth-consumer-key            "The Key"\
                      --foreman-proxy-oauth-consumer-secret         "The Secret"\
                      --foreman-proxy-content-certs-tar             "/root/capsule.example.com-certs.tar"\
                      --puppet-server-foreman-url                   "https://sat6.example.com"

Feedback welcome…

Have fun 🙂

Installing Red Hat Satellite 6 with Letsencrypt certificates

Red Hat Satellite 6 is a nice tool for system life cycle management. It can get complex and even installation is sometimes tricky. This article is about how to install Satellite, it does not explain the principals and concepts behind it.

Requirements

A valid subscription for the Satellite (and optional for the capsule).

The system requirements are listed here.

There is one important thing the install guide is missing: Satellite 6.4 will not work in IPv6 only environments. There must be an IPv4 address configured, even if it is just an RFC1918 private address. You need to add this IP in /etc/hosts. The reason is that several daemons are listening on IPv4 addresses only. One of them is important: Apache QPID which is used for i.e. errata application via katello-agent. But there are two [update] undocumented installer parameters [/update].

Proxy is mandatory in IPv6 only environments

If your Satellite is only able to connect to the internet via IPv6, you need a IPv4 capable proxy to talk to subscription.rhsm.redhat.com which is not reachable by IPv6. That is the host the subscription manager is talking to.

Install EPEL and certbot

It makes sense to use officially valid certs since they are available for free usage from Letsencrypt. Certbot is available from EPEL and a handy way to request certificates using the ACME protocol.

[root@sat6 ~]# yum -y install http://ftp.tu-chemnitz.de/pub/linux/epel/7Server/x86_64/Packages/e/epel-release-7-11.noarch.rpm

It is important to disable EPEL by default to not get conflicts with RPMs from other repositories. Just enable EPEL when needed and double check.

[root@sat6 ~]# yum-config-manager --disable epel

Install the certbot package

[root@sat6 ~]# yum -y install certbot --enablerepo=epel

Issue the cert

[root@sat6 ~]# certbot certonly -n --standalone --agree-tos --domains sat6.example.com -m user@example.com

Download the CA-Certs

Root-CA

[root@sat6 ~]# wget https://www.identrust.com/node/935 -O trustidrootx3_chain.p7b

Convert the p7b to PEM format.

[root@sat6 ~]# openssl pkcs7 -in trustidrootx3_chain.p7b -inform DER -print_certs -out trustidrootx3_chain.pem

Intermediate CA-Cert

[root@sat6 ~]# wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

Create the CA bundle file

[root@sat6 ~]# cp trustidrootx3_chain.pem bundle-ca-cert.pem
[root@sat6 ~]# cat lets-encrypt-x3-cross-signed.pem.txt >> bundle-ca-cert.pem

Check the certificate

You can check if you made all things as expected, run the check command.

[root@sat6 ~]# katello-certs-check -c "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" -k "/etc/letsencrypt/live/sat6.example.com/privkey.pem" -b "/root/bundle-ca-cert.pem"

Running the installer

Note: The parameters –foreman-proxy-content-qpid-router-hub-addr :: –foreman-proxy-content-qpid-router-agent-addr :: are not documented and only needed if you want to be able that the capsule and/or clients will be able to communicate over IPv6.

[root@sat6 ~]# satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/archive/sat6.example.com/fullchain2.pem" --certs-server-key "/etc/letsencrypt/archive/sat6.example.com/privkey2.pem" --certs-server-ca-cert "/root/bundle-ca-cert.pem" --certs-update-server --certs-update-server-ca --katello-proxy-url=http://proxy.example.com --katello-proxy-port=3128 --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

Installing a Capsule Server

If you want to use a capsule server within an environment with Letsencrypt certificates, its a bit more complex, but however, it works.

Install the software

Needless to say that your capsule needs to have the correct repos enabled. For details, please see here.

[root@capsule ~]# yum install satellite-capsule

Request the Certificate

On the Satellite, request a certificate for the capsule. Note: This only works with the DNS challenge, so you need access to your DNS server.

[root@sat6 ~]# certbot -d capsule.example.com --manual --preferred-challenges dns certonly

Prepare the certificates and key

Create a directory and copy the certificates

[root@sat6 ~]# mkdir /root/capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/privkey.pem capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/cert.pem capsule.example.com
[root@sat6 ~]# cp /root/bundle-ca-cert.pem capsule.example.com

Validate the certificate

[root@sat6 ~]# katello-certs-check -c /root/capsule.example.com/cert.pem -b /root/capsule.example.com/bundle-ca-cert.pem -k /root/capsule.example.com/privkey.pem

If all is fine, run the capsule generator:

[root@sat6 ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar /root/capsule.example.com-certs.tar --server-cert /root/capsule.example.com/cert.pem --server-key /root/capsule.example.com/privkey.pem --server-ca-cert /root/capsule.example.com/bundle-ca-cert.pem

Copy the resulting tarball to the capsule server:

[root@sat6 ~]# scp capsule.example.com-certs.tar capsule.example.com

Running the installer

[root@capsule ~]#  satellite-installer --scenario capsule\
--foreman-proxy-content-parent-fqdn      "sat6.example.com"\
--foreman-proxy-register-in-foreman      "true"\
--foreman-proxy-foreman-base-url         "https://sat6.example.com"\
--foreman-proxy-trusted-hosts            "sat6.example.com"\
--foreman-proxy-trusted-hosts            "capsule.example.com"\
--foreman-proxy-oauth-consumer-key       "the key"\
--foreman-proxy-oauth-consumer-secret    "the secret"\
--foreman-proxy-content-certs-tar        "/root/capsule.example.com-certs.tar"\
--puppet-server-foreman-url              "https://sat6.example.com" \
--foreman-proxy-content-qpid-router-hub-addr :: \
--foreman-proxy-content-qpid-router-agent-addr ::

That’s it 🙂

Do not ask me how certificate renewal works, I’ll let you know in three months 😉

Workaround for IPv6-only Networks

Unfortunately the satellite-installer configures Apache QPID not correctly, it will be set up to use IPv4 only by default. That means, IPv6-only hosts (i.e. including the capsule) are unable to communicate with the Satellite.

There is a workaround: Add two additional listeners on the Satellite and one on the Capsule. Be aware: The Satellite installer overwrites your changes every time when you run it, i.e. for upgrades or adding new features. create a backup of the config file.

There are two undocumented parameters for the satellite installer: –foreman-proxy-content-qpid-router-hub-addr :: and –foreman-proxy-content-qpid-router-agent-addr ::. You can add them during first time run as well as after an initial installation.

It is the same procedure in the Satellite as well as on the Capsule.

satellite-installer --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

This behavior is already fixes upstream as you can see it here: https://github.com/theforeman/puppet-foreman_proxy_content/commit/89b4ea988d18f100b806e7cddc2dca623b68f084″.

Improve your bash shell working experience

This article shows some hints how to improve your bash shell working experience to reach higher productivity. Just simple shortcuts that are not so well known.

Using the History

The bash history is underestimated when it comes to usability. Here some nice stuff to do with the history.

Search the history

Every command is kept in the history. The simplest way to use the history is using the cursor-up/down keys. Most users are aware or [ctrl]-r. Usually you hit [ctrl]-r (r like reverse search) several times and miss the command, roll your eyes, hit [ctrl]-c and do it again. Why not using forward search with [ctrl]-s in such a case? Well, that suspends your terminal. It comes from the ancient times and is not needed anymore.

Turn off terminal suspension

echo "stty -ixon" >> /etc/profile

Now you can search the history back and forward by using [ctrl]-r and [ctrl]-s.

Using another command with the same last argument

When you i.e. do ls a file and decide to edit it, you don’t need to retype the whole file path or using the mouse to copy-paste it. Use the [Alt]-. (dot) combination. It inserts the last argument used. So after ls -la /tmp/file.txt you type vi [Alt].. Review and hit enter to execute.

You can also reuse other than the last arguments, but this is more complex and does not speed up things a lot, copy-paste with your mouse is usually faster in such a case.

Forgot to sudo?

When you want to cat i.e. /etc/sssd/sssd.conf you need root access. As a normal user, access is denied.

[luc@fedora ~]$ cat /etc/sssd/sssd.conf
cat: /etc/sssd/sssd.conf: Permission denied
[luc@fedora ~]$ sudo !!
sudo cat /etc/sssd/sssd.conf
[domain/example.com]

The !! also called bash bang does the trick. It just repeats the same command as used before which all arguments. Be aware that the command is executed immediately.

Bash can copy-paste as well!

Copy-paste is not only available in graphical environments but in the bash shell as well.

If you need to type some different commands all with the same arguments, cut the stuff. Position the curser to the position on the line from where you want to copy and hit [ctrl]-k. When you want to paste, hit [ctrl]-y.

You may also achieve that using othercommand !*. Using !(bash bang) can be dangerous because the command will be executed immediately, the copy-paste method is more safe.

That also works with single words etc. basically everything where you cut or delete some stuff like [alt]-d, [ctrl]-w, [ctrl]-u

Using an editor for copy-paste from websites and word processors

There are a number of reasons why you don’t want to directly copy-paste to a shell. Sometimes the source content has not properly escaped line ends or its just garbage from word processors. You may want to review and edit appropriately before fire the command. There is a super lazy and convenient trick to do so.

The security usecase

Copy-Paste from a Website is a security nightmare. Copy-Paste the following two lines into an editor and you see what I mean.

Sample command
echo “Dont copy-paste”

Second sample

The HTML code used for that is:

Sample command<span style="font-size: 0; position: absolute; left: -100px; top: -100px"><br>:echo "Dont copy-paste"</span>
Second sample

Nice! Use an editor before pasting anything in a terminal, for the sake of security.

The word processor garbage usecase

Lot of documentation is written in word processors such as Libreoffice, MS-Office and others. They replace double hyphens to a single one and nasty stuff such as single quotes to backticks. Just for a thing called usability.

When copy-paste that stuff, you probably want to review and edit it first.

Set the EDITOR environment variable

If you are too lazy to fire up vim, you can set the EDITOR environment variable to an editor of your choice (vim, emacs, nano, whatever), system wide in /etc/profile or /etc/bashrc. A better idea is to put it in ~/.profile or ~/.bashrc.

echo "export EDITOR=vim" >>: ~./bashrc

Afterwards you can just hit [ctrl]-x-e and vim starts up. When save and exit vim, the command will be executed.

What are my Keybindings?

If you wonder what kind of shortcuts are defined in a shell, a lot are. use bind -p to show them.

Have fun 🙂