Installing Red Hat Satellite 6 with Letsencrypt certificates

Red Hat Satellite 6 is a nice tool for system life cycle management. It can get complex and even installation is sometimes tricky. This article is about how to install Satellite, it does not explain the principals and concepts behind it.

Requirements

A valid subscription for the Satellite (and optional for the capsule).

The system requirements are listed here.

There is one important thing the install guide is missing: Satellite 6.4 will not work in IPv6 only environments. There must be an IPv4 address configured, even if it is just an RFC1918 private address. You need to add this IP in /etc/hosts. The reason is that several daemons are listening on IPv4 addresses only. One of them is important: Apache QPID which is used for i.e. errata application via katello-agent. But there are two [update] undocumented installer parameters [/update].

Proxy is mandatory in IPv6 only environments

If your Satellite is only able to connect to the internet via IPv6, you need a IPv4 capable proxy to talk to subscription.rhsm.redhat.com which is not reachable by IPv6. That is the host the subscription manager is talking to.

Install EPEL and certbot

It makes sense to use officially valid certs since they are available for free usage from Letsencrypt. Certbot is available from EPEL and a handy way to request certificates using the ACME protocol.

[root@sat6 ~]# yum -y install http://ftp.tu-chemnitz.de/pub/linux/epel/7Server/x86_64/Packages/e/epel-release-7-11.noarch.rpm

It is important to disable EPEL by default to not get conflicts with RPMs from other repositories. Just enable EPEL when needed and double check.

[root@sat6 ~]# yum-config-manager --disable epel

Install the certbot package

[root@sat6 ~]# yum -y install certbot --enablerepo=epel

Issue the cert

[root@sat6 ~]# certbot certonly -n --standalone --agree-tos --domains sat6.example.com -m user@example.com

Download the CA-Certs

Root-CA

[root@sat6 ~]# wget https://www.identrust.com/node/935 -O trustidrootx3_chain.p7b

Convert the p7b to PEM format.

[root@sat6 ~]# openssl pkcs7 -in trustidrootx3_chain.p7b -inform DER -print_certs -out trustidrootx3_chain.pem

Intermediate CA-Cert

[root@sat6 ~]# wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

Create the CA bundle file

[root@sat6 ~]# cp trustidrootx3_chain.pem bundle-ca-cert.pem
[root@sat6 ~]# cat lets-encrypt-x3-cross-signed.pem.txt >> bundle-ca-cert.pem

Check the certificate

You can check if you made all things as expected, run the check command.

[root@sat6 ~]# katello-certs-check -c "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" -k "/etc/letsencrypt/live/sat6.example.com/privkey.pem" -b "/root/bundle-ca-cert.pem"

Running the installer

Note: The parameters –foreman-proxy-content-qpid-router-hub-addr :: –foreman-proxy-content-qpid-router-agent-addr :: are not documented and only needed if you want to be able that the capsule and/or clients will be able to communicate over IPv6.

[root@sat6 ~]# satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/archive/sat6.example.com/fullchain2.pem" --certs-server-key "/etc/letsencrypt/archive/sat6.example.com/privkey2.pem" --certs-server-ca-cert "/root/bundle-ca-cert.pem" --certs-update-server --certs-update-server-ca --katello-proxy-url=http://proxy.example.com --katello-proxy-port=3128 --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

Installing a Capsule Server

If you want to use a capsule server within an environment with Letsencrypt certificates, its a bit more complex, but however, it works.

Install the software

Needless to say that your capsule needs to have the correct repos enabled. For details, please see here.

[root@capsule ~]# yum install satellite-capsule

Request the Certificate

On the Satellite, request a certificate for the capsule. Note: This only works with the DNS challenge, so you need access to your DNS server.

[root@sat6 ~]# certbot -d capsule.example.com --manual --preferred-challenges dns certonly

Prepare the certificates and key

Create a directory and copy the certificates

[root@sat6 ~]# mkdir /root/capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/privkey.pem capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/cert.pem capsule.example.com
[root@sat6 ~]# cp /root/bundle-ca-cert.pem capsule.example.com

Validate the certificate

[root@sat6 ~]# katello-certs-check -c /root/capsule.example.com/cert.pem -b /root/capsule.example.com/bundle-ca-cert.pem -k /root/capsule.example.com/privkey.pem

If all is fine, run the capsule generator:

[root@sat6 ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar /root/capsule.example.com-certs.tar --server-cert /root/capsule.example.com/cert.pem --server-key /root/capsule.example.com/privkey.pem --server-ca-cert /root/capsule.example.com/bundle-ca-cert.pem

Copy the resulting tarball to the capsule server:

[root@sat6 ~]# scp capsule.example.com-certs.tar capsule.example.com

Running the installer

[root@capsule ~]#  satellite-installer --scenario capsule\
--foreman-proxy-content-parent-fqdn      "sat6.example.com"\
--foreman-proxy-register-in-foreman      "true"\
--foreman-proxy-foreman-base-url         "https://sat6.example.com"\
--foreman-proxy-trusted-hosts            "sat6.example.com"\
--foreman-proxy-trusted-hosts            "capsule.example.com"\
--foreman-proxy-oauth-consumer-key       "the key"\
--foreman-proxy-oauth-consumer-secret    "the secret"\
--foreman-proxy-content-certs-tar        "/root/capsule.example.com-certs.tar"\
--puppet-server-foreman-url              "https://sat6.example.com" \
--foreman-proxy-content-qpid-router-hub-addr :: \
--foreman-proxy-content-qpid-router-agent-addr ::

That’s it 🙂

Do not ask me how certificate renewal works, I’ll let you know in three months 😉

Workaround for IPv6-only Networks

Unfortunately the satellite-installer configures Apache QPID not correctly, it will be set up to use IPv4 only by default. That means, IPv6-only hosts (i.e. including the capsule) are unable to communicate with the Satellite.

There is a workaround: Add two additional listeners on the Satellite and one on the Capsule. Be aware: The Satellite installer overwrites your changes every time when you run it, i.e. for upgrades or adding new features. create a backup of the config file.

There are two undocumented parameters for the satellite installer: –foreman-proxy-content-qpid-router-hub-addr :: and –foreman-proxy-content-qpid-router-agent-addr ::. You can add them during first time run as well as after an initial installation.

It is the same procedure in the Satellite as well as on the Capsule.

satellite-installer --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

This behavior is already fixes upstream as you can see it here: https://github.com/theforeman/puppet-foreman_proxy_content/commit/89b4ea988d18f100b806e7cddc2dca623b68f084″.

Setting up a 6in4 tunnel with Fedora

Why using IPv6 Tunnels anyway?

Today, most Internet access providers are IPv6 enabled. However, unfortunately the majority of them do not provide a static /64 prefix, you will get it dynamically assigned. Some providers can assign you a static prefix for a surcharge.

That’s useless if you want to ensure end-to-end connectivity with your Gadgets at home.

Choosing a tunnel provider

Since 2004 I had my own IPv6 prefix from SixXS. Pretty sad that they are shutting down its services on 2017-06-06.

Time to look for an alternative. Wikipedia has a list of public tunnel brokers. Most brokers are providing only PoP’s in one country. For most users, the only option left is Hurricane Electric which offers tunnels to PoP’s on three continents in various cities.

Setup in Fedora

The whole setup is rather simple, there is just one thing you should keep in mind. The provided Client IPv6 Address is not in the same subnet as the Routed /64. You easily copy-paste the wrong address and you will end up in a nice routing loop. The difference is i.e. 2001:470:6c:something vs. 2001:470:6d:something, only the one character of difference. It was taking me more that an hour to figure out 😉

Tunnel configuration

Create a new interface for the tunnel.

cat >>/etc/sysconfig/network-scripts/ifcfg-he-ipv6 <<EOF
DEVICE=he-ipv6
TYPE=sit
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
# The IPv4 address depends on the PoP you choose
IPV6TUNNELIPV4=216.66.86.114
# That is the IPv6 address of the client, not from the routed prefix
IPV6ADDR=2001:db8:dead:beef::2/64
EOF

LAN interface configuration

In my case I use a bridge to be able to provide IPv6 connectivity not only for the LAN but for Wifi and VPN as well.

cat >>/etc/sysconfig/network-scripts/ifcfg-br0 <<EOF
DEVICE=br0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=192.168.100.1
NETMASK=255.255.255.0
IPV6_AUTOCONF=no
IPV6INIT=yes
IPV6TO4INIT=no
# That is a random IP from your routed /64 prefix. Usually just use the first one
IPV6ADDR=2001:db8:cafe:1::1/64
EOF

 

Enable IPv6 routing

echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf

Setting the default device for IPv6 routing

echo "IPV6_DEFAULTDEV=he-ipv6"  >> /etc/sysconfig/network

Setting up the Route Advertisement Daemon (RADVD

There are several ways of how to configure the clients with an IPv6 address. DHCP6, Static manual configuration and the most easy way is to use RADVD which tells the clients which prefix to use (prefix + fffe + MAC). The client itself adds the MAC address on top of the prefix.

Your clients will always get the same IPv6 address, this may be a privacy problem for you or not. In contrary to SixXS, Hurricane Electric does not provide your name and address to whois, only the city and the ZIP code is made public.

Install radvd if not yet done

router:~# dnf install radvd
router:~# systemctl enable radvd.service

Configuration for the example of the prefix 2001:db8:cafe:1/64

cat >> /etc/radvd.conf << EOF
 interface br0
 {
        AdvSendAdvert on;
        MinRtrAdvInterval 30;
        MaxRtrAdvInterval 100;
        AdvLinkMTU 1480;
        prefix 2001:db8:cafe:1::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
 
} ;
EOF

After restarting your network your done, have fun with IPv6 🙂

Setting up DNS

I’m not going into the details here. I’m using FreeIPA for DNS management, DNS entries are created automatically when you enroll your clients. The only thing you need to do is adding the prefix to be able to do reverse lookups.

[root@ipa1 ~]# ipa dnszone-add --name-from-ip=2001:db8:cafe::/64 --dynamic-update=true 
Zone name [0.0.0.0.e.f.a.c.8.b.d.0.1.0.0.2.ip6.arpa.]: 
  Zone name: 0.0.0.0.e.f.a.c.8.b.d.0.1.0.0.2.ip6.arpa.
  Active zone: TRUE
  Authoritative nameserver: ipa1.example.com.
  Administrator e-mail address: hostmaster
  SOA serial: 1490512663
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  BIND update policy: grant EXAMPLE.COM krb5-subdomain 0.0.0.0.e.f.a.c.8.b.d.0.1.0.0.2.ip6.arpa. PTR;
  Dynamic update: TRUE
  Allow query: any;
  Allow transfer: none;
[root@ipa1 ~]# 

Reverse DNS delegation

Hurricane Electric allows you to delegate DNS lookups of your prefix to your DNS server(s). Make use of that is good practice.

Read further

Fancy stuff

If you finished setting up all your services such as DNS, HTTP, SMTP etc. with IPv6, get “certified” at https://ipv6.he.net/certification/cert-main.php and get a fancy batch like this: IPv6 Certification Badge for ldelouw

Have fun! 🙂

Configure SSSD to work on IPv6-only Hosts

SSSD is used for the client side of IPA and other centralized Identity Management Services. Unfortunately it does not behave as it should. The default is to look up first IPv4 addresses and if that fails IPv6 should be used. Well, if IPv4 fails, the whole request fails and you got weird error messages when joining an IPA domain.

As the pool for IPv4 addresses is depleted, IPv6 is getting more and more important. Thus, IPv6-only hosts are on the rise.

Here is an example error message from the IPA client.

[root@ipv6host ~]# ipa-client-install
[output ommited] 
SSSD enabled
Configured /etc/openldap/ldap.conf
Unable to find 'admin' user with 'getent passwd admin@example.com'!
Unable to reliably detect configuration. Check NSS setup manually.
[output ommited]

The host itself gets properly joined to the IPA domain and authentication works with Kerberos but you can not log in because SSSD fails.

Workaround

Configure SSSD to only use IPv6. This is done in /etc/sssd/sssd.conf

[domain/example.com]
lookup_family_order = ipv6_only
cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = example.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = ipv6host.example.com
chpass_provider = ipa
ipa_server = _srv_, ipa1.example.com
ldap_tls_cacert = /etc/ipa/ca.crt
[sssd]
services = nss, sudo, pam, ssh

domains = example.com
[nss]
homedir_substring = /home

[pam]

[sudo]

[autofs]

[ssh]

[pac]

[ifp]

Solution

At the moment there is no solution yet (just the workaround described), but its addressed at the SSSD project team, as you can see in https://pagure.io/SSSD/sssd/issue/2128 and https://bugzilla.redhat.com/show_bug.cgi?id=1021435

Happy IPv6-ing 🙂