Using modern Protocols like HTTP/2 and QUIC

First there was HTTP, then HTTP/2 and now HTTP/2 over the QUIC protocol. Lets have a look at the available HTTP Clients and Servers that support HTTP/2 and the experimental QUIC protocol.

Introduction

The Hypertext Transfer Protocol (HTTP) was invented in 1991. Up to 2015 then there was only little to no evolution. In 2015 the HTTP/2 protocol was defined as a standard. HTTP/2 is much more efficient that its ancestors.

It features multiplexing, stream prioritization, binary transmission and much more. Its a huge step forward.

Nevertheless, there is a need for something more efficient. HTTP/2 is using TCP (Transmission Control Protocol) which was created in the early days of the Internet to have a reliable connection over unreliable networks. Today’s networks are much more reliable which allows the usage of the unreliable but very efficient UDP (User Datagram Protocol) to transmit data. As a consequence, QUIC was born. It is using UDP instead of TCP.

QUIC includes the crypto layer, so there is no need of a separate TLS layer. The goal is to use TLS 1.3 which is not ready as of writing this post.

Both, QUIC and TLS 1.3 are currently being defined as standards, the current state of the TLS Working group is publish here, the work of the QUIC working group is vailable here.

A good overview about QUIC can be found here.

Client Software

As of writing this post, all major Browsers are supporting HTTP/2 over TCP. When it comes to QUIC, there is little left. At the moment only Chrome and Opera are capable to access web sites with QUIC.

It is expected that this will change as soon as the standard is finalized.

Web sites

I’m not aware of any prominent Website using QUIC beside of google. HTTP/2 is used by a lot of prominent sites such as facebook, google and many others.

Server Software

The situation for HTTP/2 looks good, most webservers such as Apache HTTPD, NGINX etc. come with support for HTTP/2. Well, Apache does not work with the prefork MPM, that means you can not use mod_php with HTTP/2. You can make use of FastCGI but this means that Apache will be the slowest webserver available on the market. Better use NGINX.

If it comes to QUIC support, there is an experimental NGINX module available.

An option could be the commercial LiteSpeed Server.

From my point of view, the only usable Webserver for both, HTTP/2 and QUIC is Caddy. Its a relatively new open source project implementing a lot of new and experimental technologies. A nice feature is automatic HTTPS with Letsencrypt.

Caddy Webserver

Lets have a closer look to Caddy on Fedora 27. Its quite straight forward to install and configure.

Installation

[root@f27 ~]# dnf install caddy certbot

Configuration

cat > /etc/caddy/caddy.conf << EOF
:80 {
    gzip
    root /usr/share/caddy
}
EOF

Get a Letsencrypt Certficate

[root@f27 ~]# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
-------------------------------------------------------------------------------
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Plugins selected: Authenticator webroot, Installer None
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): f27.ldelouw.ch
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for f27.ldelouw.ch
Input the webroot for f27.ldelouw.ch: (Enter 'c' to cancel): /usr/share/caddy/
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/f27.ldelouw.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/f27.ldelouw.ch/privkey.pem
   Your cert will expire on 2018-05-31. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"

Configure TLS

cat >> /etc/caddy/caddy.conf << EOF
:443 {
    gzip
    root /usr/share/caddy
    tls /etc/letsencrypt/live/f27.ldelouw.ch/fullchain.pem /etc/letsencrypt/live/f27.ldelouw.ch/privkey.pem
}

EOF

Give the caddy user access to the cert and key

[root@f27 ~]# setfacl -m u:caddy:r-X /etc/letsencrypt/live

Enable QUIC

[root@f27 ~]# cp /usr/lib/systemd/system/caddy.service /etc/systemd/system/
[root@f27 ~]# sed -i 's#ExecStart=/usr/bin/caddy -conf /etc/caddy/caddy.conf -log stdout -root /tmp -agree#ExecStart=/usr/bin/caddy -conf /etc/caddy/caddy.conf -log stdout -root /tmp -agree -quic#g' /etc/systemd/system/caddy.service
[root@f27 ~]# systemctl daemon-reload
[root@f27 ~]# systemctl restart caddy

Checking the Result

Enabling QUIC in your brower

Point Chrome to chrome://flags/ and search for QUIC. Enable it and relaunch the browser.

Open Chrome and a second tab with chrome://net-internals/#quicType the URL, i.e. https://f27.ldelouw.ch. Switch the to chrome tab and see the Result.

QUIC Screenhot

QUIC Screenhot

Upgrading Redhat Satellite 5.7 to 5.8

Couple of days ago, Redhat released its latest and last major upgrade for Satellite 5.x. Its a rather important upgrade, you are advised to upgrade soon.

This upgrade contains some major improvements like stated in an earlier article

Disclaimer

I’m not responsible for any damage caused by the procedure provided here. Always create a backup before even thinking about upgrading a Satellite server.

Preparation

As always when you plan to upgrade your Satellite server to the latest version, you need to do some preparations first.

Ensure you will have enough disk space free in /var/opt/rh. The upgrade to 5.8 will also install a new PostgreSQL version located at /var/opt/rh/rh-postgresql95/lib/pgsql. The new version will roughly use the same diskspace as the old version 9.2 in /opt/rh/postgresql92/root/var/lib/pgsql.

Download the ISO

Visit https://access.redhat.com/downloads/content/250/ver=5.8/rhel—6/5.8/x86_64/product-software and make sure you select 5.8 and the architecture fitting you system (x86_64 or S390)

Get a new Satellite Certificate Manifest

Satellite 5.8 switches from Certificates to Manifests like Satellite 6. You need a Manifest to get it activated. You can create it by your own at the Subscription Management Application site, ensure you attach enough subscriptions to your Satellite server(s).

Backup

Usually an upgrade runs smooth, but just in case… it is recommended practice to have a recent backup ready. If your Satellite is running on a virtual machine, power off, snapshot and power on to have a consistent backup ready. For physical systems, db-control and the choice.

Backup the rest of your Satellite:

Create a copy of your rhn configuration directory as we need some information from the old files after the upgrade.

rhnsat:~# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")

Update your OS and Satellite 5.7

First step is to update the operating system and the Satellite 5.7 and apply the latest database schema updates as well.

rhnsat:~# yum -y update && reboot

Update the database schema if needed

To update the database schema, run the following command. Ideally it looks as follows:

rhnsat:~# spacewalk-schema-upgrade 
Schema upgrade: [satellite-schema-5.7.0.27-1.el6sat] -> [satellite-schema-5.7.0.27-1.el6sat]
Your database schema already matches the schema package version [satellite-schema-5.7.0.27-1.el6sat].
rhnsat:~# 

Switch from RHN to Subscription Manager

It is important to ensure you switched from RHN to subscription manager before doing the upgrade. You can check if this is the case with:

rhnsat:~# subscription-manager repos --list-enabled
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   rhel-6-server-rpms
Repo Name: Red Hat Enterprise Linux 6 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/os
Enabled:   1

Repo ID:   rhel-6-server-satellite-5.7-rpms
Repo Name: Red Hat Satellite 5.7 (for RHEL 6 Server) (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/satellite/5.7/os
Enabled:   1

rhnsat:~# 

If this is not yet done, have a look at the knowledge base article located here: https://access.redhat.com/articles/2884191

Functionality Check with the old version 5.7

It is recommended to restart and check a software functionality before upgrading to be able to pinpoint problems if there are some.

rhnsat:~# rhn-satellite restart

Cleanup

Review the software channels in use and delete unused channels as this can free up quite some disk space and reduces the size of the database significantly.

rhnsat:~# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
Deleting package metadata (20):
                  ________________________________________
Removing:         ######################################## - complete
rhnsat:~# 

Delete old system snapshots which are not used anymore. The following example deletes all snapshots which are older than one month:

rhnsat:~#  sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000") 

Check for old MD5 user passwords and certificates

Check if there are still some users with an md5 hashed password. The same applies to certificates.

rhnsat:~# spacewalk-report users-md5
rhnsat:~# spacewalk-report system-md5-certificates

If there are any, please have a look to https://access.redhat.com/documentation/en-us/red_hat_satellite/5.8/html/installation_guide/ch10s03

RTFM

If not done yet, install or update the rhn-upgrade package which contains the instructions how to proceed.

rhnsat:~# yum -y install rhn-upgrade

The package contains not only SQL- and other useful scripts needed for the upgrade but also important documents to read. The are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

For most users, the document satellite-upgrade-postgresql.txt applies.

Do not forget to read the updated product documentation as well:

Performing the upgrade

rhnsat:~# mount satellite-5.8-rhel-6-x86_64-dvd.iso /mnt  -o loop
rhnsat:~#  cd /mnt
rhnsat:/mnt# ./install.pl --upgrade
* Starting Red Hat Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHSM Registration.
** Registration: System is already registered with RHSM.  Not re-registering.
* RHSM Subscriptions.
** Subscriptions: Subscription providing 'Red Hat Satellite' already attached.
** Subscriptions: Subscription providing 'Red Hat Enterprise Linux Server' already attached.
** Subscriptions: Disabling all RHSM repositories (rhel-6-server-rpms, rhel-6-server-satellite-5.7-rpms).
** Subscriptions: All repositories disabled.
** Subscriptions: Enabling RHEL repository.
** Subscriptions: RHEL repository enabled.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Installing Satellite packages.
Warning: more packages were installed by yum than expected:
        python-backports
        python-backports-ssl_match_hostname
        python-chardet
        python-requests
        python-urllib3
* Now running spacewalk-setup.
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
*** Upgrading embedded database.
** Database: Populating database.
** Database: Skipping database population.
* Configuring tomcat.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating Red Hat Satellite.
** Manifest not activated.
** Upgrade process requires the manifest to be activated after the schema is upgraded.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? 
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? 
This portion of the Red Hat Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.
rhnsat:/mnt# 

Active and Updating the Satellite

Since the ISO image is always a bit outdated, you need to activate and update the Satellite after its installation.

Upgrading the Database schema

rhnsat:~# rhn-satellite stop
rhnsat:~# /etc/init.d/rh-postgresql95-postgresql start
rhnsat:~# spacewalk-schema-upgrade
rhnsat:~# rhn-satellite-activate --manifest=/root/manifest.zip --ignore-version-mismatch

Some more work to do

After the upgrade succeed there is some work work to do.

Initial Sync with CDN

Unfortunately cdn-sync does not inherit the history which channels have been synced previously with satellite-sync. You need to once sync each channel again. Only missing data will be downloaded.

rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listbasechannels); do cdn-sync -c $i; done
rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listchildchannels); do cdn-sync -c $i; done

If you have custom channels, this will produce errors as the custom channels are not available in CDN. Just ignore them.

Rebuild the search index

rhnsat:~# service rhn-search cleanindex

Have fun 🙂

Using OTP Tokens and 2FA with FreeIPA 4.0

On 2014-07-08 FreeIPA 4.0 was released. One of the most interesting new features is the support of two factor authentication (2FA). I was curious about how to set it up and get it running. Unfortunately the documentation does not tell much about the OTP setup.

What is OTP and 2FA? An overview
OTP stands for One Time Password and 2FA for two factor authentication. OTP is available since long time, in the beginning usually as a list of passwords printed on paper. It was enhancing security gradually but was an operational nightmare.

RSA then came up with harware tokens somewhere in the 1990this which made it much more usable. Also 2FA was introduced. the two factors are ownership (or possession) and knowledge. One needs to obtain a piece of hardware (Hardware Token or a smart phone with a software token) and knowledge (knowing the password).

Meanwhile a lot of competing tokens are on the market, as well as so called soft-tokens. Most (or all?) of the hardware tokens are proprietary, making system configuration a nightmare (RSA PAM modules and stuff). On the other hand, every proprietary solution comes with the support of Radius. There is a quite new definition of using a Radius proxy to use those tokens with Kerberos and connect them with IPA.

However, hardware tokens and Radius proxies have been out of scope for my initial test. Lets go for the simpler soft token way.

Installing FreeIPA 4.0
It is planed to include FreeIPA 4.0 in Fedora 21 which will be released later this year. For testing you can either use Fedora Rawhide 21 or Fedora 20 with an external Yum repository. I was choosing the later way.

wget https://copr.fedoraproject.org/coprs/pviktori/freeipa/repo/fedora-20-i386/pviktori-freeipa-fedora-20-i386.repo -O /etc/yum.repos.d/pviktori-freeipa-fedora-20-i386.repo

The rest of the installation is the same as with (Free)IPA2 and (Free)IPA3. Please have a look at my earlier Post

Enabling OTP
You can either enable OTP on a global scope or per user. At the moment I recommend it on a per-user base.

ipa user-mod username --user-auth-type=otp

If you want to enable users to authenticate with more than one method, user –user-auth-type={otp,password}

Adding a new user with OTP enabled will probably be possible in the future. There seems to be a bug, according to ipa user-add –help it is supposed to be working.

ipa user-add hwurst --first="Hans" --last="Wurst" --user-auth-type=otp

Adding a token
The best way for a user to add a token is probably the web interface. Lets call it self-service. The user first authenticates with username and the initial password set by the admin to set a new one. The OTP field can be ignored for the moment.

After authentication, the user can navigate to “OTP Tokens” on the top navigation bar and add a new token. This looks as following:

ipa-otpThe ID needs to be unique, this can case problems when users are adding the tokens by themself as people would tend to provide a simple ID by themself. When not providing an ID, one will be generated. The field Unique ID should IMHO not be available for ordinary users.

After adding the token, login via password only is not possible anymore (unless explicitly enabled with the user-auth-type).

After hitting “Add”, a QR code will be shown. This allows users to scan the code with the Smartphone app, such as FreeOTP and Google Authenticator.

The next step users needs to do is to sync the token. This can be done by returning to the login screen and clicking on “Sync OTP Token” right left to the Login button.

ipa-otp2With a generated Unique ID (=Token ID) its quite annoying to enter that ID. However, usually this only needs to be one once 🙂

 

 

 

 

Limitations

The release notes mentions that there are concerns about the scalability when using HOTP, where TOTP has a known issue that tokens can be reused, but only within a short timeframe.

I see another issue which is a kind of a chicken-and-egg problem: After adding a user, this user is able to login with its password only until a token has been added. This ability is needed to log in to the IPA WebUI to add the token at the first place. However, password-only access should be limited to the token add facility.

Conclusion

I’m pretty amazed how well it works as this is a brand new feature for FreeIPA. The involved engineers made a brilliant job! I’m looking forward to see this feature in Redhat IPA/IdM somewhere in the future as 2FA is an often requested killer feature in enterprise environments.

Read more

Have fun! 🙂

Intercepting proxies and spacewalk-repo-sync

More and more companies are using intercepting proxies to scan for malware. Those malware scanners can be problematic due to added latency.

If you using spacewalk-repo-sync to synchronize external yum repositories to your custom software channels and experience the famous message [Errno 256] No more mirrors to try in your log files, then you need to configure spacewalk-repo-sync.

Unfortunately the documentation for that is a bit hidden in the man page. You need to create a directory and create a file.

mkdir /etc/rhn/spacewalk-repo-sync/

Create the configuration item:

echo "[main]" >> /etc/rhn/spacewalk-repo-sync/yum.conf
echo timeout=300 >> /etc/rhn/spacewalk-repo-sync/yum.conf

You need to experiment a bit with the value of the timeout setting, 5min should be good enough for most environments.

/etc/rhn/spacewalk-repo-sync/yum.conf has the same options like yum.conf, have a look for more information in the man page.

Have fun 🙂