Centrally manage SELinux user mapping with (Free)IPA

SELinux LogoSELinux allows to confine users with SELinux user mappings. This article covers some basics about the confinement of users and shows how to manage them in central way with the help of (Free)IPA. It will greatly enhance your systems security.

SELinux is available and enabled on all Red Hat based distributions such as RHEL, CentOS and Fedora. for the basics please have a look at article. Before proceeding with the examples in this article:

  • ensure your system is running in enforcing mode otherwise you will experience strange results with the sysadm_r role when logging in.
  • temporary enable root login via SSH or have access to the systems console, used for debugging

Helpful packages

It is recommended to install a few packages to manage SELinux.

# yum -y install libselinux-python policycoreutils-python policycoreutils-devel policycoreutils-newrole setools-console

The basics

By default, every user is mapped to the SELinux user unconfined_u as you can proof with semanage login -l

rhel7:~# semanage login -l

Login Name           SELinux User         MLS/MCS Range        Service

__default__          unconfined_u         s0-s0:c0.c1023       *
root                 unconfined_u         s0-s0:c0.c1023       *
system_u             system_u             s0-s0:c0.c1023       *

Overview about standard SELinux users and roles

The following SELinux users are predefined with the standard policy:

User Role Domain su/sudo Exec * X11 Networking
sysadm_u sysadm_r sysadm_t sudo, su yes yes yes
staff_u staff_r staff_t sudo yes yes yes
user_u user_r user_t yes yes yes
guest_u guest_r guest_t no no no
xguest_u xguest_r xguest_t no yes yes, http(s) only

* Execution of scripts and binaries in /home and /tmp

Role transition

This is important to understand. Which users or which roles are allowed to switch the role?

seinfo is your friend…

seinfo -xu
[..some output ommited...]
   staff_u
      default level: s0
      range: s0 - s0:c0.c1023
      roles:
         object_r
         staff_r
         sysadm_r
         system_r
         unconfined_r
[..some output ommited...]

This means that a Linux user mapped to staff_u can switch the role to sysadm_r, but not to guest_r.

Allowed target types

How do I figure out what is allowed to do for a certain role? Again, seinfo is your friend:

root@server ~]# seinfo -rsysadm_r -x      
   sysadm_r
      Dominated Roles:
         sysadm_r
      Types:
         aide_t
         alsa_home_t
         amanda_recover_t
         antivirus_home_t
         httpd_helper_t
         auth_home_t
         chkpwd_t
         pam_timestamp_t
         updpwd_t
         utempter_t
         bacula_admin_t
         ndc_t
         bootloader_t
[.. lots of output ommited ..]

Note: There must be no space between r and sysadm_r, the parameter is really rsysadm_r.

There you can see every target type the role sysadm_r is allowed to access, it is a lot. Of course this looks different for each role, have a look at them.

Implementation with IPA

Change the default SELinux usermap order

In this example we will map the sysadmins group to the sysadm_r role. The SELinux user for this role, sysadm_u is not defined in the IPA configuration. Lets change that.

ipa config-mod --ipaselinuxusermaporder='guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$sysadm_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023'

To ensure non-mapped users are confined, create a default mapping:

ipa config-mod --ipaselinuxusermapdefault='user_u:s0'

Create users, groups and add members

echo Welcome123|ipa user-add --first=Luc --last="de Louw" --password luc
echo Welcome123|ipa user-add --first=Luc --last="de Louw" --password ldelouw
echo Welcome123|ipa user-add --first=Joe --last=Doe --password jdoe
echo Welcome123|ipa user-add --first=Guest --last=User --password guest

ipa group-add sysadmins
ipa group-add-member sysadmins --users=luc

ipa group-add staff
ipa group-add-member staff --users=ldelouw

ipa group-add users
ipa group-add-member users --users=jdoe

ipa group-add guests
ipa group-add-member guests --users=guest

The password of the users will immediately expire and need to be changed on the first login. Please log in with every user and change the password before proceeding.

Adding some HBAC rules

In this example I’ll make user of a host group testservers. Please create that and add one or more of your hosts to that group first.

ipa hbacrule-add sysadmins-allhosts --hostcat=all --servicecat=all
ipa hbacrule-add-user --groups=sysadmins sysadmins-allhosts

ipa hbacrule-add staff-testservers --servicecat=all
ipa hbacrule-add-user --groups=staff staff-testservers
ipa hbacrule-add-host --hostgroup=testservers staff-testservers

ipa hbacrule-add users-testservers --servicecat=all
ipa hbacrule-add-user --groups=users users-testservers
ipa hbacrule-add-host --hostgroup=testservers users-testservers

ipa hbacrule-add guests-testservers --servicecat=all
ipa hbacrule-add-user --groups=guests guests-testservers
ipa hbacrule-add-host --hostgroup=testservers guests-testservers

This creates four HBAC rules. The first allows everyone in the sysadmins group to log in to all hosts, the others restrict its users to the hostgroup testservers.

Create SELinux maps

ipa selinuxusermap-add --selinuxuser='sysadm_u:s0-s0:c0.c1023' --hbacrule=sysadmins-allhosts sysadmins
ipa selinuxusermap-add --selinuxuser='staff_u:s0-s0:c0.c1023' --hbacrule=staff-testservers staff
ipa selinuxusermap-add --selinuxuser='user_u:s0' --hbacrule=users-testservers users
ipa selinuxusermap-add --selinuxuser='guest_u:s0' --hbacrule=guests-testservers guests

In this example, I use HBAC rules for the mapping. This is recommended practice because the HBAC rules are the central point to manage user access rules.

However, you also can assign users (and groups of them), hosts (and groups of them) to a SELinux map, i.e. with ipa selinuxusermap-add-host –hostgroups=testservers semapname and ipa selinuxusermap-add-user –groups=users semapname. For single user or hosts it is the parameter –hosts or –users.

Sudo

Note that roles like user_r and lower can not do sudo to other users (incl. root). guest_r does even not allow to use the network. Please have a look at the table above.

Lets create the sudoers rules for the groups staff and sysadmins.

ipa sudocmd-add "/bin/bash"
ipa sudorule-add --hostcat=all --runasusercat=all --runasgroupcat=all sysadmins-can-sudo-i
ipa sudorule-add-user --group=sysadmins sysadmins-can-sudo-i
ipa sudorule-add-allow-command --sudocmds=/bin/bash sysadmins-can-sudo-i
ipa sudorule-add-option --sudooption='!authenticate' sysadmins-can-sudo-i
ipa sudorule-add-option --sudooption='type=sysadm_t' sysadmins-can-sudo-i
ipa sudorule-add-option --sudooption='role=sysadm_r' sysadmins-can-sudo-i

ipa sudorule-add --runasusercat=all --runasgroupcat=all staff-can-sudo-i
ipa sudorule-add-user --group=staff staff-can-sudo-i
ipa sudorule-add-host --hostgroups=testservers staff-can-sudo-i
ipa sudorule-add-allow-command --sudocmds=/bin/bash staff-can-sudo-i
ipa sudorule-add-option --sudooption='!authenticate' staff-can-sudo-i
ipa sudorule-add-option --sudooption='type=sysadm_t' staff-can-sudo-i
ipa sudorule-add-option --sudooption='role=sysadm_r' staff-can-sudo-i

Switching roles

Switching roles can be done in three different ways:

  • Using the newrole command when already logged in with the default role
  • Provide the role when logging in with ssh
  • Using sudo, please see above

If you are already logged in, you can switch the role with the newrole command.

[ldelouw@server ~]$ newrole -r sysadm_r

The output if id -Z shows that you are now mapped to the same SELinux user staff_u but in the role of sysadm_r

staff_u:sysadm_r:sysadm_t:s0-s0:c0.c1023

Most of the time you will know what you gonna do on a target system. You can provide the role as a parameter to ssh:

[luc@client ~]$ ssh ldelouw/unconfined_r@server.example.com
ldelouw@server.example.com's password:
Last login: Sun Jun 24 11:29:54 2018 from client.example.com
Kickstarted on 2018-01-16
[ldelouw@server ~]$ id -Z
staff_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[ldelouw@server ~]$

This example allows you to use the unconfined_r role instead of the default defined in the SELinux user map (staff_r).

Be aware that this is by default not possible with the sysadm_r role, as logging in with that role is turned off.

Troubleshooting

sysadm_u can not log in

By default a user with the role of sysadm_r is not allowed to log in via ssh, only console logins are allowed.

To change this for testing, set the following boolean:

# setsebool ssh_sysadm_login on

Use the -P parameter to make the change persistent:

# setsebool -P ssh_sysadm_login on

User mapped to sysadm_u can login but is not supposed to do so

If you can log in into a host as a user mapped to sysadm_u, and id shows the following obviously wrong SELinux user role and domain:

[luc@server ~]$ id
uid=594600001(luc) gid=594600001(luc) groups=594600001(luc),594600005(sysadmins)  context=system_u:system_r:unconfined_t:s0-s0:c0.c1023

Then probably the ssh_sysadm_login boolean is set to false and your system runs in permissive mode.

Sudo to root fails with permission denied to .bash_profile

[ldelouw@server ~]$ sudo -i
-bash: /root/.bash_profile: Permission denied

You dont have the role sysadm_r, this is why access to files owned by root are denied. Please have a look at the sudo configuration as described in this article.

Sudo trows errors

[ldelouw@server ~]$ sudo -i
sudo: sudoRole sudo_root_admins: unknown defaults entry "TYPE"
sudo: sudoRole sudo_root_admins: unknown defaults entry "ROLE"
-bash: /root/.bash_profile: Permission denied

There is a difference between using traditional /etc/sudoers files and IPA. The man sudoers reads SELinux_Spec ::= (‘ROLE=role’ | ‘TYPE=type’) which is correct for file based configuration but wrong for IPA.

All sudoers options defined in IPA must be lowercase to get recognized in the correct way.

Changes in IPA are not working on the clients

This is can be a caching problem.

sss_cache -E

This wipes everything in the sssd cache. Sometimes sudoers will still not be working, it is safe to remove the whole SSSD database and restart sssd.

rm -rf /var/lib/sss/db/*
systemctl restart sssd

Read further

There are tons of documentation available. Lets list the most notable.

Getting help

If you run into problems, there are different sources for getting help

Using modern Protocols like HTTP/2 and QUIC

First there was HTTP, then HTTP/2 and now HTTP/2 over the QUIC protocol. Lets have a look at the available HTTP Clients and Servers that support HTTP/2 and the experimental QUIC protocol.

Introduction

The Hypertext Transfer Protocol (HTTP) was invented in 1991. Up to 2015 then there was only little to no evolution. In 2015 the HTTP/2 protocol was defined as a standard. HTTP/2 is much more efficient that its ancestors.

It features multiplexing, stream prioritization, binary transmission and much more. Its a huge step forward.

Nevertheless, there is a need for something more efficient. HTTP/2 is using TCP (Transmission Control Protocol) which was created in the early days of the Internet to have a reliable connection over unreliable networks. Today’s networks are much more reliable which allows the usage of the unreliable but very efficient UDP (User Datagram Protocol) to transmit data. As a consequence, QUIC was born. It is using UDP instead of TCP.

QUIC includes the crypto layer, so there is no need of a separate TLS layer. The goal is to use TLS 1.3 which is not ready as of writing this post.

Both, QUIC and TLS 1.3 are currently being defined as standards, the current state of the TLS Working group is publish here, the work of the QUIC working group is vailable here.

A good overview about QUIC can be found here.

Client Software

As of writing this post, all major Browsers are supporting HTTP/2 over TCP. When it comes to QUIC, there is little left. At the moment only Chrome and Opera are capable to access web sites with QUIC.

It is expected that this will change as soon as the standard is finalized.

Web sites

I’m not aware of any prominent Website using QUIC beside of google. HTTP/2 is used by a lot of prominent sites such as facebook, google and many others.

Server Software

The situation for HTTP/2 looks good, most webservers such as Apache HTTPD, NGINX etc. come with support for HTTP/2. Well, Apache does not work with the prefork MPM, that means you can not use mod_php with HTTP/2. You can make use of FastCGI but this means that Apache will be the slowest webserver available on the market. Better use NGINX.

If it comes to QUIC support, there is an experimental NGINX module available. Unfortunately it seems to be abandoned.

An option could be the commercial LiteSpeed Server.

From my point of view, the only usable Webserver for both, HTTP/2 and QUIC is Caddy. Its a relatively new open source project implementing a lot of new and experimental technologies. A nice feature is automatic HTTPS with Letsencrypt.

Caddy Webserver

Lets have a closer look to Caddy on Fedora 27. Its quite straight forward to install and configure.

Installation

[root@f27 ~]# dnf install caddy certbot

Configuration

cat > /etc/caddy/caddy.conf << EOF
:80 {
    gzip
    root /usr/share/caddy
}
EOF

Get a Letsencrypt Certficate

[root@f27 ~]# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
-------------------------------------------------------------------------------
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Plugins selected: Authenticator webroot, Installer None
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): f27.ldelouw.ch
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for f27.ldelouw.ch
Input the webroot for f27.ldelouw.ch: (Enter 'c' to cancel): /usr/share/caddy/
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/f27.ldelouw.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/f27.ldelouw.ch/privkey.pem
   Your cert will expire on 2018-05-31. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"

Configure TLS

cat >> /etc/caddy/caddy.conf << EOF
:443 {
    gzip
    root /usr/share/caddy
    tls /etc/letsencrypt/live/f27.ldelouw.ch/fullchain.pem /etc/letsencrypt/live/f27.ldelouw.ch/privkey.pem
}

EOF

Give the caddy user access to the cert and key

[root@f27 ~]# setfacl -m u:caddy:r-X /etc/letsencrypt/live

Enable QUIC

[root@f27 ~]# cp /usr/lib/systemd/system/caddy.service /etc/systemd/system/
[root@f27 ~]# sed -i 's#ExecStart=/usr/bin/caddy -conf /etc/caddy/caddy.conf -log stdout -root /tmp -agree#ExecStart=/usr/bin/caddy -conf /etc/caddy/caddy.conf -log stdout -root /tmp -agree -quic#g' /etc/systemd/system/caddy.service
[root@f27 ~]# systemctl daemon-reload
[root@f27 ~]# systemctl restart caddy

Checking the Result

Enabling QUIC in your brower

Point Chrome to chrome://flags/ and search for QUIC. Enable it and relaunch the browser.

Open Chrome and a second tab with chrome://net-internals/#quicType the URL, i.e. https://f27.ldelouw.ch. Switch the to chrome tab and see the Result.

QUIC Screenhot

QUIC Screenhot

Leveraging Network-Bound Disk Encryption at Enterprise Scale

Tang and Clevis

Tang and Clevis

Network-Bound Disk Encryption (NBDE) adds scaling to LUKS by automated disk unlocking on system startup.

Why should I encrypt disks? If you dont want to see your corporate and private data leaked, you should do so as an additional security measure.

Use cases

There are basically two use cases for disk encryption. The first one is to prevent data leaks when a device gets stolen or lost (mobile computers, unsecured server rooms etc.). Theft of devices is usually not a threat for enterprise grade data centers with physical security.

Here comes the second use case for this enterprise grade data centers: At some point in time, disks will get disposed, either because of a defect or they get outdated technology wise. That means a data leak is possible at the end of the disks life cycle. A defect disk can not be wiped at all. For someone with deep pockets, there is still a chance to at least partially access the data. Wiping six TiB disks takes many hours just for overwriting them with zeros, not even with random data. An encrypted disk without a passphrase set can just simply get disposed without considering if it needs to be wiped or physically destroyed.

Note: Disk encryption does not help protecting you from a data theft by a person having access to the data, it also does not help against misbehaving software.

As you can imagine, it is a good idea to encrypt your storage. The standard for disk encryption in Linux is LUKS (Linux Unified Key Setup).

Adding Tang and Clevis for scaling

Unfortunately LUKS does not scale at all, because the passphrase must be entered manually on system startup, a no-go for data center operations. Tang and Clevis adds the scaling factor to the game.

Tang is the server component, Clevis and LUKS-meta the client component. The secret itself is stored on the client, the client asks the server for the data needed for the decryption of the key stored in the LUKS meta data. For more information on the crypto algorithms used, please see the Slide Deck “Tang and Clevis” by Fraser Tweedale

Availability and support

Tang and clevis have been added to RHEL 7.4 and are supported. The packages tang-nagios and clevis-udisk2 are in technical preview phase and are not supported. The packages are included in the base subscription.

It is also available for Fedora as well.

Set up the Tang servers

Setting up a Tang server is straight forward. For redundancy, please set up at least two Tang servers, a maximum of seven Tang Servers are supported by the client, which corresponds to the number of LUKS slots (eight) minus the one used for the initial passphrase.

[root@tang1 ~]# yum -y install tang
[root@tang1 ~]# systemctl enable --now tangd.socket
[root@tang1 ~]# jose jwk gen -i '{"alg":"ES512"}' -o /var/db/tang/new_sig.jwk
[root@tang1 ~]# jose jwk gen -i '{"alg":"ECMR"}' -o /var/db/tang/new_exc.jwk

Display the Thumbprint to be added to the Kickstart later on.

[root@tang1 ~]# jose jwk thp -i /var/db/tang/new_sig.jwk

Automated client setup during Kickstart

Be aware that you can run into problems when re-provisioning a system that contains old LUKS keys. You probably want to wipe them. In the following setup, all the slots are located on the second partition.

# Wipe LUKS keys on the second partition of disk vda
%pre
cryptsetup isLuks /dev/vda2  && dd if=/dev/zero of=/dev/vda2 bs=512 count=2097152
%end

part /boot      --fstype ext2 --size=512 --ondisk=vda
part pv.0       --size=1 --grow --ondisk=vda --encrypted --passphrase=dummy-master-pass

volgroup vg_luksclient pv.0

logvol /        --name=lv_root    --vgname=vg_luksclient --size=4096
logvol /home    --name=lv_home    --vgname=vg_luksclient --size=512 --fsoption=nosuid,nodev
logvol /tmp     --name=lv_tmp    --vgname=vg_luksclient --size=512 --fsoption=nosuid,nodev,noexec
logvol /var     --name=lv_var    --vgname=vg_luksclient --size=2048 --fsoption=nosuid,nodev
logvol /var/log --name=lv_var_log --vgname=vg_luksclient --size=2048 --fsoption=nosuid,nodev
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_luksclient --size=4096

Be aware that the transfer of the Kickstart file will be done in clear text, that means that this dummy-master-pass is exposed. It should be automatically removed. You can add a master key via a secure way after the installation with Ansible, Puppet or simply manually via SSH.

Ensure you have the clevis-dracut package installed so that the init ramdisk will get created in the right way.

%packages
clevis-dracut
%end

In the %post section of the Kickstart file, add the following to register your system to the Tang servers.

%post
clevis bind luks -f -k- -d /dev/vda2 tang '{"url":"http://tang1.example.com","thp":"vkaGTzcBNEeF_X5KX-w9754Gl80"}' <<< "dummy-master-pass"
clevis bind luks -f -k- -d /dev/vda2 tang '{"url":"http://tang2.example.com","thp":"x_KcDG92bVP3SUL9KOzmzps4sZg"}' <<< "dummy-master-pass"
%end

In case you want to remove the master password, put the following line into your %post section of the Kickstart file:

%post
cryptsetup luksRemoveKey /dev/vda2 - <<<"dummy-master-pass"
%end

Usage of a passphrase

There are pros and cons about doing so. On one hand, if all Tang servers are unavailable, there is not a slight chance to access the data if there is no master password set. On the other hand, a master password can be leaked and it should be changed from time to time which needs to be automated (i.e. with Ansible) to scale.

I personally tend to use a master password. Choose wisely depending on your specific use case if you set a master password or not.

Good to know

Be aware that the password prompt on system startup will always show up. It disappears automatically after a few seconds if a Tang server have been reached.

Documentation

The following documents helps you further to get an idea about the Tang/Clevis setup:

A nice presentation from a conference is available here: https://www.usenix.org/conference/lisa16/conference-program/presentation/atkisson

Another more technical presentation is available here: http://redhat.slides.com/npmccallum/sad#/

Important commands

There are a few LUKS and clevis related commands you should know about.

cryptsetup

Cryptsetup is used to handle the LUKS slots, adding and removal of passphrases. More information is available in man 8 cryptsetup

luksmeta

luksmeta gives you access to the meta data of LUKS. I.e. showing which slots are in use:

[root@luksclient ~]# luksmeta show -d /dev/vda2 
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
2   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
3   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty
[root@luksclient ~]#

The following command is reading the meta data and put the encrypted content to the file meta

luksmeta load -d /dev/vda2 -s 1  > meta

It looks like this:

eyJhbGciOiJFQ0RILUVTIiwiY2xldmlzIjp7InBpbiI6InRhbmciLCJ0YW5nIjp7ImFkdiI6eyJrZXlzIjpbeyJhbGciOiJFUzUxMiIsImNydiI6IlAtNTIxIiwia2V5X29wcyI6WyJ2ZXJpZnkiXSwia3R5IjoiRUMiLCJ4IjoiQVdCeFZSYk9MOXBYNjhRU0lqSEZyNzVuNUVXdDZGblkySmNaNVgxX0s4MldaNW9kMUNQTUJwQ0dsS1ZFZ29LOFQwMERPazFsMHJRQ2kyOEg4SDBsVXlfaCIsInkiOiJBTzNLdmsyc2pqYVpSM3RrbW5KcVQyWGYtd1lnbXZSa0JqNUpmNFgzWmtHTDRHbTYtbE5qemhzVEdraEZLRmZUdnJLUElDTHBSQndCTnNXc0JuZUlVTEViIn0seyJhbGciOiJFQ01SIiwiY3J2IjoiUC01MjEiLCJrZXlfb3BzIjpbImRlcml2ZUtleSJdLCJrdHkiOiJFQyIsIngiOiJBTm05WmQwUDFyT1F6MXhhQVFNTzJxRjRua3ZHMVpKS2VHNkFaWjdPTEo1ejhKS000N0otMUhZWnkyZk0zT29ZQVdiUndQdnJ6aUt4MFJmNWh0QlkzNXBxIiwieSI6IkFTdVZYR3JRQ0c4R3dKTENXbWpVbC1jN0llUUh4TC01cFRGYTJaOU1ESnU4Ym9JZFo3WlNiZHBHZUFWMnhMTzlCTnlqbE5zSzB2ZWJrR3ZDcmU5bDl0aFYifSx7ImFsZyI6IkVTNTEyIiwiY3J2IjoiUC01MjEiLCJrZXlfb3BzIjpbInZlcmlmeSJdLCJrdHkiOiJFQyIsIngiOiJBT01Yc2JqcGd3MUVwMEdmSHd2NFRGTzFhWlNxZlY2NWlURVpWWDc4Y3M1SkI2dlBHaUZwd2RiZnpVWlpzR2FCZVludXdzTHJ0UTZTYm0zWHVTdGNHTFlFIiwieSI6IkFSZUZMckNHZnB6S2tzaTZvVXdLdEZjOW9IbHVHdDJjd3AwNmR1M3dEUGgta0t5N3RfTmZEU1JOSXRuWkZIbWs3eVlwSnkxYlpiRDRUTklTcXFIZDlDbDIifV19LCJ1cmwiOiJodHRwOi8vdGFuZzEuaG9tZS5kZWxvdXcuY2gifX0sImVuYyI6IkEyNTZHQ00iLCJlcGsiOnsiY3J2IjoiUC01MjEiLCJrdHkiOiJFQyIsIngiOiJBQnlMWjZmcWJKVVdzalZVc1ZjN0hwWlhLQ1BIZjJWenhyTExkODdvajBERnhGeTJRUTJHSXNEbFJ6OTg0cmtkNDJVQ3pDVy1VcGE4bG9nTl9BT0hsU0syIiwieSI6IkFBcHJaLUxFMUk3NUxWMXZtTHhkYUl0TmlETnpjUUVpLXJsR1FwVjFnT2IwWU5rbDFyWVgxdU45OE9WcHdiWUowTEpYYnYtRGZnSjU2RjBPMkNFczJOck4ifSwia2lkIjoicTAzQXd4VG5sU3lpQjRnelBTYTBfcXhsVzU4In0..Ws5k2fgQ26yN-mMv.1NwlYoyaUmF5X0jqGDcKO3HWn02StXotqnjZKaZtSUXioyW0-rc8HxH6HkkJTMQJk_EXr8ZXB4hmTXfUqAtRqgpEW4SdzJIw_AsGbJm5h_8lQLPIF4o.fbbNxK51MC14hX46Dgkj6Q

You can decrypt it:

[root@luksclient ~]# clevis decrypt tang < meta 
OTQy6NGfqTjppwIrrM4cc15zr-sxy5PPmKExHul1m-pcMjEHjGdoN5uqD9vcEiuMM56VapPV_LedXYEkktYO-g[root@luksclient ~]#

OTQy6NGfqTjppwIrrM4cc15zr-sxy5PPmKExHul1m-pcMjEHjGdoN5uqD9vcEiuMM56VapPV_LedXYEkktYO-g is the cleartext passphrase returned. It actually can be used to type it in the console, I recommend a serial console where you can copy-paste 😉

If you run the same command again when both Tang servers are down, you will get an error:

[root@luksclient ~]# clevis decrypt tang < meta
Error communicating with the server!
[root@luksclient ~]#

As you can see, you don’t need to provide a Tang Server URL.

lsblk

Lsblk is a nice little tool which shows the available storage in a tree. You can see the different layers of the storage subsystem.

[root@luksclient ~]# lsblk 
NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda                                           252:0    0   20G  0 disk  
├─vda1                                        252:1    0  512M  0 part  /boot
└─vda2                                        252:2    0 19.5G  0 part  
  └─luks-f0a70f08-b745-429f-ba8e-ec07e8953c3d 253:0    0 19.5G  0 crypt 
    ├─vg_luksclient-lv_root                   253:1    0    4G  0 lvm   /
    ├─vg_luksclient-lv_swap                   253:2    0    4G  0 lvm   [SWAP]
    ├─vg_luksclient-lv_var_log                253:3    0    2G  0 lvm   /var/log
    ├─vg_luksclient-lv_var                    253:4    0    2G  0 lvm   /var
    ├─vg_luksclient-lv_tmp                    253:5    0  512M  0 lvm   /tmp
    └─vg_luksclient-lv_home                   253:6    0  512M  0 lvm   /home
[root@luksclient ~]# 

json_reformat

If you want to play with JSON, install the package yajl.

With json_reformat you can minimize JSON and you are required to do so as clevis encrypt sss does not allow spaces, it fails.

Lets reformat this:

[root@luksclient ~]# echo '{"t": 1,"pins": {"tang": [{"url": "http://tang1.example.com"}, {"url": "http://tang2.example.com"}]}}'|json_reformat -m && echo ""
{"t":1,"pins":{"tang":[{"url":"http://tang1.example.com"},{"url":"http://tang2.example.com"}]}}
[root@localhost ~]# 

How to figure out to which servers the client is enrolled

I was curious how clevis figures out what Tang server to connect to. There is nothing written to the initrd, that means it must be stored somewhere in the LUKS metadata. It was taking me some time to figure out how it works.

Just decode the meta data to JSON:

 luksmeta load -d /dev/vda2 -s 1|jose b64 dec -i- |json_reformat 

Unfortunately the JSON seems to be invalid, at least json_reformat brings an error parse error: premature EOF. However, you will see the URL.

Test scenarios

I made a few tests with to figure out how Tang and Clevis works when something is going south.

Tang server(s) not available during system installatioon

If only one Tang server is available, installation work, server gets enrolled to only one Tang server. The server must be enrolled to the second Tang server manually after it came up again.

If both servers are down during installation, the installation finished successful, the temporary passphrase is still active as LUKS will deny removing the last passphrase available. Of course, the LUKS metadata is not available. You can enroll the servers manually after one or both servers come back online. Remember to remove the temporary passphrase afterwards.

Tang Server(s) not available during reboot

If one Tang server is not available, the other one is used, no impact.

If both servers are down, Plymouth asks for the LUKS passphrase. If you removed the the passphrase, you will not be able to boot the server. After starting one or both Tang servers, boot continues.

Drawbacks

Tang and Clevis are both very young projects and not yet mature. I’ve figured out the following drawbacks:

Missing Registry

At the moment there is no way to report which servers are registered to what Tang server. This makes it hard to check from a central point if a server is really registered to two (or more) Tang servers to ensure smooth operation in the case of a failed Tang server.

This is particular true if one (or more) Tang server is down during install time of the client system. As a workaround, set up a monitoring script that checks if there are two active slots. I.e.

if [ $(luksmeta show -d /dev/vda2|grep " active"|grep -v empty|wc -l) -ne 2 ] || [ $? -eq 0 ]; then
        echo "Something is wrong with the LUKS metadata, please check"|mail -s "LUKS Metadata failure" monitoring@example.com
fi

Logging

Logging of Tang requests is very basic at the moment, some improvement is needed here as well. Again, the documentation for the return codes is lacking

Scalability

When using more than one Tang server, always that one defined in the first slot be be accessed. There is no round-robin or similar load-balancing method. This means that that the sequence of Tang Server must be shuffled on the client which involves some logic in the Kickstart file.

One Tang server should be able to handle more than 2k requests per second, so the problem only kicks in very large environments, where more than 2000 server are booting (or getting installed) at the same time.

Maturity

Its a brand new project using completely new ideas and methods. At the moment not much experience is there, an issue that will be solved over time.

Documentation

There is almost no documentation available which goes beyond a few lines to show how to set up the server and client. Whats missing is how to troubleshoot the environment. Another missing part is how to handle key rotation, its unclear for me if and what has to be done on the client.

Easy-to-read documentation is important, in particular for Tang and Clevis which is using some new style die-hard cryptographic mathematics.

Conclusion

Both, client and server have a very small footprint and are performing well. The idea of Tang and Clevis is brilliant and a first incarnation is ready to use. Due to the drawbacks mentioned above I think it is not yet ready for production and it will take a while until it is.

Due to the nature of the project, stability and reliability is a key point, that is why people should test it and provide feedback.

I would like to thank the involved engineers, cool stuff.

Have fun:-)

Blueborne – How to disable Bluetooth in Fedora

Yesterday 2017-09-13 Redhat released infomation about the mitigation of the Blueborne vulnerability in RHEL: https://access.redhat.com/security/vulnerabilities/blueborne.

For Fedora the new updates are probably still in the build queue and/or being QAed by the community. For a quick fix, you can disable Bluetooth similar than in RHEL:

Stopping Bluetooth related service

systemctl stop bluetooth.service
systemctl disable bluetooth.service
systemctl mask bluetooth.service

Disable the Kernel modules

echo "install bnep /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install bluetooth /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btusb /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btintel /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btrtl /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btbcm /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf

Removing the Kernel Modules from a running System

  rmmod bnep
  rmmod btusb
  rmmod btintel
  rmmod btrtl
  rmmod btbcm
  rmmod bluetooth