Leveraging Network-Bound Disk Encryption at Enterprise Scale

Tang and Clevis

Tang and Clevis

Network-Bound Disk Encryption (NBDE) adds scaling to LUKS by automated disk unlocking on system startup.

Why should I encrypt disks? If you dont want to see your corporate and private data leaked, you should do so as an additional security measure.

Use cases

There are basically two use cases for disk encryption. The first one is to prevent data leaks when a device gets stolen or lost (mobile computers, unsecured server rooms etc.). Theft of devices is usually not a threat for enterprise grade data centers with physical security.

Here comes the second use case for this enterprise grade data centers: At some point in time, disks will get disposed, either because of a defect or they get outdated technology wise. That means a data leak is possible at the end of the disks life cycle. A defect disk can not be wiped at all. For someone with deep pockets, there is still a chance to at least partially access the data. Wiping six TiB disks takes many hours just for overwriting them with zeros, not even with random data. An encrypted disk without a passphrase set can just simply get disposed without considering if it needs to be wiped or physically destroyed.

Note: Disk encryption does not help protecting you from a data theft by a person having access to the data, it also does not help against misbehaving software.

As you can imagine, it is a good idea to encrypt your storage. The standard for disk encryption in Linux is LUKS (Linux Unified Key Setup).

Adding Tang and Clevis for scaling

Unfortunately LUKS does not scale at all, because the passphrase must be entered manually on system startup, a no-go for data center operations. Tang and Clevis adds the scaling factor to the game.

Tang is the server component, Clevis and LUKS-meta the client component. The secret itself is stored on the client, the client asks the server for the data needed for the decryption of the key stored in the LUKS meta data. For more information on the crypto algorithms used, please see the Slide Deck “Tang and Clevis” by Fraser Tweedale

Availability and support

Tang and clevis have been added to RHEL 7.4 and are supported. The packages tang-nagios and clevis-udisk2 are in technical preview phase and are not supported. The packages are included in the base subscription.

It is also available for Fedora as well.

Set up the Tang servers

Setting up a Tang server is straight forward. For redundancy, please set up at least two Tang servers, a maximum of seven Tang Servers are supported by the client, which corresponds to the number of LUKS slots (eight) minus the one used for the initial passphrase.

[root@tang1 ~]# yum -y install tang
[root@tang1 ~]# systemctl enable --now tangd.socket
[root@tang1 ~]# jose jwk gen -i '{"alg":"ES512"}' -o /var/db/tang/new_sig.jwk
[root@tang1 ~]# jose jwk gen -i '{"alg":"ECMR"}' -o /var/db/tang/new_exc.jwk

Display the Thumbprint to be added to the Kickstart later on.

[root@tang1 ~]# jose jwk thp -i /var/db/tang/new_sig.jwk

Automated client setup during Kickstart

Be aware that you can run into problems when re-provisioning a system that contains old LUKS keys. You probably want to wipe them. In the following setup, all the slots are located on the second partition.

# Wipe LUKS keys on the second partition of disk vda
cryptsetup isLuks /dev/vda2  && dd if=/dev/zero of=/dev/vda2 bs=512 count=2097152

part /boot      --fstype ext2 --size=512 --ondisk=vda
part pv.0       --size=1 --grow --ondisk=vda --encrypted --passphrase=dummy-master-pass

volgroup vg_luksclient pv.0

logvol /        --name=lv_root    --vgname=vg_luksclient --size=4096
logvol /home    --name=lv_home    --vgname=vg_luksclient --size=512 --fsoption=nosuid,nodev
logvol /tmp     --name=lv_tmp    --vgname=vg_luksclient --size=512 --fsoption=nosuid,nodev,noexec
logvol /var     --name=lv_var    --vgname=vg_luksclient --size=2048 --fsoption=nosuid,nodev
logvol /var/log --name=lv_var_log --vgname=vg_luksclient --size=2048 --fsoption=nosuid,nodev
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_luksclient --size=4096

Be aware that the transfer of the Kickstart file will be done in clear text, that means that this dummy-master-pass is exposed. It should be automatically removed. You can add a master key via a secure way after the installation with Ansible, Puppet or simply manually via SSH.

Ensure you have the clevis-dracut package installed so that the init ramdisk will get created in the right way.


In the %post section of the Kickstart file, add the following to register your system to the Tang servers.

clevis bind luks -f -k- -d /dev/vda2 tang '{"url":"http://tang1.example.com","thp":"vkaGTzcBNEeF_X5KX-w9754Gl80"}' <<< "dummy-master-pass"
clevis bind luks -f -k- -d /dev/vda2 tang '{"url":"http://tang2.example.com","thp":"x_KcDG92bVP3SUL9KOzmzps4sZg"}' <<< "dummy-master-pass"

In case you want to remove the master password, put the following line into your %post section of the Kickstart file:

cryptsetup luksRemoveKey /dev/vda2 - <<<"dummy-master-pass"

Usage of a passphrase

There are pros and cons about doing so. On one hand, if all Tang servers are unavailable, there is not a slight chance to access the data if there is no master password set. On the other hand, a master password can be leaked and it should be changed from time to time which needs to be automated (i.e. with Ansible) to scale.

I personally tend to use a master password. Choose wisely depending on your specific use case if you set a master password or not.

Good to know

Be aware that the password prompt on system startup will always show up. It disappears automatically after a few seconds if a Tang server have been reached.


The following documents helps you further to get an idea about the Tang/Clevis setup:

A nice presentation from a conference is available here: https://www.usenix.org/conference/lisa16/conference-program/presentation/atkisson

Another more technical presentation is available here: http://redhat.slides.com/npmccallum/sad#/

Important commands

There are a few LUKS and clevis related commands you should know about.


Cryptsetup is used to handle the LUKS slots, adding and removal of passphrases. More information is available in man 8 cryptsetup


luksmeta gives you access to the meta data of LUKS. I.e. showing which slots are in use:

[root@luksclient ~]# luksmeta show -d /dev/vda2 
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
2   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
3   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty
[root@luksclient ~]#

The following command is reading the meta data and put the encrypted content to the file meta

luksmeta load -d /dev/vda2 -s 1  > meta

It looks like this:


You can decrypt it:

[root@luksclient ~]# clevis decrypt tang < meta 
OTQy6NGfqTjppwIrrM4cc15zr-sxy5PPmKExHul1m-pcMjEHjGdoN5uqD9vcEiuMM56VapPV_LedXYEkktYO-g[root@luksclient ~]#

OTQy6NGfqTjppwIrrM4cc15zr-sxy5PPmKExHul1m-pcMjEHjGdoN5uqD9vcEiuMM56VapPV_LedXYEkktYO-g is the cleartext passphrase returned. It actually can be used to type it in the console, I recommend a serial console where you can copy-paste 😉

If you run the same command again when both Tang servers are down, you will get an error:

[root@luksclient ~]# clevis decrypt tang < meta
Error communicating with the server!
[root@luksclient ~]#

As you can see, you don’t need to provide a Tang Server URL.


Lsblk is a nice little tool which shows the available storage in a tree. You can see the different layers of the storage subsystem.

[root@luksclient ~]# lsblk 
NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda                                           252:0    0   20G  0 disk  
├─vda1                                        252:1    0  512M  0 part  /boot
└─vda2                                        252:2    0 19.5G  0 part  
  └─luks-f0a70f08-b745-429f-ba8e-ec07e8953c3d 253:0    0 19.5G  0 crypt 
    ├─vg_luksclient-lv_root                   253:1    0    4G  0 lvm   /
    ├─vg_luksclient-lv_swap                   253:2    0    4G  0 lvm   [SWAP]
    ├─vg_luksclient-lv_var_log                253:3    0    2G  0 lvm   /var/log
    ├─vg_luksclient-lv_var                    253:4    0    2G  0 lvm   /var
    ├─vg_luksclient-lv_tmp                    253:5    0  512M  0 lvm   /tmp
    └─vg_luksclient-lv_home                   253:6    0  512M  0 lvm   /home
[root@luksclient ~]# 


If you want to play with JSON, install the package yajl.

With json_reformat you can minimize JSON and you are required to do so as clevis encrypt sss does not allow spaces, it fails.

Lets reformat this:

[root@luksclient ~]# echo '{"t": 1,"pins": {"tang": [{"url": "http://tang1.example.com"}, {"url": "http://tang2.example.com"}]}}'|json_reformat -m && echo ""
[root@localhost ~]# 

How to figure out to which servers the client is enrolled

I was curious how clevis figures out what Tang server to connect to. There is nothing written to the initrd, that means it must be stored somewhere in the LUKS metadata. It was taking me some time to figure out how it works.

Just decode the meta data to JSON:

 luksmeta load -d /dev/vda2 -s 1|jose b64 dec -i- |json_reformat 

Unfortunately the JSON seems to be invalid, at least json_reformat brings an error parse error: premature EOF. However, you will see the URL.

Test scenarios

I made a few tests with to figure out how Tang and Clevis works when something is going south.

Tang server(s) not available during system installatioon

If only one Tang server is available, installation work, server gets enrolled to only one Tang server. The server must be enrolled to the second Tang server manually after it came up again.

If both servers are down during installation, the installation finished successful, the temporary passphrase is still active as LUKS will deny removing the last passphrase available. Of course, the LUKS metadata is not available. You can enroll the servers manually after one or both servers come back online. Remember to remove the temporary passphrase afterwards.

Tang Server(s) not available during reboot

If one Tang server is not available, the other one is used, no impact.

If both servers are down, Plymouth asks for the LUKS passphrase. If you removed the the passphrase, you will not be able to boot the server. After starting one or both Tang servers, boot continues.


Tang and Clevis are both very young projects and not yet mature. I’ve figured out the following drawbacks:

Missing Registry

At the moment there is no way to report which servers are registered to what Tang server. This makes it hard to check from a central point if a server is really registered to two (or more) Tang servers to ensure smooth operation in the case of a failed Tang server.

This is particular true if one (or more) Tang server is down during install time of the client system. As a workaround, set up a monitoring script that checks if there are two active slots. I.e.

if [ $(luksmeta show -d /dev/vda2|grep " active"|grep -v empty|wc -l) -ne 2 ] || [ $? -eq 0 ]; then
        echo "Something is wrong with the LUKS metadata, please check"|mail -s "LUKS Metadata failure" monitoring@example.com


Logging of Tang requests is very basic at the moment, some improvement is needed here as well. Again, the documentation for the return codes is lacking


When using more than one Tang server, always that one defined in the first slot be be accessed. There is no round-robin or similar load-balancing method. This means that that the sequence of Tang Server must be shuffled on the client which involves some logic in the Kickstart file.

One Tang server should be able to handle more than 2k requests per second, so the problem only kicks in very large environments, where more than 2000 server are booting (or getting installed) at the same time.


Its a brand new project using completely new ideas and methods. At the moment not much experience is there, an issue that will be solved over time.


There is almost no documentation available which goes beyond a few lines to show how to set up the server and client. Whats missing is how to troubleshoot the environment. Another missing part is how to handle key rotation, its unclear for me if and what has to be done on the client.

Easy-to-read documentation is important, in particular for Tang and Clevis which is using some new style die-hard cryptographic mathematics.


Both, client and server have a very small footprint and are performing well. The idea of Tang and Clevis is brilliant and a first incarnation is ready to use. Due to the drawbacks mentioned above I think it is not yet ready for production and it will take a while until it is.

Due to the nature of the project, stability and reliability is a key point, that is why people should test it and provide feedback.

I would like to thank the involved engineers, cool stuff.

Have fun:-)

Blueborne – How to disable Bluetooth in Fedora

Yesterday 2017-09-13 Redhat released infomation about the mitigation of the Blueborne vulnerability in RHEL: https://access.redhat.com/security/vulnerabilities/blueborne.

For Fedora the new updates are probably still in the build queue and/or being QAed by the community. For a quick fix, you can disable Bluetooth similar than in RHEL:

Stopping Bluetooth related service

systemctl stop bluetooth.service
systemctl disable bluetooth.service
systemctl mask bluetooth.service

Disable the Kernel modules

echo "install bnep /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install bluetooth /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btusb /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btintel /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btrtl /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf
echo "install btbcm /bin/true" >> /etc/modprobe.d/disable-bluetooth.conf

Removing the Kernel Modules from a running System

  rmmod bnep
  rmmod btusb
  rmmod btintel
  rmmod btrtl
  rmmod btbcm
  rmmod bluetooth

Improve your bash shell working experience

This article shows some hints how to improve your bash shell working experience to reach higher productivity. Just simple shortcuts that are not so well known.

Using the History

The bash history is underestimated when it comes to usability. Here some nice stuff to do with the history.

Search the history

Every command is kept in the history. The simplest way to use the history is using the cursor-up/down keys. Most users are aware or [ctrl]-r. Usually you hit [ctrl]-r (r like reverse search) several times and miss the command, roll your eyes, hit [ctrl]-c and do it again. Why not using forward search with [ctrl]-s in such a case? Well, that suspends your terminal. It comes from the ancient times and is not needed anymore.

Turn off terminal suspension

echo "stty -ixon" >> /etc/profile

Now you can search the history back and forward by using [ctrl]-r and [ctrl]-s.

Using another command with the same last argument

When you i.e. do ls a file and decide to edit it, you don’t need to retype the whole file path or using the mouse to copy-paste it. Use the [Alt]-. (dot) combination. It inserts the last argument used. So after ls -la /tmp/file.txt you type vi [Alt].. Review and hit enter to execute.

You can also reuse other than the last arguments, but this is more complex and does not speed up things a lot, copy-paste with your mouse is usually faster in such a case.

Forgot to sudo?

When you want to cat i.e. /etc/sssd/sssd.conf you need root access. As a normal user, access is denied.

[luc@fedora ~]$ cat /etc/sssd/sssd.conf
cat: /etc/sssd/sssd.conf: Permission denied
[luc@fedora ~]$ sudo !!
sudo cat /etc/sssd/sssd.conf

The !! also called bash bang does the trick. It just repeats the same command as used before which all arguments. Be aware that the command is executed immediately.

Bash can copy-paste as well!

Copy-paste is not only available in graphical environments but in the bash shell as well.

If you need to type some different commands all with the same arguments, cut the stuff. Position the curser to the position on the line from where you want to copy and hit [ctrl]-k. When you want to paste, hit [ctrl]-y.

You may also achieve that using othercommand !*. Using !(bash bang) can be dangerous because the command will be executed immediately, the copy-paste method is more safe.

That also works with single words etc. basically everything where you cut or delete some stuff like [alt]-d, [ctrl]-w, [ctrl]-u

Using an editor for copy-paste from websites and word processors

There are a number of reasons why you don’t want to directly copy-paste to a shell. Sometimes the source content has not properly escaped line ends or its just garbage from word processors. You may want to review and edit appropriately before fire the command. There is a super lazy and convenient trick to do so.

The security usecase

Copy-Paste from a Website is a security nightmare. Copy-Paste the following two lines into an editor and you see what I mean.

Sample command
echo “Dont copy-paste”

Second sample

The HTML code used for that is:

Sample command<span style="font-size: 0; position: absolute; left: -100px; top: -100px"><br>:echo "Dont copy-paste"</span>
Second sample

Nice! Use an editor before pasting anything in a terminal, for the sake of security.

The word processor garbage usecase

Lot of documentation is written in word processors such as Libreoffice, MS-Office and others. They replace double hyphens to a single one and nasty stuff such as single quotes to backticks. Just for a thing called usability.

When copy-paste that stuff, you probably want to review and edit it first.

Set the EDITOR environment variable

If you are too lazy to fire up vim, you can set the EDITOR environment variable to an editor of your choice (vim, emacs, nano, whatever), system wide in /etc/profile or /etc/bashrc. A better idea is to put it in ~/.profile or ~/.bashrc.

echo "export EDITOR=vim" >>: ~./bashrc

Afterwards you can just hit [ctrl]-x-e and vim starts up. When save and exit vim, the command will be executed.

What are my Keybindings?

If you wonder what kind of shortcuts are defined in a shell, a lot are. use bind -p to show them.

Have fun 🙂

Upgrading Redhat Satellite 5.7 to 5.8

Couple of days ago, Redhat released its latest and last major upgrade for Satellite 5.x. Its a rather important upgrade, you are advised to upgrade soon.

This upgrade contains some major improvements like stated in an earlier article


I’m not responsible for any damage caused by the procedure provided here. Always create a backup before even thinking about upgrading a Satellite server.


As always when you plan to upgrade your Satellite server to the latest version, you need to do some preparations first.

Ensure you will have enough disk space free in /var/opt/rh. The upgrade to 5.8 will also install a new PostgreSQL version located at /var/opt/rh/rh-postgresql95/lib/pgsql. The new version will roughly use the same diskspace as the old version 9.2 in /opt/rh/postgresql92/root/var/lib/pgsql.

Download the ISO

Visit https://access.redhat.com/downloads/content/250/ver=5.8/rhel—6/5.8/x86_64/product-software and make sure you select 5.8 and the architecture fitting you system (x86_64 or S390)

Get a new Satellite Certificate Manifest

Satellite 5.8 switches from Certificates to Manifests like Satellite 6. You need a Manifest to get it activated. You can create it by your own at the Subscription Management Application site, ensure you attach enough subscriptions to your Satellite server(s).


Usually an upgrade runs smooth, but just in case… it is recommended practice to have a recent backup ready. If your Satellite is running on a virtual machine, power off, snapshot and power on to have a consistent backup ready. For physical systems, db-control and the choice.

Backup the rest of your Satellite:

Create a copy of your rhn configuration directory as we need some information from the old files after the upgrade.

rhnsat:~# cp -rp /etc/rhn/ /etc/rhn-$(date +"%F")

Update your OS and Satellite 5.7

First step is to update the operating system and the Satellite 5.7 and apply the latest database schema updates as well.

rhnsat:~# yum -y update && reboot

Update the database schema if needed

To update the database schema, run the following command. Ideally it looks as follows:

rhnsat:~# spacewalk-schema-upgrade 
Schema upgrade: [satellite-schema-] -> [satellite-schema-]
Your database schema already matches the schema package version [satellite-schema-].

Switch from RHN to Subscription Manager

It is important to ensure you switched from RHN to subscription manager before doing the upgrade. You can check if this is the case with:

rhnsat:~# subscription-manager repos --list-enabled
    Available Repositories in /etc/yum.repos.d/redhat.repo
Repo ID:   rhel-6-server-rpms
Repo Name: Red Hat Enterprise Linux 6 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/os
Enabled:   1

Repo ID:   rhel-6-server-satellite-5.7-rpms
Repo Name: Red Hat Satellite 5.7 (for RHEL 6 Server) (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/satellite/5.7/os
Enabled:   1


If this is not yet done, have a look at the knowledge base article located here: https://access.redhat.com/articles/2884191

Functionality Check with the old version 5.7

It is recommended to restart and check a software functionality before upgrading to be able to pinpoint problems if there are some.

rhnsat:~# rhn-satellite restart


Review the software channels in use and delete unused channels as this can free up quite some disk space and reduces the size of the database significantly.

rhnsat:~# spacewalk-remove-channel -c rhel-i386-rhev-agent-6-server
Deleting package metadata (20):
Removing:         ######################################## - complete

Delete old system snapshots which are not used anymore. The following example deletes all snapshots which are older than one month:

rhnsat:~#  sw-system-snapshot --delete --all --start-date 200001010000 --end-date $(date -d "-1 months" "+%Y%m%d0000") 

Check for old MD5 user passwords and certificates

Check if there are still some users with an md5 hashed password. The same applies to certificates.

rhnsat:~# spacewalk-report users-md5
rhnsat:~# spacewalk-report system-md5-certificates

If there are any, please have a look to https://access.redhat.com/documentation/en-us/red_hat_satellite/5.8/html/installation_guide/ch10s03


If not done yet, install or update the rhn-upgrade package which contains the instructions how to proceed.

rhnsat:~# yum -y install rhn-upgrade

The package contains not only SQL- and other useful scripts needed for the upgrade but also important documents to read. The are located in /etc/sysconfig/rhn/satellite-upgrade/doc.

For most users, the document satellite-upgrade-postgresql.txt applies.

Do not forget to read the updated product documentation as well:

Performing the upgrade

rhnsat:~# mount satellite-5.8-rhel-6-x86_64-dvd.iso /mnt  -o loop
rhnsat:~#  cd /mnt
rhnsat:/mnt# ./install.pl --upgrade
* Starting Red Hat Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHSM Registration.
** Registration: System is already registered with RHSM.  Not re-registering.
* RHSM Subscriptions.
** Subscriptions: Subscription providing 'Red Hat Satellite' already attached.
** Subscriptions: Subscription providing 'Red Hat Enterprise Linux Server' already attached.
** Subscriptions: Disabling all RHSM repositories (rhel-6-server-rpms, rhel-6-server-satellite-5.7-rpms).
** Subscriptions: All repositories disabled.
** Subscriptions: Enabling RHEL repository.
** Subscriptions: RHEL repository enabled.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Installing Satellite packages.
Warning: more packages were installed by yum than expected:
* Now running spacewalk-setup.
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
*** Upgrading embedded database.
** Database: Populating database.
** Database: Skipping database population.
* Configuring tomcat.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating Red Hat Satellite.
** Manifest not activated.
** Upgrade process requires the manifest to be activated after the schema is upgraded.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? 
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? 
This portion of the Red Hat Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.

Active and Updating the Satellite

Since the ISO image is always a bit outdated, you need to activate and update the Satellite after its installation.

Upgrading the Database schema

rhnsat:~# rhn-satellite stop
rhnsat:~# /etc/init.d/rh-postgresql95-postgresql start
rhnsat:~# spacewalk-schema-upgrade
rhnsat:~# rhn-satellite-activate --manifest=/root/manifest.zip --ignore-version-mismatch

Some more work to do

After the upgrade succeed there is some work work to do.

Initial Sync with CDN

Unfortunately cdn-sync does not inherit the history which channels have been synced previously with satellite-sync. You need to once sync each channel again. Only missing data will be downloaded.

rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listbasechannels); do cdn-sync -c $i; done
rhnsat:~# for i in $(spacecmd -u admin -p secret -q softwarechannel_listchildchannels); do cdn-sync -c $i; done

If you have custom channels, this will produce errors as the custom channels are not available in CDN. Just ignore them.

Rebuild the search index

rhnsat:~# service rhn-search cleanindex

Have fun 🙂