Updating Fedora to version 23 – how to workaround some issues

November 8th, 2015

After upgrading two machines from Fedora 22 to 23 I stumbled upon some severe issues. Most of them are easy to solve.

This weekend I’ve found some time to upgrade my headless router and one of my workstations. Unfortunately is did not went that smooth like the past few upgrades.

No initrd created and grub config lacks initrd reference
This seems to be connected to the Plymouth issue as described here: Common F23 bugs. On my headless machine I only had “details” and “text” themes installed, the result is that the machine can not access the root fs and the Kernel panics.

Solution: before upgrading, ensure you have the Plymouth theme “charge” active.

plymouth-set-default-theme charge && dracut -f && reboot

If you already upgraded and the machine fails to boot, select the Fedora 22 Kernel to boot from, create a new initrd and update the grub config as follows:

dracut /boot/initramfs-4.2.5-300.fc23.$(uname -i).img 4.2.5-300.fc23.$(uname -i)
grub2-mkconfig -o /boot/grub2/grub.cfg

Renamed network interfaces
On one machine I ended up having no network connectivity at all because both interfaces got a new name. I.e. “p135p1” was now known as “enp2s0”. If you do not use NetworkManager, just rename your ifcfg scripts in /etc/sysconfig/network-scripts/ and edit it them accordingly (DEVICE=new-interface-name). No clue how to manage that issue with NetworkManager, probably in a similar way.

No keyboard and mouse available
After the upgrading the machine I’m using as a workstation, keyboard and mouse have not been working anymore which makes the usage as a Workstation “a bit” problematic. The reason is a missing package due to a broken dependency.

dnf -y install  xorg-x11-drv-evdev
# Restart X11
systemctl isolate multi-user.target
systemctl isolate graphical.target

See also Bugzilla #1212833

KDM does not show any user
Since I do not have any local users (I’m using IPA for centralized identity management), its unclear if this is reason. I was not able to enter a username manually. I was also unable to track down the problem.

Solution: Switch to GDM:

system-switch-displaymanager GDM
# Restart X11
systemctl isolate multi-user.target
systemctl isolate graphical.target

KDE Plasma 5 garbled graphics
When I logged in to Plasma, the desktop was quite odd. All graphic stuff (the whole desktop) was garbled. No clue why, in a virtual machine this is working (somehow). Can not be the Nouveau driver because then also XFCE and Gnome would be affected as well.

Solution: Switch to XFCE (or Gnome if you like)

While the major part of the upgrade went really smooth, there are some issues which I did not expected to see. From my point of view the Plymouth issue and the keyboard/mouse issue should have been a blocker for the release. The rest of the issues I’ve stumbled on was probably just bad luck.

A word to KDE Plasma: Its available in the KDE spin since Fedora 22 but it is still not yet usable for daily work. I personally consider Plasma be a pre-alpha software. KDE repeats the same mistake they made when switching from KDE3 to KDE4. This will cause more users to switch away from KDE which is a petty.

Have fun 🙂

Identity Management und 2FA mit (Free)IPA @Chemnitzer Linuxtage 2015

April 9th, 2015

My first post in German, publishing the Slide Deck (in German) for my presentation about IPA and 2FA held at Chemnitzer Linux days 2015.

Mein erster Post in Deutsch. Hier die Slides von meinem Vortrag an den Chemnitzer Linux Tagen 2015.

IPA ist ein Identity Management System fĂŒr Linux und Unix, das stetig an Bedeutung gewinnt. Mittlerweile ist es des öfteren in Behörden, Banken, Versicherungen, aber auch in KMUs im Einsatz. IPA kann man sich als «Active Directory» fĂŒr Linux vorstellen. IPA verheiratet LDAP und Kerberos zu einem Opensource Produkt das leicht zu installieren und zu unterhalten ist. Mit IPA kann dank Kerberos Single-Sign-On realiert werden (Authentifizierung). RegelsĂ€tze legen fest, welche Benutzer von welchen Benutzergruppen auf welche Services und Hosts zugreifen dĂŒrfen.

Seit einiger Zeit lassen sich mit IPA auch sehr einfach 2FA-Lösungen (Zwei-Faktor-Autentifizierung) realisieren, um die Sicherheit weiter zu erhöhen.

Das Slide Deck gibt es hier:

Slides vom Vortrag

Die Slides habe ich ĂŒbringens bei einem Spontanvortrag bei der Berliner Linux User Group am 2015-04-08 wiederverwendet. Aufgrund des Feedbacks wird in den nĂ€chsten Wochen ein ca. 4h Workshop an einem Samstag organisiert.

Ich hoffe es hat allen anwesenden Spass gemacht und konnte Euch etwas Wissen vermitteln. Feedback zu beiden AnlÀssen willkommen.

2FA with (Free) IPA. The good, the bad and the ugly

April 9th, 2015

Two factor authentication (2FA) is more and more emerging which is good to enhance security. Since the release of IPA4 it comes with 2FA included.

Over time I made a lot of experiments and experience I wanted to share with you. Its is easy to set up and maintain as long as you use it only for system authentication. If you are using such things as webmail, it fails. This post shows you the capabilities as they are of today. Almost all bad issues apply not only to Fee(IPA) but 2FA in general.

The good
All your systems are Fedora 21, RHEL 7.1 or Ubuntu 14.02 all is working fine as the included SSSD is new enough to handle 2FA. All kerberized services can be used with 2FA w/o logging in again during the validity of your Kerberos ticket. Very convenient, very secure.

3rd Party applications can use LDAP authentication (Depending on the usecase)

The bad
Systems with older distributions such as RHEL6.6 come with a SSSD version which is to outdated to handle kerberized 2FA at all. This will probably change soon.


  • Use LDAP authentication (See later on)
  • Use a Jump host with a recent Linux distribution

If you are logging in to your workstation with a local user, you can not grab a Kerberos ticket with kinit and use this ticket further on. (i.e for ssh logins on remote server, mail etc.)


  • Switch to a IPA managed user if your workstation is recent enough.
  • Use a Jump host with a recent Linux distribution
  • Wait until krb5-PAKE is in place, software is being developed, see http://k5wiki.kerberos.org/wiki/Projects/Improve_OTP_deployability and https://github.com/npmccallum/krb5-pake
    • The ugly

      Looks like most mobile applications such as the IMAP client in Android do not prompt for the password, they expect it configured. Needless to say that you can not reconfigure the password each time you want to check your emails with your phone.


      • 3rd party email app? One that prompts for the password if needed
      • Configure IPA to accepts password and 2FA which lets the user choose to either use the password only or 2FA. Needless to say that this makes 2FA less useful as people tend to be lazy
      • Turn off 2FA in IPA and use a Yubikey with a static password (spit password). This is not a real 2FA it is a single password split in two. Password change is a horror.
      • Accessing Webmail clients (I tested roundcube mail) causes headaches as well. They authenticate the users with IMAP and use this credentials to access the mail storage. As the second factor is a one time password (OTP) this will result in failure to retrieve mails after logging in.

        Workaround: Same as for mobile applications. I would appreciate if someone can point me to a webmail software which can handle this.

        Offline usage

        One sentence: Offline usage does not work because it can not work.


        • Create a local user and use a Yubikey and configure it with a static password (split password). This is not a real 2FA it is a single password split in two. Password change is a horror.
        • Install a IPA server on your Notebook 😉 This will scale up to 18 Notebooks (plus two replicas in the datacenter) but introduce a lot of other problems, so: Not seriously to be considered.

        LDAP Authentication as a Workaround
        Configure PAM/SSSD to use LDAP authentication for your users. IPA comes with a very nice feature called ipa-advise.

        [root@ipa1 ~]# ipa-advise config-redhat-nss-pam-ldapd
        # ----------------------------------------------------------------------
        # Instructions for configuring a system with nss-pam-ldapd as a IPA
        # client. This set of instructions is targeted for platforms that
        # include the authconfig utility, which are all Red Hat based platforms.
        # ----------------------------------------------------------------------
        # Schema Compatibility plugin has not been configured on this server. To
        # configure it, run "ipa-adtrust-install --enable-compat"
        # Install required packages via yum
        yum install -y wget openssl nss-pam-ldapd pam_ldap authconfig
        # NOTE: IPA certificate uses the SHA-256 hash function. SHA-256 was
        # introduced in RHEL5.2. Therefore, clients older than RHEL5.2 will not
        # be able to interoperate with IPA server 3.x.
        # Please note that this script assumes /etc/openldap/cacerts as the
        # default CA certificate location. If this value is different on your
        # system the script needs to be modified accordingly.
        # Download the CA certificate of the IPA server
        mkdir -p -m 755 /etc/openldap/cacerts
        wget http://ipa1.example.com/ipa/config/ca.crt -O /etc/openldap/cacerts/ipa.crt
        # Generate hashes for the openldap library
        command -v cacertdir_rehash
        if [ $? -ne 0 ] ; then
         wget "https://fedorahosted.org/authconfig/browser/cacertdir_rehash?format=txt" -O cacertdir_rehash ;
         chmod 755 ./cacertdir_rehash ;
         ./cacertdir_rehash /etc/openldap/cacerts/ ;
         cacertdir_rehash /etc/openldap/cacerts/ ;
        # Use the authconfig to configure nsswitch.conf and the PAM stack
        authconfig --updateall --enableldap --enableldapauth --ldapserver=ldap://ipa1.example.com --ldapbasedn=cn=compat,dc=example,dc=com
        [root@ipa1 ~]#

        The output actually reflects your environment, example.com will be replaced with your domain, its copy-paste ready. I love this feature 🙂 For other Linux systems, run ipa-advise without parameters to see which advises are available.

        2FA works well, convenient and secure in a datacenter and office environment. Notebooks are fine as well as long as there is a network connection available. The mobile world (Smartphones and Tablets) is not yet ready for 2FA. Some issues can be worked around (with some drawbacks) while others render 2FA not usable at all (offline usage).

        Hopefully there will be some smart solutions available for mobile usage soon, as mobile usage causes the most of the security headaches.

Migrating legacy servers to FreeIPA authentication using ID-views

April 6th, 2015

ID-Views are a new feature of FreeIPA4 which allows you to map UID/GID user/group names to another. This is a very handy solution when migrating legacy servers.

There are legacy servers in the field with a lot of history. They have been migrated from one operating system to another since the last decade(s). It is unfortunately also not uncommon on those legacy servers to find software with hardcoded UID/GID and/or user/group names. Along with an unknown number of scripts installed on such servers, its always problematic to migrate such systems when it comes to users and authentication. Another issue is that in the early years it was very common to have regular users with UID of >=500 while it is >=1000 as of today.

Unfortunately, almost nobody has the time to clean up the mess. Here is solution: ID-views. ID-Views can be applied to single hosts or group of hosts.

At the moment ID-Views are only working with newer SSSD versions as it is available with RHEL 7.1.

Creating a view

[root@ipa1 ~]# ipa idview-add --desc "Old servers with legacy users" oldservers
Added ID View "oldservers"
  ID View Name: oldservers
  Description: Old servers with legacy users
[root@ipa1 ~]# 

Override a group

[root@ipa1 ~]# ipa idoverridegroup-add --desc "Old group" --gid=500 --group-name=users oldservers users
Added Group ID override "users"
  Anchor to override: users
  Description: Old group
  Group name: users
  GID: 500
[root@ipa1 ~]#

Override a user
If you ommit the --login parameter (or any other) then the value in question is not overridden. Ususally you just override the numeric UID and/or GID.

[root@ipa1 ~]# ipa idoverrideuser-add --desc="John Doe is actually Hans Tester" --login=jdoe --uid=500 --gidnumber=500 --homedir=/home/jdoe --shell=/bin/csh oldservers tester
Added User ID override "tester"
  Anchor to override: tester
  Description: John Doe is actually Hans Tester
  User login: jdoe
  UID: 500
  GID: 500
  Home directory: /home/jdoe
  Login shell: /bin/csh
[root@ipa1 ~]# 

Apply the ID-View to a server

[root@ipa1 ~]# ipa idview-apply --hosts=legacy.example.com oldservers
Applied ID View "oldservers"
  hosts: legacy.example.com
Number of hosts the ID View was applied to: 1
[root@ipa1 ~]# 

To enable the view on the client side, clean the SSSD cache and restart the sssd service. Login to legacy.example.com.

[root@legacy ~]# sss_cache -E
[root@legacy ~]# systemctl restart sssd

You also need to change the PAM configuration to accept logins with UID &lt1000.

Now do some tests. Both users, “jdoe” and “tester” have UID 500.

[root@legacy ~]# getent passwd jdoe
jdoe:*:500:500:Hans Tester:/home/jdoe:/bin/csh
[root@legacy ~]# getent passwd tester
jdoe:*:500:500:Hans Tester:/home/jdoe:/bin/csh
[root@legacy ~]# 

On other servers, the “jdoe” login is unknown, and “tester” has the normal UID assigned by IPA

[root@ipa1 ~]# getent passwd jdoe
[root@ipa1 ~]# echo $?
[root@ipa1 ~]# getent passwd tester
tester:*:1225800004:1225800004:Hans Tester:/home/tester:/bin/bash
[root@ipa1 ~]# 

Please keep in mind that not cleaning up a messy system is just a workaround 🙂

Building a virtual CEPH storage cluster

April 3rd, 2015

cephThis post will guide you trough the procedure to build up a testbed on RHEL7 for a complete CEPH cluster. At the end you will have an admin server, one monitoring node and three storage nodes. CEPH is a object and block storage mostly used for virtual machine images and bulk BLOBS such as video- and other media. It is not intended to be used as a file storage (yet).

Machine set up
I’ve set up five virtual machines, one admin and monitoring server and five OSD servers.

  • ceph-admin.example.com
  • ceph-mon01.example.com
  • ceph-osd01.example.com
  • ceph-osd02.example.com
  • ceph-osd03.example.com

Each of them have a disk for the OS of 10GB, the OSD servers additional 3x10GB disks for the storage, in total 90GB for the stroage. Each virtual machine got 1GB RAM assigned, which is barley good enough for some first tests.

Configure your network
While it is recommended to have two separate networks, one public and one for cluster interconnect (heartbeat, replication etc). However, for this testbed only one network is used.

While it is recommended practice to have your servers configured using the Fully qualified hostname (FQHN) you must also configure the short hostname for CEPH.

Check if this is working as needed:

[root@ceph-admin ~]# hostname
[root@ceph-admin ~]# hostname -s
[root@ceph-admin ~]# 

To be able to resolve the short hostname, edit your /etc/resolv.conf and enter a domain search path

[root@ceph-admin ~]# cat /etc/resolv.conf 
search example.com
[root@ceph-admin ~]# 

Note: In my network, all is IPv6 enabled and I first tried to set CEPH up with all IPv6. I was unable to get it working properly with IPv6! Disable IPv6 before you start. Disclaimer: Maybe I made some mistakes.

You also need to keep time in sync. The usuage of NTP or chrony is best practice anyway.

Register and subscribe the machines and attach the repositories needed

This procedure needs to be repeated on every node, inlcuding the admin server and the monitoring node(s)

[root@ceph-admin ~]# subscription-manager register
[root@ceph-admin ~]# subscription-manager list --available > pools

Search the pools file for the Ceph subscription and attach the pool in question.

[root@ceph-admin ~]# subscription-manager attach --pool=<the-pool-id>

Disable all repositories and enable the needed ones

[root@ceph-admin ~]# subscription-manager repos --disable="*"
[root@ceph-admin ~]# subscription-manager repos --enable=rhel-7-server-rpms \
--enable=rhel-7-server-rhceph-1.2-calamari-rpms \
--enable=rhel-7-server-rhceph-1.2-installer-rpms \
--enable=rhel-7-server-rhceph-1.2-mon-rpms \

Set up a CEPH user
Of course, you should set a secure password instead of this example 😉

[root@ceph-admin ~]# useradd -d /home/ceph -m -p $(openssl passwd -1 <super-secret-password>) ceph

Creating the sudoers rule for the ceph user

[root@ceph-admin ~]# echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
[root@ceph-admin ~]# chmod 0440 /etc/sudoers.d/ceph

Setting up passwordless SSH logins. First create a ssh key for root. Do not set a pass phrase!

[root@ceph-admin ~]# ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa

And add the key to ~/.ssh/authorized_keys of the ceph user on the other nodes.

[root@ceph-admin ~]# ssh-copy-id ceph@ceph-mon01
[root@ceph-admin ~]# ssh-copy-id ceph@ceph-osd01
[root@ceph-admin ~]# ssh-copy-id ceph@ceph-osd02
[root@ceph-admin ~]# ssh-copy-id ceph@ceph-osd03

Configure your ssh configuration.

To make your life easier (not providing –username ceph) when you run ceph-deploy) set up the ssh client config file. This can be done for the user root in ~/.ssh/config or in /etc/ssh/ssh_config.

Host ceph-mon01
     Hostname ceph-mon01
     User ceph

Host ceph-osd01
     Hostname ceph-osd01
     User ceph

Host ceph-osd02
     Hostname ceph-osd02
     User ceph

Host ceph-osd03
     Hostname ceph-osd03
     User ceph

Set up the admin server

Go to https://access.redhat.com and download the ISO image. Copy the image to your admin server and mount it loop-back.

[root@ceph-admin ~]# mount rhceph-1.2.3-rhel-7-x86_64.iso /mnt -o loop

Copy the required product certificated to /etc/pki/product

[root@ceph-admin ~]# cp /mnt/RHCeph-Calamari-1.2-x86_64-c1e8ca3b6c57-285.pem /etc/pki/product/285.pem
[root@ceph-admin ~]# cp /mnt/RHCeph-Installer-1.2-x86_64-8ad6befe003d-281.pem /etc/pki/product/281.pem
[root@ceph-admin ~]# cp /mnt/RHCeph-MON-1.2-x86_64-d8afd76a547b-286.pem /etc/pki/product/286.pem
[root@ceph-admin ~]# cp /mnt/RHCeph-OSD-1.2-x86_64-25019bf09fe9-288.pem /etc/pki/product/288.pem

Install the setup files

[root@ceph-admin ~]# yum install /mnt/ice_setup-*.rpm

Set up a config directory:

[root@ceph-admin ~]# mkdir ~/ceph-config
[root@ceph-admin ~]# cd ~/ceph-config

and run the installer

[root@ceph-admin ~]# ice_setup -d /mnt

To initilize, run calamari-ctl

[root@ceph-admin ceph-config]# calamari-ctl initialize
[INFO] Loading configuration..
[INFO] Starting/enabling salt...
[INFO] Starting/enabling postgres...
[INFO] Initializing database...
[INFO] Initializing web interface...
[INFO] You will now be prompted for login details for the administrative user account.  This is the account you will use to log into the web interface once setup is complete.
Username (leave blank to use 'root'): 
Email address: luc@example.com
Password (again): 
Superuser created successfully.
[INFO] Starting/enabling services...
[INFO] Restarting services...
[INFO] Complete.
[root@ceph-admin ceph-config]#

Create the cluster

Ensure you are running the following command in the config directory! In this example it is ~/ceph-config.

[root@ceph-admin ceph-config]# ceph-deploy new ceph-mon01

Edit some settings in ceph.conf

osd_journal_size = 1000
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128

In production, the first value should be bigger, at least 10G. The number of placement groups is according the number of your cluster members, the OSD servers. For small clusters up to 5, 128 pgs are fine.

Install the CEPH software on the nodes.

[root@ceph-admin ceph-config]# ceph-deploy install ceph-admin ceph-mon01 ceph-osd01 ceph-osd02 ceph-osd03

Adding the initual monitor server

[root@ceph-admin ceph-config]# ceph-deploy mon create-initial

Connect the all nodes server to calamari:

[root@ceph-admin ceph-config]# ceph-deploy calamari connect ceph-mon01 ceph-osd01 ceph-osd02 ceph-osd03 ceph-admin

Make your admin server being an admin server

[root@ceph-admin ceph-config]# yum -y install ceph ceph-common
[root@ceph-admin ceph-config]# ceph-deploy admin ceph-mon01 ceph-osd01 ceph-osd02 ceph-osd03 ceph-admin

Purge and add your data disks:

[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdb
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdc
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdd
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdb
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdc
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdd
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdb
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdc
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd03:vdd

[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd01:vdb
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd01:vdc
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd01:vdd
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd02:vdb
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd02:vdc
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd02:vdd
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd03:vdb
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd03:vdc
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd03:vdd

You now can check the health of your cluster:

[root@ceph-admin ceph-config]# ceph health
[root@ceph-admin ceph-config]# 

Or with some more information:

[root@ceph-admin ceph-config]# ceph status
    cluster 117bf1bc-04fd-4ae1-8360-8982dd38d6f2
     health HEALTH_OK
     monmap e1: 1 mons at {ceph-mon01=}, election epoch 2, quorum 0 ceph-mon01
     osdmap e42: 9 osds: 9 up, 9 in
      pgmap v73: 192 pgs, 3 pools, 0 bytes data, 0 objects
            318 MB used, 82742 MB / 83060 MB avail
                 192 active+clean
[root@ceph-admin ceph-config]# 

Whats next?
A storage is worthless if not used. A follow-up post will guide you trough how to use CEPH as storage for libvirt.

Further reading