Archive for the ‘Uncategorized’ Category

Updating Fedora to version 23 – how to workaround some issues

Sunday, November 8th, 2015

After upgrading two machines from Fedora 22 to 23 I stumbled upon some severe issues. Most of them are easy to solve.

This weekend I’ve found some time to upgrade my headless router and one of my workstations. Unfortunately is did not went that smooth like the past few upgrades.

No initrd created and grub config lacks initrd reference
This seems to be connected to the Plymouth issue as described here: Common F23 bugs. On my headless machine I only had “details” and “text” themes installed, the result is that the machine can not access the root fs and the Kernel panics.

Solution: before upgrading, ensure you have the Plymouth theme “charge” active.

plymouth-set-default-theme charge && dracut -f && reboot

If you already upgraded and the machine fails to boot, select the Fedora 22 Kernel to boot from, create a new initrd and update the grub config as follows:

dracut /boot/initramfs-4.2.5-300.fc23.$(uname -i).img 4.2.5-300.fc23.$(uname -i)
grub2-mkconfig -o /boot/grub2/grub.cfg

Renamed network interfaces
On one machine I ended up having no network connectivity at all because both interfaces got a new name. I.e. “p135p1” was now known as “enp2s0”. If you do not use NetworkManager, just rename your ifcfg scripts in /etc/sysconfig/network-scripts/ and edit it them accordingly (DEVICE=new-interface-name). No clue how to manage that issue with NetworkManager, probably in a similar way.

No keyboard and mouse available
After the upgrading the machine I’m using as a workstation, keyboard and mouse have not been working anymore which makes the usage as a Workstation “a bit” problematic. The reason is a missing package due to a broken dependency.

dnf -y install  xorg-x11-drv-evdev
# Restart X11
systemctl isolate
systemctl isolate

See also Bugzilla #1212833

KDM does not show any user
Since I do not have any local users (I’m using IPA for centralized identity management), its unclear if this is reason. I was not able to enter a username manually. I was also unable to track down the problem.

Solution: Switch to GDM:

system-switch-displaymanager GDM
# Restart X11
systemctl isolate
systemctl isolate

KDE Plasma 5 garbled graphics
When I logged in to Plasma, the desktop was quite odd. All graphic stuff (the whole desktop) was garbled. No clue why, in a virtual machine this is working (somehow). Can not be the Nouveau driver because then also XFCE and Gnome would be affected as well.

Solution: Switch to XFCE (or Gnome if you like)

While the major part of the upgrade went really smooth, there are some issues which I did not expected to see. From my point of view the Plymouth issue and the keyboard/mouse issue should have been a blocker for the release. The rest of the issues I’ve stumbled on was probably just bad luck.

A word to KDE Plasma: Its available in the KDE spin since Fedora 22 but it is still not yet usable for daily work. I personally consider Plasma be a pre-alpha software. KDE repeats the same mistake they made when switching from KDE3 to KDE4. This will cause more users to switch away from KDE which is a petty.

Have fun :-)

Identity Management und 2FA mit (Free)IPA @Chemnitzer Linuxtage 2015

Thursday, April 9th, 2015

My first post in German, publishing the Slide Deck (in German) for my presentation about IPA and 2FA held at Chemnitzer Linux days 2015.

Mein erster Post in Deutsch. Hier die Slides von meinem Vortrag an den Chemnitzer Linux Tagen 2015.

IPA ist ein Identity Management System für Linux und Unix, das stetig an Bedeutung gewinnt. Mittlerweile ist es des öfteren in Behörden, Banken, Versicherungen, aber auch in KMUs im Einsatz. IPA kann man sich als «Active Directory» für Linux vorstellen. IPA verheiratet LDAP und Kerberos zu einem Opensource Produkt das leicht zu installieren und zu unterhalten ist. Mit IPA kann dank Kerberos Single-Sign-On realiert werden (Authentifizierung). Regelsätze legen fest, welche Benutzer von welchen Benutzergruppen auf welche Services und Hosts zugreifen dürfen.

Seit einiger Zeit lassen sich mit IPA auch sehr einfach 2FA-Lösungen (Zwei-Faktor-Autentifizierung) realisieren, um die Sicherheit weiter zu erhöhen.

Das Slide Deck gibt es hier:

Slides vom Vortrag

Die Slides habe ich übringens bei einem Spontanvortrag bei der Berliner Linux User Group am 2015-04-08 wiederverwendet. Aufgrund des Feedbacks wird in den nächsten Wochen ein ca. 4h Workshop an einem Samstag organisiert.

Ich hoffe es hat allen anwesenden Spass gemacht und konnte Euch etwas Wissen vermitteln. Feedback zu beiden Anlässen willkommen.

2FA with (Free) IPA. The good, the bad and the ugly

Thursday, April 9th, 2015

Two factor authentication (2FA) is more and more emerging which is good to enhance security. Since the release of IPA4 it comes with 2FA included.

Over time I made a lot of experiments and experience I wanted to share with you. Its is easy to set up and maintain as long as you use it only for system authentication. If you are using such things as webmail, it fails. This post shows you the capabilities as they are of today. Almost all bad issues apply not only to Fee(IPA) but 2FA in general.

The good
All your systems are Fedora 21, RHEL 7.1 or Ubuntu 14.02 all is working fine as the included SSSD is new enough to handle 2FA. All kerberized services can be used with 2FA w/o logging in again during the validity of your Kerberos ticket. Very convenient, very secure.

3rd Party applications can use LDAP authentication (Depending on the usecase)

The bad
Systems with older distributions such as RHEL6.6 come with a SSSD version which is to outdated to handle kerberized 2FA at all. This will probably change soon.


  • Use LDAP authentication (See later on)
  • Use a Jump host with a recent Linux distribution

If you are logging in to your workstation with a local user, you can not grab a Kerberos ticket with kinit and use this ticket further on. (i.e for ssh logins on remote server, mail etc.)


  • Switch to a IPA managed user if your workstation is recent enough.
  • Use a Jump host with a recent Linux distribution
  • Wait until krb5-PAKE is in place, software is being developed, see and
    • The ugly

      Looks like most mobile applications such as the IMAP client in Android do not prompt for the password, they expect it configured. Needless to say that you can not reconfigure the password each time you want to check your emails with your phone.


      • 3rd party email app? One that prompts for the password if needed
      • Configure IPA to accepts password and 2FA which lets the user choose to either use the password only or 2FA. Needless to say that this makes 2FA less useful as people tend to be lazy
      • Turn off 2FA in IPA and use a Yubikey with a static password (spit password). This is not a real 2FA it is a single password split in two. Password change is a horror.
      • Accessing Webmail clients (I tested roundcube mail) causes headaches as well. They authenticate the users with IMAP and use this credentials to access the mail storage. As the second factor is a one time password (OTP) this will result in failure to retrieve mails after logging in.

        Workaround: Same as for mobile applications. I would appreciate if someone can point me to a webmail software which can handle this.

        Offline usage

        One sentence: Offline usage does not work because it can not work.


        • Create a local user and use a Yubikey and configure it with a static password (split password). This is not a real 2FA it is a single password split in two. Password change is a horror.
        • Install a IPA server on your Notebook 😉 This will scale up to 18 Notebooks (plus two replicas in the datacenter) but introduce a lot of other problems, so: Not seriously to be considered.

        LDAP Authentication as a Workaround
        Configure PAM/SSSD to use LDAP authentication for your users. IPA comes with a very nice feature called ipa-advise.

        [root@ipa1 ~]# ipa-advise config-redhat-nss-pam-ldapd
        # ----------------------------------------------------------------------
        # Instructions for configuring a system with nss-pam-ldapd as a IPA
        # client. This set of instructions is targeted for platforms that
        # include the authconfig utility, which are all Red Hat based platforms.
        # ----------------------------------------------------------------------
        # Schema Compatibility plugin has not been configured on this server. To
        # configure it, run "ipa-adtrust-install --enable-compat"
        # Install required packages via yum
        yum install -y wget openssl nss-pam-ldapd pam_ldap authconfig
        # NOTE: IPA certificate uses the SHA-256 hash function. SHA-256 was
        # introduced in RHEL5.2. Therefore, clients older than RHEL5.2 will not
        # be able to interoperate with IPA server 3.x.
        # Please note that this script assumes /etc/openldap/cacerts as the
        # default CA certificate location. If this value is different on your
        # system the script needs to be modified accordingly.
        # Download the CA certificate of the IPA server
        mkdir -p -m 755 /etc/openldap/cacerts
        wget -O /etc/openldap/cacerts/ipa.crt
        # Generate hashes for the openldap library
        command -v cacertdir_rehash
        if [ $? -ne 0 ] ; then
         wget "" -O cacertdir_rehash ;
         chmod 755 ./cacertdir_rehash ;
         ./cacertdir_rehash /etc/openldap/cacerts/ ;
         cacertdir_rehash /etc/openldap/cacerts/ ;
        # Use the authconfig to configure nsswitch.conf and the PAM stack
        authconfig --updateall --enableldap --enableldapauth --ldapserver=ldap:// --ldapbasedn=cn=compat,dc=example,dc=com
        [root@ipa1 ~]#

        The output actually reflects your environment, will be replaced with your domain, its copy-paste ready. I love this feature :-) For other Linux systems, run ipa-advise without parameters to see which advises are available.

        2FA works well, convenient and secure in a datacenter and office environment. Notebooks are fine as well as long as there is a network connection available. The mobile world (Smartphones and Tablets) is not yet ready for 2FA. Some issues can be worked around (with some drawbacks) while others render 2FA not usable at all (offline usage).

        Hopefully there will be some smart solutions available for mobile usage soon, as mobile usage causes the most of the security headaches.

Building a virtual CEPH storage cluster

Friday, April 3rd, 2015

cephThis post will guide you trough the procedure to build up a testbed on RHEL7 for a complete CEPH cluster. At the end you will have an admin server, one monitoring node and three storage nodes. CEPH is a object and block storage mostly used for virtual machine images and bulk BLOBS such as video- and other media. It is not intended to be used as a file storage (yet).

Machine set up
I’ve set up five virtual machines, one admin and monitoring server and five OSD servers.


Each of them have a disk for the OS of 10GB, the OSD servers additional 3x10GB disks for the storage, in total 90GB for the stroage. Each virtual machine got 1GB RAM assigned, which is barley good enough for some first tests.

Configure your network
While it is recommended to have two separate networks, one public and one for cluster interconnect (heartbeat, replication etc). However, for this testbed only one network is used.

While it is recommended practice to have your servers configured using the Fully qualified hostname (FQHN) you must also configure the short hostname for CEPH.

Check if this is working as needed:

[root@ceph-admin ~]# hostname
[root@ceph-admin ~]# hostname -s
[root@ceph-admin ~]# 

To be able to resolve the short hostname, edit your /etc/resolv.conf and enter a domain search path

[root@ceph-admin ~]# cat /etc/resolv.conf 
[root@ceph-admin ~]# 

Note: In my network, all is IPv6 enabled and I first tried to set CEPH up with all IPv6. I was unable to get it working properly with IPv6! Disable IPv6 before you start. Disclaimer: Maybe I made some mistakes.

You also need to keep time in sync. The usuage of NTP or chrony is best practice anyway.

Register and subscribe the machines and attach the repositories needed

This procedure needs to be repeated on every node, inlcuding the admin server and the monitoring node(s)

[root@ceph-admin ~]# subscription-manager register
[root@ceph-admin ~]# subscription-manager list --available > pools

Search the pools file for the Ceph subscription and attach the pool in question.

[root@ceph-admin ~]# subscription-manager attach --pool=<the-pool-id>

Disable all repositories and enable the needed ones

[root@ceph-admin ~]# subscription-manager repos --disable="*"
[root@ceph-admin ~]# subscription-manager repos --enable=rhel-7-server-rpms \
--enable=rhel-7-server-rhceph-1.2-calamari-rpms \
--enable=rhel-7-server-rhceph-1.2-installer-rpms \
--enable=rhel-7-server-rhceph-1.2-mon-rpms \

Set up a CEPH user
Of course, you should set a secure password instead of this example 😉

[root@ceph-admin ~]# useradd -d /home/ceph -m -p $(openssl passwd -1 <super-secret-password>) ceph

Creating the sudoers rule for the ceph user

[root@ceph-admin ~]# echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
[root@ceph-admin ~]# chmod 0440 /etc/sudoers.d/ceph

Setting up passwordless SSH logins. First create a ssh key for root. Do not set a pass phrase!

[root@ceph-admin ~]# ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa

And add the key to ~/.ssh/authorized_keys of the ceph user on the other nodes.

[root@ceph-admin ~]# ssh-copy-id ceph@ceph-mon01
[root@ceph-admin ~]# ssh-copy-id ceph@ceph-osd01
[root@ceph-admin ~]# ssh-copy-id ceph@ceph-osd02
[root@ceph-admin ~]# ssh-copy-id ceph@ceph-osd03

Configure your ssh configuration.

To make your life easier (not providing –username ceph) when you run ceph-deploy) set up the ssh client config file. This can be done for the user root in ~/.ssh/config or in /etc/ssh/ssh_config.

Host ceph-mon01
     Hostname ceph-mon01
     User ceph

Host ceph-osd01
     Hostname ceph-osd01
     User ceph

Host ceph-osd02
     Hostname ceph-osd02
     User ceph

Host ceph-osd03
     Hostname ceph-osd03
     User ceph

Set up the admin server

Go to and download the ISO image. Copy the image to your admin server and mount it loop-back.

[root@ceph-admin ~]# mount rhceph-1.2.3-rhel-7-x86_64.iso /mnt -o loop

Copy the required product certificated to /etc/pki/product

[root@ceph-admin ~]# cp /mnt/RHCeph-Calamari-1.2-x86_64-c1e8ca3b6c57-285.pem /etc/pki/product/285.pem
[root@ceph-admin ~]# cp /mnt/RHCeph-Installer-1.2-x86_64-8ad6befe003d-281.pem /etc/pki/product/281.pem
[root@ceph-admin ~]# cp /mnt/RHCeph-MON-1.2-x86_64-d8afd76a547b-286.pem /etc/pki/product/286.pem
[root@ceph-admin ~]# cp /mnt/RHCeph-OSD-1.2-x86_64-25019bf09fe9-288.pem /etc/pki/product/288.pem

Install the setup files

[root@ceph-admin ~]# yum install /mnt/ice_setup-*.rpm

Set up a config directory:

[root@ceph-admin ~]# mkdir ~/ceph-config
[root@ceph-admin ~]# cd ~/ceph-config

and run the installer

[root@ceph-admin ~]# ice_setup -d /mnt

To initilize, run calamari-ctl

[root@ceph-admin ceph-config]# calamari-ctl initialize
[INFO] Loading configuration..
[INFO] Starting/enabling salt...
[INFO] Starting/enabling postgres...
[INFO] Initializing database...
[INFO] Initializing web interface...
[INFO] You will now be prompted for login details for the administrative user account.  This is the account you will use to log into the web interface once setup is complete.
Username (leave blank to use 'root'): 
Email address:
Password (again): 
Superuser created successfully.
[INFO] Starting/enabling services...
[INFO] Restarting services...
[INFO] Complete.
[root@ceph-admin ceph-config]#

Create the cluster

Ensure you are running the following command in the config directory! In this example it is ~/ceph-config.

[root@ceph-admin ceph-config]# ceph-deploy new ceph-mon01

Edit some settings in ceph.conf

osd_journal_size = 1000
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128

In production, the first value should be bigger, at least 10G. The number of placement groups is according the number of your cluster members, the OSD servers. For small clusters up to 5, 128 pgs are fine.

Install the CEPH software on the nodes.

[root@ceph-admin ceph-config]# ceph-deploy install ceph-admin ceph-mon01 ceph-osd01 ceph-osd02 ceph-osd03

Adding the initual monitor server

[root@ceph-admin ceph-config]# ceph-deploy mon create-initial

Connect the all nodes server to calamari:

[root@ceph-admin ceph-config]# ceph-deploy calamari connect ceph-mon01 ceph-osd01 ceph-osd02 ceph-osd03 ceph-admin

Make your admin server being an admin server

[root@ceph-admin ceph-config]# yum -y install ceph ceph-common
[root@ceph-admin ceph-config]# ceph-deploy admin ceph-mon01 ceph-osd01 ceph-osd02 ceph-osd03 ceph-admin

Purge and add your data disks:

[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdb
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdc
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdd
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdb
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdc
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdd
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd01:vdb
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd02:vdc
[root@ceph-admin ceph-config]# ceph-deploy disk zap ceph-osd03:vdd

[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd01:vdb
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd01:vdc
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd01:vdd
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd02:vdb
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd02:vdc
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd02:vdd
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd03:vdb
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd03:vdc
[root@ceph-admin ceph-config]# ceph-deploy osd create ceph-osd03:vdd

You now can check the health of your cluster:

[root@ceph-admin ceph-config]# ceph health
[root@ceph-admin ceph-config]# 

Or with some more information:

[root@ceph-admin ceph-config]# ceph status
    cluster 117bf1bc-04fd-4ae1-8360-8982dd38d6f2
     health HEALTH_OK
     monmap e1: 1 mons at {ceph-mon01=}, election epoch 2, quorum 0 ceph-mon01
     osdmap e42: 9 osds: 9 up, 9 in
      pgmap v73: 192 pgs, 3 pools, 0 bytes data, 0 objects
            318 MB used, 82742 MB / 83060 MB avail
                 192 active+clean
[root@ceph-admin ceph-config]# 

Whats next?
A storage is worthless if not used. A follow-up post will guide you trough how to use CEPH as storage for libvirt.

Further reading

Using IPA to provide automount maps for NFSv4 home directories

Saturday, March 14th, 2015

Since the invention of NFSv4, automount NFS home directories is secure. Since the invention of IPA, its easier to set up and maintain. This article guides you trough the steps needed to set it up. The procedures have been tested on RHEL7.1 for the IPA servers, RHEL6.6 and 7.1 as clients but should work on Fedora and CentOS. Unfortunately it seems not to work (yet) for Debian Sid and Ununtu. [Update] Works in Ubuntu 14.04[/Update]


  • Your Domain is
  • Your Kerberos Realm is EXAMPLE.COM
  • The NFS server is
  • The exported home directories are on /exports/home
  • The client is
  • A few words about security and kerbrized NFS
    There are basically three different modes: krb5, krb5i and krb5p.

    • krb5 means that the server and client authenticate each other, traffic can be intercepted.
    • krb5i the same as krb5 but providing integrity. It verifies that the data has not been tampered with, but traffic still can be intercepted.
    • krb5p like the two above, plus privacy protection, all traffic is encrypted.

    Depending on the sensitivity of the data to be transferred krb5i or krb5p should be used. Also keep in mind that the higher the security the lower the throughput is.

    Work to do on one of the IPA replicas

    Add the NFS service principal for the server and client to Kerberos.

    [root@ipa1 ~]# ipa service-add nfs/
    [root@ipa1 ~]# ipa service-add nfs/

    Assume you are only using one location, you can use the default one.

    Add the auto.home map

    [root@ipa1 ~]# ipa automountmap-add default auto.home
    Added automount map "auto.home"
      Map: auto.home
    [root@ipa1 ~]# 

    And add the auto.home map to auto.master

    [root@ipa1 ~]# ipa automountkey-add default --key "/home" --info auto.home auto.master
    Added automount key "/home"
      Key: /home
      Mount information: auto.home
    [root@ipa1 ~]# 

    Finally add the key to the auto.home map

    [root@ipa1 ~]# ipa automountkey-add default --key "*" --info "-fstype=nfs4,rw,sec=krb5,soft,rsize=8192,wsize=8192" auto.home
    Added automount key "*"
      Key: *
      Mount information: -fstype=nfs4,rw,sec=krb5i,soft,rsize=8192,wsize=8192
    [root@ipa1 ~]# 

    Configure the NFS server
    Create a Kerberos Keytab for your NFS server

    [root@nfs ~]# kinit admin
    [root@nfs ~]# ipa-getkeytab -s -p nfs/ -k /etc/krb5.keytab

    Tell your NFS service to use NFSv4

    [root@nfs ~]# perl -npe 's/#SECURE_NFS="yes"/SECURE_NFS="yes"/g' -i /etc/sysconfig/nfs

    Create your NFS share and start the NFS server

    [root@nfs ~]# mkdir /exports/home
    [root@nfs ~]# echo "/exports/home  *(rw,sec=sys:krb5:krb5i:krb5p)" >> /etc/exports
    [root@nfs ~]# service nfs start
    [root@nfs ~]# chkconfig nfs on

    Configure your clients

    Get the Kerberos keytab

    [root@ipaclient1 ~]# ipa-getkeytab -s -p nfs/ -k /etc/krb5.keytab

    Finally you need to configure your client systems to map use of the automount maps provided by IPA

    [root@login ~]# ipa-client-automount --location=default
    Searching for IPA server...
    IPA server: DNS discovery
    Location: default
    Continue to configure the system with these values? [no]: yes
    Configured /etc/nsswitch.conf
    Configured /etc/sysconfig/nfs
    Configured /etc/idmapd.conf
    Started rpcidmapd
    Started rpcgssd
    Restarting sssd, waiting for it to become available.
    Started autofs
    [root@login ~]# 

    Strange problems you can run into

    If you run into troubles, enable debugging in the related daemons. In /etc/sysconfig/autofs, add a line LOGGING=debug.
    Add debug_level = 9 in the [autofs] stanza.

    If you have something like this in /var/log/messages

    lookup(file): failed to read included master map auto.master

    Then probably your nsswitch.conf does not point to sss. Ensure you have

    automount:  files sss

    In your nsswitch.conf. This should actually be configured by ipa-client-automount but it seems that it is not 100% reliable to do so.

    If you have something like this in /var/log/messages:

    Mar 14 20:02:37 ipaclient nfsidmap[3039]: nss_getpwnam: name '' does not map into domain 'localdomain'

    Then check your /etc/hosts file if all is correct. Also ensure that the short hostname is not in front of the FQHN. Another mistake can trigger the same error: DNS. Ensure you have a working DNS setup for both A (and/or AAAA) and PTR records.

    Read further
    There are plenty of docs available, there is a choice

    Have fun! :-)