Posts Tagged ‘Reliability’

Identity Management with IPA Part I

Saturday, December 17th, 2011

Red Hat released RHEL 6.2 on December 6th. From my point of view, the greatest news in the release is that IPA (or now called Identity Management) is now fully supported and available in the RHEL 6 base channel without additional subscription costs.

Upstream project is freeIPA and is available trough the default Fedora repos.

About central Identity Management
IPA stands for Identification, Auditing, Policy. The focus in this article is on identification of users.

In the past, there have been a lot of solutions available to centrally manage users and its access to services. Just to name a few: LDAP, Kerberos, PAM, MS Active Directory, Novell Directory Server and countless others. All of those solutions have one in common: They are very powerful and very complex to set up and maintain. Because they are so complex, a lot of system administrators just do not use them and distribute SSH-keys, user credentials etc. by script without real central management, the nightmare of every security officer.

What is IPA?
The missing solution was a glue of LDAP and Kerberos which is easy to install and maintain, redundant and scalable from small office environments up to large enterprise installations. here it comes: IPA, which makes system administrators and security managers friends again.

IPA comes with a powerful CLI and a web interface for people that are afraid of a shell.

One of the cool stuff in IPA is its multi-master replication feature and automatic fail over facility. The clients are able to look up IPA servers with SRV DNS records, which are – of course – handled by IPA.

Lets do some stuff
One thing is just writing about how cool IPA is, but lets set up a high available centrally managed identity management system. This guide is written for RHEL 6.2 IPA-Servers and clients but should also work with freeIPA and Fedora 15 and later (Let me know if you have some issues).

Requirements
Requirements are straightforward:

  • 1Gbyte of RAM
  • approx. 6Gbyte of disk (including operating system)
  • NTP
  • DNS entries for all IPA servers (including PTR records)
  • Fully updated RHEL 6.2 GA
  • Firefox on the IPA servers if you want to use the web interface

NTP is very important since Kerberos is quite picky about synchronized system time. Ensure it is configured and running on all involved servers.

Assumptions

  • IP network is 192.168.100.0/24
  • Domain is example.com
  • Kerberos realm is EXAMPLE.COM
  • IPA-Server 1 is ipa1.example.com
  • IPA-Server 2 is ipa2.example.com
  • IPA-Client 1 is ipa-client1.example.com
  • IPA-Client 2 is ipa-client2.example.com
  • All passwords used are “somepassword” (needles to tell you to choose your own passwords
  • Main DNS is at 192.168.100.1
  • IPA-Clients are using ipa1.example.com and ipa2.example.com as there DNS servers.

Installation of the first IPA Server

yum -y install ipa-server bind-dyndb-ldap firefox xorg-x11-xauth

You are now ready to set up IPA. There are just a couple of questions, the non-default answers for this example are in red.

[root@ipa1 ~]# ipa-server-install --setup-dns --forwarder=192.168.100.1
The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)

To accept the default shown in brackets, press the Enter key.

Existing BIND configuration detected, overwrite? [no]: yes
Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
.
Example: master.example.com.


Server host name [ipa1.example.com]:

Warning: skipping DNS resolution of host ipa1.example.com
The domain name has been calculated based on the host name.

Please confirm the domain name [example.com]:

The IPA Master Server will be configured with
Hostname:    ipa1.example.com
IP address:  192.168.100.227
Domain name: example.com

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [EXAMPLE.COM]:
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password: somepassword
Password (confirm): somepassword

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password: somepassword
Password (confirm): somepassword

Do you want to configure the reverse zone? [yes]:
Please specify the reverse zone name [100.168.192.in-addr.arpa.]:
Using reverse zone 100.168.192.in-addr.arpa.

The following operations may take some minutes to complete.
Please wait until the prompt is returned.
Configuring ntpd
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
done configuring ntpd.
Configuring directory server for the CA: Estimated time 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
done configuring pkids.

Lot of output omitted

Configuring named:
  [1/9]: adding DNS container
  [2/9]: setting up our zone
  [3/9]: setting up reverse zone
  [4/9]: setting up our own record
  [5/9]: setting up kerberos principal
  [6/9]: setting up named.conf
  [7/9]: restarting named
  [8/9]: configuring named to start on boot
  [9/9]: changing resolv.conf to point to ourselves
done configuring named.
==============================================================================
Setup complete

Next steps:
        1. You must make sure these network ports are open:
                TCP Ports:
                  * 80, 443: HTTP/HTTPS
                  * 389, 636: LDAP/LDAPS
                  * 88, 464: kerberos
                  * 53: bind
                UDP Ports:
                  * 88, 464: kerberos
                  * 53: bind
                  * 123: ntp

        2. You can now obtain a kerberos ticket using the command: 'kinit admin'
           This ticket will allow you to use the IPA tools (e.g., ipa user-add)
           and the web user interface.

Be sure to back up the CA certificate stored in /root/cacert.p12
This file is required to create replicas. The password for this
file is the Directory Manager password
[root@ipa1 ~]#

You now need to get a Kerberos ticket:

[root@ipa1 ~]# kinit admin
Password for admin@EXAMPLE.COM:
[root@ipa1 ~]#

Fire up firefox and point it to https://ipa1.example.com and follow the link provided in the error message. You will see the instructions needed to use Kerberos as authentication method. When importing the cert into Firefox, REALLY check all three boxes!

Afterwards you are automatically logged in, if you got your Kerberos ticket before (kinit admin)

Setting up a Recplica
For now, we one IPA server. If it failes, no one can log in to any system anymore. This is of course unacceptable and needs to be changed. So lets set up a replica to add high availability to our central identity management system.

Log in to ipa1.example.com and fire up ipa-replica-prepare to collect the data needed for the replica.

Non-default answers are coloured red

[root@ipa1 ~]# ipa-replica-prepare ipa2.example.com

Directory Manager (existing master) password: somepassword

Preparing replica for ipa2.example.com from ipa1.example.com
Creating SSL certificate for the Directory Server
Creating SSL certificate for the dogtag Directory Server
Creating SSL certificate for the Web Server
Exporting RA certificate
Copying additional files
Finalizing configuration
Packaging replica information into /var/lib/ipa/replica-info-ipa2.example.com.gpg
[root@ipa1 ~]#

/var/lib/ipa/replica-info-ipa2.example.com.gpg keeps all the information needed to set up the replica. You need to copy it by i.e scp to ipa2.example.com.

Now log in to ipa2.example.com and fire up ipa-replica-install

[root@ipa2 ~]# ipa-replica-install --setup-dns --forwarder=192.168.100.1 replica-info-ipa2.example.com.gpg

Directory Manager (existing master) password: somepassword

Run connection check to master
Check connection from replica to remote master 'ipa1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: port 80 (80): OK
   HTTP Server: port 443(https) (443): OK

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
admin@EXAMPLE.COM password:

Execute check on remote master
Check connection from master to remote replica 'ipa2.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: port 80 (80): OK
   HTTP Server: port 443(https) (443): OK

Connection from master to replica is OK.

Connection check OK
Configuring ntpd
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
done configuring ntpd.
Configuring directory server: Estimated time 1 minute

Lot of output omitted

Using reverse zone 100.168.192.in-addr.arpa.
Configuring named:
  [1/8]: adding NS record to the zone
  [2/8]: setting up reverse zone
  [3/8]: setting up our own record
  [4/8]: setting up kerberos principal
  [5/8]: setting up named.conf
  [6/8]: restarting named
  [7/8]: configuring named to start on boot
  [8/8]: changing resolv.conf to point to ourselves
done configuring named.
[root@ipa2 ~]#

On ipa2, you need a Kerberos Ticket as well:

root@ipa2 ~]# kinit admin

Some adjustment
Unfortunately the default shell for new users is /bin/sh, which should probably be changed.

ipa config-mod --defaultshell=/bin/bash

Testing the replication
Log in to ipa1.example.com and add a new user:

ipa user-add tester1
ipa passwd tester1

You now can check if the user is really available on both servers by firing a ldapsearch command:

ldapsearch -x -b "dc=example, dc=com" uid=tester1

Compare the results of both servers. If they are the same, you have been successfully set up you two-node replicated high available IPA server.

What if ipa1.example.com is not available when I need to add a new user?
Simple answer: There is one way to find out….

Shut down ipa1.example.com
Log in to ipa2.example.com and add a new user:

root@ipa2 ~]# ipa user-add tester2

Start up ipa1.example.com again and run a ldapsearch again:

ldapsearch -x -b "dc=example, dc=com" uid=tester2

Set up a IPA-Client
Whats a centrally managed Identity Management server worth without a client? Nada! Lets set up a RHEL 6.2 server as a client:

[root@ipaclient1 ~]# yum  install ipa-client

After installation the setup program needs to be fired up. Non-default answers are coloured red

[root@ipaclient1 ~]# ipa-client-install -p admin
Discovery was successful!
Hostname: ipaclient1.example.com
Realm: EXAMPLE.COM
DNS Domain: example.com
IPA Server: ipa1.example.com
BaseDN: dc=example,dc=com


Continue to configure the system with these values? [no]: yes
Synchronizing time with KDC...
Password for admin@EXAMPLE.COM: somepassword

Enrolled in IPA realm EXAMPLE.COM
Created /etc/ipa/default.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm EXAMPLE.COM
Warning: Hostname (ipaclient1.example.com) not found in DNS
DNS server record set to: ipaclient1.example.com -> 192.168.100.253
SSSD enabled
NTP enabled
Client configuration complete.
[root@ipaclient1 ~]# 

Testing the login
Log in to your client, you will need to change your password first:

[luc@bond ~]$ ssh 192.168.100.253 -l tester1
tester1@192.168.100.253's password: 
Password expired. Change your password now.
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for user tester1.
Current Password: 
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.
Connection to 192.168.100.253 closed.
[luc@bond ~]$ ssh 192.168.100.253 -l tester1
tester1@192.168.100.253's password: 
Last login: Sat Dec 17 19:40:10 2011 from bond.home.delouw.ch
Could not chdir to home directory /home/tester1: No such file or directory
-bash-4.1$ 

In this case we do not have a home directory for the user tester1. NFS automount of home directories will be discussed in Part II oder III of this guide.

Now log out of ipaclient1.example.com and shut down ipa1.example.com to check if it is working when one IPA server failed. Needless to say that it is working… (okay, there is a delay of a few seconds)

Drawbacks
IPA is not that powerful like MS Active Directory or Novell Directory. There is no support (and most probably there will never be) for multiple and or custom LDAP schemata to keep it simple and easily maintainable, this actually makes the drawbacks into a feature . If you need such features like custom LDAP schemata, you may have a look to RHDS.

Conclusion
Never in the past of information technology is was easier to set up and maintain a centrally managed identity management system. In just a few minutes of work you will have a basic set up of a highly available fault tolerant and scalable identity management server.

Outlook to Part II of this guide
IPA does not only allow users to be authenticated, but also to restrict them to use particular services only an particular systems. Thanks to Kerberos, it also provides single-sign-on capabilities without providing a password.

As soon as I get some time I’ll write about the following topics:

  • Passwordless (and key-less) SSH logins
  • Kerberized web applications
  • Centralized sudo management

Having fun?
Yes definitively , I have fun with IPA, and as a Linux consultant I expect a lot of work waiting for me.

Implement a high available Cobbler provisioning system

Sunday, April 24th, 2011

A not so well known feature of Cobbler is its replication facility. It allows you to create a high available system provisioning system. The whole set up is straight forward.

Background
Today people tend to NOT backup systems, only data is being backed up. In the case of a system failure, they just re-provisioning the system and automatically configure it with configuration management tools such as Puppet, CFengine or RHN Satellite.

You not only need to have your configuration management system(s) to be redundant, you also need to replicate your Cobbler provisioning systems.

It even works in a set up where you have multiple datacenters for disaster recovery, having one or two Cobbler servers at each site.

Luckily, Cobbler comes with that feature out-of-the-box.

Assumptions

  • Both Cobbler servers are in the same network
  • Your master cobbler has a DNS-entry cobbler-master, your slave is cobbler-slave

Lets do it

Unfortunately, Cobbler replication, at the moment, does not work when SELinux is running in enforcing mode. You need to turn it off on both Cobbler servers.

setenforce 0

Hopefully this will change soon.

For a initial replication just run:

cobbler replicate --master=cobbler-master --sync-all

On the slave server. It is recommended to run this command nightly by a cronjob to ensure you have the latest RPM-packages of your distros in sync. Feel free to run this command manually at every time if you think it is needed to immediately sync the whole Cobbler system.

Update on-the-fly
You can use a Cobbler trigger script that automatically connects to the slave and triggers a replication with the master whenever a system is added or deleted.

I’m using the following script to do so:

#!/bin/bash
#
# This is a Cobbler trigger script that triggers a replication on the slave
# server. You need to create a private/public key-pair on the master and 
# add the public key to ~/.ssh/authorized_keys on the slave

SLAVE=cobbler-slave

ssh $SLAVE "cobbler replicate --master=cobbler-master --systems=* --prune"
ssh $SLAVE "cobbler sync"

Put this script in /var/lib/cobbler/triggers/add/system/post and create a symlink in /var/lib/cobbler/triggers/delete/system/post on your master server.

Security
Cobbler handles the file /etc/rsyncd.conf. So far so good. You need to be aware that EVERY host can rsync your distros, kickstarts, snippets etc unless you take some security measures with some firewall and/or other restrictions.

Possibly the simplest way is configuring xinetd.

A sample /etc/xinetd.d/rsync, where 192.0.2.2 is the IP of your slave, would be:

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
#       allows crc checksumming etc.
service rsync
{
        disable         = no
        only_from       = 192.0.2.2
        flags           = IPv6
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

DHCP redundancy
You also need to operate a dhcp server on both Cobbler servers. If you have both Cobbler servers on the same subnet, you may have a look how to configure ISC dhcpd. A good guide (untestet) is at http://www.madboa.com/geek/dhcp-failover/. The lazy ones just turn off dhcp on the slave and activate it as needed.

Conclusion
As I wrote before, Cobbler is just great. Not only “batteries are included”, high availability, disaster safety and other cool stuff is included as well and can be quickly implemented.

Having fun? I rarely needed to re-provision systems because they went foobar and hopefully I will never experience a disaster such a burned-down, exploded or whatever datacenter. Cobbler does not protect you from such disasters, it helps you to recover.

Ubuntu 10.04 LTS released

Sunday, May 2nd, 2010

End of April 2010, Ubuntu 10.04 was released. As always it is based on Debian’s Testing-Release. Canonical “stabilizes” the testing tree of Debian and adds its own look.

This time, Ubuntu radically changed its look. From my point of view it looks ugly, very ugly. Strange colors, low contrasts in menus, orange icons in Nautilus… window buttons on the left side… At the end of the day an usability-horror.

Under the hood Ubuntu is a very stable distribution with recent software. Ubuntu 10.04 is a LTS (Long Term Support) version and is thus suited as a enterprise server. Support for the server variant of Ubuntu is five years. Ubuntu is – like Debian – capable to upgrade to a new major release w/o service interruption.

Managebility

You “can” mirror Debian and Ubuntu repositories locally but it is difficult if you to not like to mirror all architectures available. Unfortunately there is (AFAIK) no software available such as Spacewalk/RHN Satellite to manage your servers.

The best method is to allow each single system installed to talk directly or via proxy to the mirror servers. This is a nightmare for firewall administrators.

To my knowledge there is no convenient way to install Ubuntu over the net. There are rumors that spacewalk and cobbler is going to get Debian/Ubuntu support at some time.

Reliability

Debian and thus Ubuntu has an evidence to be reliable. This also  seems to be true for the current release 10.04. The software came from Debians testing repository but was stabilized during months. Canonical (The sponsor of Ubuntu) has a reputation for its quality management. To use Ubuntu as a server operating system is sane.

Conclusion

As a desktop operating system I’ll avoid Ubuntu, since the usability is focused on dummy-users and not professional Linux users. For server usage you need to ask yourself about your needs. If you are operating Oracle DB’s or other commercial applications you probably want install Red Hat Enterprise Linux (RHEL). For a web server Ununtu is very well suited, even better than RHEL. In two years there will be another LTS variant available and you are free to upgrade online. Reliability is very good, manageability is poor, especially when used in larger companies.

In short: Ubuntu for web servers, RHEL/CentOS for other servers.

As always: Feedback is welcome…

Have fun!

Confused about write barriers on file systems…

Sunday, November 1st, 2009

As ext3 is already known as a very robust file system why is the default mount option still barrier=0? The problem is LVM and the device mapper. They  do not support barriers.

When mounting ext3 on a LV, the option barrier=1 it should be ignored and a warning written. So far so good. Trying this brings a lot of confusion. According to a Red Hat bugzilla entry one should get a warning, but no signs of that neither in /var/log/messages nor dmesg output. Even more confusing is the output of mount , it shows the LV is mounted with the barrier=1 option.

The conclusion is: Enable write barriers on physical disk-partitions brings a plus of reliability to your files system, on LVM setups it should better be disabled for the moment.

Is this fun? No…