Where the heck is RHEL6?

February 28th, 2010

Release cycle slowed down

In the past Red Hat has released a new version of its Red Hat Enterprise Linux (RHEL) roughly every two years. RHEL5 was released on march 2007. Compared to the past release cycle, RHEL6 is overdue since one year.

Official information

There is only little known about the upcoming features of RHEL6. On the Red Hat Summit 2009, there was a presentation held by Tim Burke which gives just some hints that RHEL6 is actually approaching, see http://www.redhat.com/f/pdf/summit/tburke_1050_rhel_roadmap.pdf. Quoting a note on the slide about RHEL6: Note: this information is high level planning projection and does not constitute formal product commitment.

My conclusion is that Red Hat seems to be unsure about the features planned for its upcoming Enterprise Product.

Another interesting quote from the same presentation is: RHEL6 feature previews – appearing in Fedora 11 & 12. Meanwhile, almost a half year later, Fedora 13 is approaching and still no sign of RHEL6, no schedule, no official feature list. Looking at the feature list if Fedora 13 https://fedoraproject.org/wiki/Releases/13/FeatureList, nothing special so far. It seems that the pace of development has been slowed down a bit to put more energy into stabilizing F11/F12 to RHEL6.

Inofficial information

When carefully watching git commit logs and bugzilla entries, there are some small traces of RHEL6.

There is almost no information leaking for the topic. The only valuable unofficial information is from bug #562766 which was reported by a Red Hat employee on 2010-02-08.  This bug states RHEL6 Alpha3!  Quoting a comment from the same employee: Upgrading rhel6.0 kernel to 2.6.32-14.el6 fixes the issue.

this brings me to a wild guess for a release schedule:

  • February 2010: Alpha3
  • March 2010: Beta1
  • April 2010: Beta2
  • May or June 2010: GA [Update: End of June/Early July seems to be more likely, since the Red Hat Summit will be held June 22-25 2010]

My wish list for RHEL6

  • Kernel based on version 2.6.33 instead of 2.6.32 as in Alpha3, since there are a lot of improvements when using RHEL as a VMware ESX guest.
  • Default installation with a smaller footprint
  • Cleanup of insane package dependencies
  • BusLogic drivers included as the vanilla Kernel ships it since years

The question remains

Where the heck is RHEL6? One reason could be that the focus on RHEL6 seems to be virtualization and system management. Since approximately two years, in this domain the pace of the development had increased a lot, maybe too much. Think about KVM, libvirt, virt-manager, o-virt. All of those projects are sponsored by Red Hat and included in F12. So one of the reason of the late release of RHEL6 can be problems in stabilizing those virtualization products to be enterprise-ready.

Why Red Hat makes its customers angry with late releases and no roadmap

First of all, RHEL products have a life-cycle of seven years. RHEL5 was released on march 2007. Assuming RHEL6 will be GA on May 2010. Add a few months before it is supported by ISVs such as SAP, Oracle etc. Customers can begin with deploying RHEL6 on lets say August 2010. Until then, RHEL5 has almost reached half of its life-cycle: 3 1/2 years. Means: A SAP system deployed on July 2010 is out of support some 3 years and nine months later. For an enterprise product this not acceptable! Red Hat should think about a life-cycle like “Next-Release plus five years“, this would make system deployment and company-internal life-cycle management easier.

Keeping its customers in the dark with no official roadmap at all is just bad behavior and indeed not customer friendly.

Set up a Red Hat Directory Server and Kerberos Part I

November 5th, 2009

Kerberos and LDAP are today’s way of single sign on. It is platform independent and supported by a wide range of applications.

Together with the Red Hat Directory Server (also available as CentOS Directory Server and 389 Directory Server from Fedora) you can build a neat identity management infrastructure.

Setting up the Directory Server
However there are some pitfalls when installing such a integrated solution. Installing redhat-ds is quite easy, just ensure you define your planned LDAP Namespace and default LDAP Suffix before running setup-ds-admin.pl. If you plan to setup a replica, run the script with the -k parameter: setup-ds-admin.pl -k. The servers configuration will be saved as /tmp/setup*.inf and can be used to setup the replica after changing the FullMachineName and ServerIdentifier.

In my example I used the DN “cn=Directory Manager. As base I used dc=ldap,dc=example,dc=com. This is the Internet Domain Suffix style of naming an LDAP space. The older X500 style should not be used anymore.

Have a look to man openldap.conf to see how to shorten your CLI entries such as ldapsearch -x.

Setting up Kerberos
After setting the right configurations in your /etc/krb5.conf (the sample content is self-explanatory) and its distribution, you need to initialize your key store database. This is to be done with kdb_util as follows:

[root@server]# kdb5_util create -r EXAMPLE.COM -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM',
master key name 'K/M@EXAMPLE.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:
[root@server]#

Keep in mind! Kerberos Realms are all uppercase to distinguish them from DNS names!

In the config file for the Key Distribution Center /var/kerberos/krb5kd/kdc.conf add the following in Realm Stanza: default_principal_flags =+ preauth. This will enhance security or your Kerberos Infrastructure. Also change the example Realm to what you are going to plan to use. In /var/kerberos/krb5kd/ kadm5.acl you can define the ACLs for e.g. admins or service desk employees etc. Also check the correctness of the Realm.

Feed the keystore

Now it is time to feed the database with the first principal: root. We also can create our first host principal at the same time.
Fire up kadmin.local. The kadmin.local app accesses directly the DB files on the server. Its should only be used on initial setup. Later on you will have kadmin which also works on the net, of course with Kerberos authentication.

[root@server ~]# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local:  addprinc root/admin
WARNING: no policy specified for root/admin@EXAMPLE.COM; defaulting to no policy
Enter password for principal "root/admin@EXAMPLE.COM":
Re-enter password for principal "root/admin@EXAMPLE.COM":
Principal "root/admin@EXAMPLE.COM" created.
kadmin.local:  addprinc -randkey host/server1.example.com
WARNING: no policy specified for host/server1.example.com@EXAMPLE.COM; defaulting to no policy
Principal "host/server1.example.com@EXAMPLE.COM" created.
kadmin.local:  q
[root@server ~]#

After starting the kadmin and kdc services you can access the admin server with the normal kamin tool.

service kadmin start
chkconfig kadmin on
service krb5kdc start
chkconfig krb5kdc on

Now we need to create a host principal for each to be kerberized host and store it in its keytab.

End of Part I

What comes in Part II?

  • LDAP Service Principal
  • Getting Kerberos and LDAP working together
  • Migrating users from /etc/passwd to LDAP
  • Playing with PAM

Have fun!

Skype UI gets opensource

November 3rd, 2009

An employee of Skype announced on Nov  02. 2009, Skype will publish the source code of their user interface. The planned release date is unknown, details are also unknown.

The proprietary Skype protocol remains closed, in future you will install a closed source library on your system and there are chances that you can choose between more then one user interface.

Not only different GUIs are possible, the Skype protocol can also be used as a back end for any other voice or chat application e.g Asterisk users can profit, as the now available Skype-extension is only for paying customers.

The future will be quite interesting, lets see what actually comes from Skype..

Further Reading:

http://share.skype.com/sites/linux/2009/11/skype_open_source.html

Have fun!

Managing CentOS with Spacewalk

November 2nd, 2009

spacewalk

Red Hat RHN Satellite

In 2003 Red Hat released its RHN Satellite server as a closed source management tool for RHEL and only for RHEL (okay, a legacy support for managing Solaris is available). The satellite is very useful tool for managing systems. Unfortunately it has a quite expensive price tag on it. According to the Red Hats RHN FAQ the bill is USD 13,500/year.

Additionally to the RHN satellite subscription fee you need at least a subscription for the management module which costs another USD 96/year and system. Assume you have a farm of 100 RHEL boxes it costs you about USD 23,100 every year. All prices are list prices.

RHN Satellite comes with an embedded Oracle Database which is from my point of view completely overkill and the driver for the high cost of the subscription. If Red Hat witches to PostgreSQL, I see some chances for lower price tags…

The alternative

In 2008 Red Hat open-sourced the RHN Satellite and named it  “Spacewalk” (a pretty cool name :-) ). Since then the development team already released six versions. The release cycle is quite short in a fast pace.

Release 0.6 is PostgreSQL “ready” whatever that means, I do not know how reliable it is. Full support for PostgreSQL is scheduled with version 1.0 due in Q1/2010. However, for a smaller farm of CentOS systems the free Oracle express edition is good enough.

Installation

The installation is straight forward: Just follow the Instructions how to set up Oracle XE and spacewalk. After the a little tricky installation and configuration of Oracle, you just need to add some yum repositories and run the set up script.

Uploading packages

There are basically to methods to put your RPMs to the Spacewalk server. Either trough yum repository synchronization or via rhnpush. The first method is great if you want to pull a CentOS repository, the second for additional own packages.

Bootstrapping clients

After you installed a new system, you need to add the rhn-client packages to your CentOS system. Unfortunately CentOS removes those packages from the upstream RHEL versions. I hope they will rethink about this.

If you set up your systems by provisioning with Spacewalk, you can automate this task. However I did not got the time yet to test the provisioning stuff with cobbler and kickstart files, its on the to-do list.

Manual bootstrapping works similar to the method like you bootstrap RHEL clients to a RHN Satellite.

Updates and Errata

At the moment this is the tricky part. How to get the upstream errata into Spacewalk? You can use Script that imports digests from the centos-annouce mailing list. Afterwards applying erratas to your systems works fine.

Integrate Spacewalk with other applications

Spacewalk, like RHN Satellite comes with a XML-RPC API which allows you to trigger actions from scrips or (web-)applications. I think about reporting an similar to-be-automated stuff.

Conclusion

If you do not need support from Red Hat, Spacewalk and CentOS can be an alternative for your server farm. If you like more up-to-date systems (e.g. for desktops), Fedora is also supported as a client platform.

Since Red Hat does not provide some kind of Test-Licenses of its products, Spacewalk and CentOS  are also a very nice playground for people managing RHEL systems on a daily base either to familiarize them self with the Satellite or doing some tests without bringing the production Satellite into danger.

Further readings

Unfortunately, beside of Red Hats documentations and the Spacewalk Wiki no books and other resources are available. Maybe I should start writing a book? ;-)

Spacewalk Wiki:
https://fedorahosted.org/spacewalk/

Red Hats Spacewalk homepage:
http://www.redhat.com/spacewalk/

Red Hat RHN Satellite documentations:
http://www.redhat.com/docs/manuals/satellite/

Have fun!

Confused about write barriers on file systems…

November 1st, 2009

As ext3 is already known as a very robust file system why is the default mount option still barrier=0? The problem is LVM and the device mapper. They  do not support barriers.

When mounting ext3 on a LV, the option barrier=1 it should be ignored and a warning written. So far so good. Trying this brings a lot of confusion. According to a Red Hat bugzilla entry one should get a warning, but no signs of that neither in /var/log/messages nor dmesg output. Even more confusing is the output of mount , it shows the LV is mounted with the barrier=1 option.

The conclusion is: Enable write barriers on physical disk-partitions brings a plus of reliability to your files system, on LVM setups it should better be disabled for the moment.

Is this fun? No…

Ready to upstart?

October 31st, 2009

upstart

It is time to replace the aged SysV init system with someting better

At the time when  SysV init (pronounced “System five”) appeared, hardware configurations have been quite static, no hot plug and similar fancy stuff.

SysV init is started after the kernel is loaded. The init process reads /etc/inittab and walks trough the runcontrol script and runlevels. This sequential walk-trough takes most of the time when booting a modern Unix system.

Upstart follows another approach: Starting daemons and services in parallel and event driven.  This will speed up the boot process beyond expectations.

A very nice feature of upstart is: All processes will be started in background, no more blocking of the boot process trough hanging run control scripts!

If a service unexpectedly dies, it will be respawned  automatically up to a configurable limit in times per period.

Upstart is event-driven, a event can be e.g. plugging in new hardware which ends up starting the needed service for it. There are also plans to replace cron and atd with upstart since this are basically time-triggered events. The developers also thinking about replacing the inetd, since a network connection can be considered as a event.

Transition

Since most of the software out there do not natively support upstart yet, transition methods are needed for a smooth transition from SysV init to upstart. Traditional SysV run control scrips are fully supported, even distributions slowly switch to the event/job model of upstart. E.g. one of the first distributions switched to upstart was Ubuntu 6.10, and now with Ubuntu 9.10 – three years later – they begin to ship its distribution with the first native upstart scripts.

Splitting Unix systems apart

Years ago there only have been two init systems: SysV init and BSD init, a sysadmin was comfortable to use them on whatever system. Now there are SysV init, Upstart from Ubuntu, lauchd from Apple, SMF (System Management Facility) from Sun Microsystems and possibly others. All of this SysV init replacements are working differently,  different commands, different architecture… This makes the job of a sysadmin not easier when managing a heterogeneous system landscape.

Linux distributions stay together

The good news: On the Linux side it looks like Upstart will be the future standard for system initialization, no balkanization of the Linux Landscape so far.

Linux Distribution with upstart

The following distributions are already shipping upstart:

  • Ubuntu
  • Debian
  • Fedora
  • Others?

Since Fedora 11 and 12 will be the upstream for the upcomming RHEL6 distribution it is most likely that RHEL6 comes with upstart. At openSUSE there are some discussions (see https://features.opensuse.org/305690 for details). Maybe there is a chance for openSUSE 11.3 and later on SLES12.

Further readings:

Upstart web site: http://upstart.ubuntu.com
Wikipedia article: http://en.wikipedia.org/wiki/Upstart

Have fun!


How are jornaling options affect performance of the ext3 filesystem

October 30th, 2009

The need for speed
Everyone looks for the optimum of speed in its servers. Todays servers have plenty of spare CPU power and RAM is dirty cheap. Todays common bottleneck is storage.

One way to solve the bottleneck is trowing money on it, the other smarter way is choosing the best matching file system and it options for the purpose of the server.

On Linux systems a bunch of file systems is available and ready to use. There are some high performance file systems such as SGI’s XFS or reiserfs. Both are known to be quite performant but having the drawback of being unreliable in case of a hard crash or a power loss.

As file systems are the key point for reliability, XFS and reiserfs are out of question. So whats left? ext3.

Problem
Ext3 is not known for its high performance, it is rather slow compared to xfs, especially if you handle with of lot of files such as on a web server.

Solution
Choose the right options for journaling of ext3. You have the choice of three different journaling options, data=writeback which means written data is first written to RAM and later on disk. This is the most performant option, but with the greatest risk of loosing data in case of a crash or power loss. Before choosing this option use xfs, it is more performant at the same risk.

The compromise is data=ordered Lets quote the man page: –All data is forced directly out to the main file system prior to its metadata being committed to the journal — At the end of the day this means data loss in hardly happening but not impossible. This option offers a balance between write speed and reliability.

Whats about the third option data=journal? It means that one would think if all data is written first in the journal and then to its final destination on disk, I/O performance gets decreased.

In theory, data=writeback is the fastest and data=journal the slowest option. Belief it or not: data=journal is in many cases the fastest option, especially in mostly-read applications when you concurrently read lots of small files (such as on web servers).

At the end of the day: With the ext3 file system you got a extremely reliable file system with quite a good performance if you choose the journaling options that matches your needs. However, data=journal gives you a high performance penalty on write operations.

Further reading: Th antique article on IBM developer network: http://www.ibm.com/developerworks/linux/library/l-fs8.html from 2001.

Have fun!

Directory services and Linux

October 29th, 2009

LDAP is interesting, but not that easy to set up, at least not the server part.

I made different approaches to install OpenLDAP without success, the problem was always the schemas and initial data load.

With Red Hat Directory Server and its open source pendant CentOS Directory Server I was able to successfully install and maintain a LDAP directory.

Red Hat Directory Server is the successor of the Netscape Directory Server which has been purchased by Red Hat some time ago and has been open-sourced to comply with Red Hats product policy.

Is the Red Hat directory server a replacement for OpenLDAP? Yes and no. Yes because it is a open source product, available for free, and NO because there is only a small community around it.

To have a fully supported environment you need to buy a subscription from Red Hat. The starter is List-Priced @ 5000 USD/year for 500 entries. I think price tag is completely insane.

In contrary the open source variant CentOS directory server is for free. Decide by your self whats the right solution for you, OpenLDAP is definitively not ready for enterprise authentication.

Another approach is authenticating against a Microsoft Active Directory. This causes other problems which will be discussed in a future blog

Have fun!.

302 Redirects behind SSL-terminating proxies

October 29th, 2009

Problem

You have a web site all with SSL. There is a reverse proxy or load balancer that acts as SSL termination point. Behind that reverse proxy you have an Apache web server running plain http.

Your application uses 302 redirects to announce new URLs or whatever the reason is for doing so. Since the web server does not know that https URLs should be announced, the response header is wrong and looks like following:

Location http://www.example.com/your-fancy-url

The browser interprets that location header and send a request to this non-SSL URL instead of https:///www.example.com/your-fancy-url

If your reverse proxy does not know how to handle this, a connection will time-out. How to circumvent this if you have access to the web server but not to the reverse proxy or load balancer? What to do if your load balancer (such as Blue Coat devices) are closed down appliances that are not able to rewrite response headers?

Search engines do obviously not know the answer or I simply did not asked the right question.

Solution

Since Apache version 2.2.4 mod_headers is able to rewrite response headers. Just add the following to your httpd.conf

Header edit Location ^http://(.*)$ https://$1

This configuration statement will solve your problem. Redirects triggered by your back end web servers will be re-rewritten to comply with your SSL terminating reverse proxy/load balancer.

Further reading: mod_headers

Have fun….

Starting with my own blog

October 29th, 2009
Me

Me

Since a long time I wanted to start with my own blog. Why? Actually I’m not sure… I’m working in the IT industry facing strange problems in my job as a Linux system engineer. The most difficult problems are not documented anywhere, I’ll start to post solutions for such usually non-googable problems.

From time to time I’ll also going to write about the society and current stuff in the world.

Have fun, feedback welcome