Archive for October, 2009

Ready to upstart?

Saturday, October 31st, 2009

upstart

It is time to replace the aged SysV init system with someting better

At the time when  SysV init (pronounced “System five”) appeared, hardware configurations have been quite static, no hot plug and similar fancy stuff.

SysV init is started after the kernel is loaded. The init process reads /etc/inittab and walks trough the runcontrol script and runlevels. This sequential walk-trough takes most of the time when booting a modern Unix system.

Upstart follows another approach: Starting daemons and services in parallel and event driven.  This will speed up the boot process beyond expectations.

A very nice feature of upstart is: All processes will be started in background, no more blocking of the boot process trough hanging run control scripts!

If a service unexpectedly dies, it will be respawned  automatically up to a configurable limit in times per period.

Upstart is event-driven, a event can be e.g. plugging in new hardware which ends up starting the needed service for it. There are also plans to replace cron and atd with upstart since this are basically time-triggered events. The developers also thinking about replacing the inetd, since a network connection can be considered as a event.

Transition

Since most of the software out there do not natively support upstart yet, transition methods are needed for a smooth transition from SysV init to upstart. Traditional SysV run control scrips are fully supported, even distributions slowly switch to the event/job model of upstart. E.g. one of the first distributions switched to upstart was Ubuntu 6.10, and now with Ubuntu 9.10 – three years later – they begin to ship its distribution with the first native upstart scripts.

Splitting Unix systems apart

Years ago there only have been two init systems: SysV init and BSD init, a sysadmin was comfortable to use them on whatever system. Now there are SysV init, Upstart from Ubuntu, lauchd from Apple, SMF (System Management Facility) from Sun Microsystems and possibly others. All of this SysV init replacements are working differently,  different commands, different architecture… This makes the job of a sysadmin not easier when managing a heterogeneous system landscape.

Linux distributions stay together

The good news: On the Linux side it looks like Upstart will be the future standard for system initialization, no balkanization of the Linux Landscape so far.

Linux Distribution with upstart

The following distributions are already shipping upstart:

  • Ubuntu
  • Debian
  • Fedora
  • Others?

Since Fedora 11 and 12 will be the upstream for the upcomming RHEL6 distribution it is most likely that RHEL6 comes with upstart. At openSUSE there are some discussions (see https://features.opensuse.org/305690 for details). Maybe there is a chance for openSUSE 11.3 and later on SLES12.

Further readings:

Upstart web site: http://upstart.ubuntu.com
Wikipedia article: http://en.wikipedia.org/wiki/Upstart

Have fun!


How are jornaling options affect performance of the ext3 filesystem

Friday, October 30th, 2009

The need for speed
Everyone looks for the optimum of speed in its servers. Todays servers have plenty of spare CPU power and RAM is dirty cheap. Todays common bottleneck is storage.

One way to solve the bottleneck is trowing money on it, the other smarter way is choosing the best matching file system and it options for the purpose of the server.

On Linux systems a bunch of file systems is available and ready to use. There are some high performance file systems such as SGI’s XFS or reiserfs. Both are known to be quite performant but having the drawback of being unreliable in case of a hard crash or a power loss.

As file systems are the key point for reliability, XFS and reiserfs are out of question. So whats left? ext3.

Problem
Ext3 is not known for its high performance, it is rather slow compared to xfs, especially if you handle with of lot of files such as on a web server.

Solution
Choose the right options for journaling of ext3. You have the choice of three different journaling options, data=writeback which means written data is first written to RAM and later on disk. This is the most performant option, but with the greatest risk of loosing data in case of a crash or power loss. Before choosing this option use xfs, it is more performant at the same risk.

The compromise is data=ordered Lets quote the man page: –All data is forced directly out to the main file system prior to its metadata being committed to the journal — At the end of the day this means data loss in hardly happening but not impossible. This option offers a balance between write speed and reliability.

Whats about the third option data=journal? It means that one would think if all data is written first in the journal and then to its final destination on disk, I/O performance gets decreased.

In theory, data=writeback is the fastest and data=journal the slowest option. Belief it or not: data=journal is in many cases the fastest option, especially in mostly-read applications when you concurrently read lots of small files (such as on web servers).

At the end of the day: With the ext3 file system you got a extremely reliable file system with quite a good performance if you choose the journaling options that matches your needs. However, data=journal gives you a high performance penalty on write operations.

Further reading: Th antique article on IBM developer network: http://www.ibm.com/developerworks/linux/library/l-fs8.html from 2001.

Have fun!

Directory services and Linux

Thursday, October 29th, 2009

LDAP is interesting, but not that easy to set up, at least not the server part.

I made different approaches to install OpenLDAP without success, the problem was always the schemas and initial data load.

With Red Hat Directory Server and its open source pendant CentOS Directory Server I was able to successfully install and maintain a LDAP directory.

Red Hat Directory Server is the successor of the Netscape Directory Server which has been purchased by Red Hat some time ago and has been open-sourced to comply with Red Hats product policy.

Is the Red Hat directory server a replacement for OpenLDAP? Yes and no. Yes because it is a open source product, available for free, and NO because there is only a small community around it.

To have a fully supported environment you need to buy a subscription from Red Hat. The starter is List-Priced @ 5000 USD/year for 500 entries. I think price tag is completely insane.

In contrary the open source variant CentOS directory server is for free. Decide by your self whats the right solution for you, OpenLDAP is definitively not ready for enterprise authentication.

Another approach is authenticating against a Microsoft Active Directory. This causes other problems which will be discussed in a future blog

Have fun!.

302 Redirects behind SSL-terminating proxies

Thursday, October 29th, 2009

Problem

You have a web site all with SSL. There is a reverse proxy or load balancer that acts as SSL termination point. Behind that reverse proxy you have an Apache web server running plain http.

Your application uses 302 redirects to announce new URLs or whatever the reason is for doing so. Since the web server does not know that https URLs should be announced, the response header is wrong and looks like following:

Location http://www.example.com/your-fancy-url

The browser interprets that location header and send a request to this non-SSL URL instead of https:///www.example.com/your-fancy-url

If your reverse proxy does not know how to handle this, a connection will time-out. How to circumvent this if you have access to the web server but not to the reverse proxy or load balancer? What to do if your load balancer (such as Blue Coat devices) are closed down appliances that are not able to rewrite response headers?

Search engines do obviously not know the answer or I simply did not asked the right question.

Solution

Since Apache version 2.2.4 mod_headers is able to rewrite response headers. Just add the following to your httpd.conf

Header edit Location ^http://(.*)$ https://$1

This configuration statement will solve your problem. Redirects triggered by your back end web servers will be re-rewritten to comply with your SSL terminating reverse proxy/load balancer.

Further reading: mod_headers

Have fun….

Starting with my own blog

Thursday, October 29th, 2009
Me

Me

Since a long time I wanted to start with my own blog. Why? Actually I’m not sure… I’m working in the IT industry facing strange problems in my job as a Linux system engineer. The most difficult problems are not documented anywhere, I’ll start to post solutions for such usually non-googable problems.

From time to time I’ll also going to write about the society and current stuff in the world.

Have fun, feedback welcome