Archive for the ‘Web Servers and Applications’ Category

A brief test of OTRS::ITSM Changemanagement and the insanity of ITIL compliant software

Sunday, May 23rd, 2010

OTRS is known as best-of-breed in open source incident management systems. Since quite some time, OTRS made its product ITIL V3 compliant. Means: It also comes with a change management module.

At work we use a complex and extremely user-unfriedly software. This brought me to the idea to test the OTRS change management module in order to propose OTRS as a replacement for the currently used software for the Change- Incident- and Problem Management.

Incident- and Change Management integration
I was surprised how easy it is to change the ITIL type of a ticket from “Incident” to “RfC”. As soon as the ticket is of type RfC a new button appears: Create Change. The ticket gets automatically linked to the new change. Creating the change looks strait forward. Assigning people to the CAB was also quite easy. You also can generate a CAB-template with a few clicks. So far so good…

Where the trouble begins
The newly created change is now of status “Requested”. Whats next? Right! to approve it! But how? OTRS is using something they call “state engine”. You need to add “Workorders” and “Conditions” to your change. For a standard-change I made a template with a condition “If workorder-title=Standard-Change, set change to status approved“. In this case you just link a “workorder” to the change, call it “Standard-Change” and your change will be approved. Next Condition is to set the state of the change to “successful” when the “Workorder” is of state closed. At the end of the day three tasks and approx. 20 clicks for a simple standard-change. Not too bad.

Where it goes to insanity
Non-Standard-Changes usually have a CAB (Change Advisory Board). This makes sense because the change-requester usually does mot have the full overview about complex systems and services. Now, as I wrote further up, it is quite easy to create and assign a CAB to a change. So how works the process? Usually every single member of the CAB must approve a particular change. It should be easy to send all the CAB-Members a Email with a link where they can approve or reject the change. In OTRS this is a huge and very complex task.

The change manager or change creator has to create a “workworder” of type “Approval” for every single CAB-Member AND create a condition to it. If you plan a huge change such as upgrading Powerlines in a Datacenter, the CAB can grow to dozens of people. I tried with two CAB members and it was costing me about 20 minutes to create it (Without proper texts in the change and workorders). Think about a 20-people CAB. It will take hours just to create a proper change! This is so nuts!

Why are all ITIL compliant change mangement tools just crap?
ITIL processes are quite simple. One should think it is also easy to implement them in software and in companies. Wrong! The mind of People with ITIL-Roles such as “Change Manager”, “Problem Manager”, “Availability Manager” and “you-name-it-manager” works obviously different. It looks like they add as much complexity as possible even to every simple task. Obviously the ITIL-compliant software developers think the same way or got the orders to do so. I think this is the root cause of the completely unusable software OTRS::ITSM Changemanagement and others such as Remedy and Peregrine.

Conclusion
As there is no easy usable software on the market, companies should either write its own software or getting the less-crappiest software around. At the end of the day I’m tired of this and I’m not going to test similar software again.

IUS Community RPMs for Red Hats RHEL

Sunday, May 16th, 2010

I was criticizing that software in RHEL is too outdated for web servers quite soon after release, see my blog post http://blog.delouw.ch/2010/05/02/rhel6-as-a-web-server/. While this is true for a system fully supported by Red Hat, I learned an alternative from a comment on the post. This alternative is the so called IUS community repository.

About the IUS Community Project
The project was launched in September 2009. In spite of being a young project, it has a history. At Rackspace, a large hosting company which is operating thousands of production (web) servers, it was an internal project since 2006. They decided to build up a community around it, like Fedora is for RHEL, Quote: “IUS is The Fedora of Rackspace RPMS”

Support
Like for other community repositories out there, you cannot expect a “official” support neither from Red Hat nor from IUS or Rackspace. Of course there are the usual support sources for communities such as forums, IRC, bugtracker etc.

The difference to other repositories
While most community repositories such as EPEL, rpmforge etc. are focused on providing missing software, IUS focuses on providing upgrades for web server related software which is included in RHEL. This includes PHP, Python, MySQL and others.

Package conflicts with the stock distribution
One may think replace stock software with newer version is tricky and create conflicts. There is one way to find out: Lets give it a try…

The test
The server is a basic install of the yesterday released Centos 5.5. The following installation turns this machine in a lightweight LAMP server:

yum install httpd php-mysql php php-cli php-common php-pgsql php-dba php-pdo php-gd mysql-server perl-DBD-MySQL.

Now we have the situation like it exists in many companies: An outdated webserver. Now we want to upgrade PHP to 5.3.x. Lets see what happens.


[root@centos5 ~]# rpm -i http://dl.iuscommunity.org/pub/ius/stable/Redhat/5/x86_64/ius-release-1-4.ius.el5.noarch.rpm
warning: /var/tmp/rpm-xfer.o6JH6k: Header V3 DSA signature: NOKEY, key ID 9cd4953f
[root@centos5 ~]# rpm -i http://dl.iuscommunity.org/pub/ius/stable/Redhat/5/x86_64/epel-release-1-1.ius.el5.noarch.rpm
warning: /var/tmp/rpm-xfer.MRnuo8: Header V3 DSA signature: NOKEY, key ID 9cd4953f
package epel-release-5-3.noarch (which is newer than epel-release-1-1.ius.el5.noarch) is already installed
[root@centos5 ~]#

Hmm… no GPG key…
The second output is confusing me. Is the package just a clone of epel-release-5-3.noarch? Lets go forward to see if it is working.

“yum clean-all && yum check-update” did not show any pending updates, so far so good. Now lets try to upgrade php.


root@centos5 ~]# yum install php53
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* addons: mirror.netcologne.de
* base: mirror.netcologne.de
* epel: mirror.andreas-mueller.com
* extras: mirror.netcologne.de
* ius: ftp.astral.ro
* updates: mirror.netcologne.de
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package php53.x86_64 0:5.3.2-3.ius.el5 set to be updated
--> Processing Dependency: php53-common = 5.3.2-3.ius.el5 for package: php53
--> Processing Dependency: php53-cli = 5.3.2-3.ius.el5 for package: php53
--> Processing Dependency: php53-pear >= 1:1.8 for package: php53

[omitted output]

--> Processing Conflict: php53 conflicts php < 5.3
--> Finished Dependency Resolution
php53-5.3.2-3.ius.el5.x86_64 from ius has depsolving problems
--> php53 conflicts with php
Error: php53 conflicts with php
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
package-cleanup --dupes
rpm -Va --nofiles --nodigest
The program package-cleanup is found in the yum-utils package.

Correct behaviour, since it is a replacement package. After removing php (and only php) yum was complaining about more conflicts. After removing all php related packages installed to prepare for the test, needed to be removed. So the dependencies has been proper solved. Also the installation of related stock distribution packages such as “php-pgsql” has been successfully prevented.

Conclusion
The IUS community repositories are working as expected. With such a basic test I cannot promise if there are not hidden conflicts with packages between stock RHEL/CentOS packages and those from IUS. The experience on the long term will bring more clarity. I think is is sane to do some real-life tests with servers that are in an early project phase.

Further readings:

http://iuscommunity.org/

http://wiki.iuscommunity.org/

http://saferepo.iuscommunity.org/specification/

Have fun!

An example why open source software is cool

Saturday, May 15th, 2010

Recently I have set up an Apache Tomcat. As a replacement for the Tomcat manager I deployed Psi-Probe for easy deployment and access to statistics.

Afterwards I installed the production software which needs to add a JVM parameter user.country=CH to have the proper date and time format used in Switzerland. This had a unwanted side-effect to Psi-Probe. The Interface switched to German, no way to switch the language back to English. Since my mother tongue is German, no big deal so far. Really? No! I had really problems to understand what navigation items etc.  are meaning. The German translation was that bad, it actually crippled the application.

I had the choice to either life with it, or change it and contribute it to the project. I made the later. It was about one hour of work. Hours after submitting, the changed translation file it was in SVN. The next version now comes with a much improved German translation.

This is how open source software works. If someone is not happy with the product, simply change the annoying things and submit it upstream. By the way: Psi-Probe itself is a fork of Lambda-Probe which was not maintained anymore from its origin project owner.

Try to do that with closed source software…

Have fun!

Apache HTTP server and its further development

Monday, May 3rd, 2010

The Apache httpd is one of the most stable software pieces which is still in use. The latest huge step forward was with the release of 2.0. Quo vadis Apache httpd? The most current release is 2.2.15. During the 2.2.x release cycle, there have basically been only bug-fix releases (Okay, response header rewrite starting on 2.2.9  is a nice feature). This brings me to the question: What is going on with 2.4?

The answer is quite simple: As you can read on http://httpd.apache.org/docs/trunk/new_features_2_4.html, not much. Why is the Apache httpd developing so slow? From my point of view the answer is quite simple: Apache httpd is finished. It is stable, reliable and has (almost) all features people wish. @Apache httpd developers: Great job! Thanks a lot!

Additionally there are tons of external modules to enhance the capabilities of this really great piece of software.

Honestly I can not publish my wish-list for the Apache httpd because there are no open wishes for me. Can someone have such a wish list? Please let us know and write a comment.

Have fun…

Ubuntu 10.04 LTS released

Sunday, May 2nd, 2010

End of April 2010, Ubuntu 10.04 was released. As always it is based on Debian’s Testing-Release. Canonical “stabilizes” the testing tree of Debian and adds its own look.

This time, Ubuntu radically changed its look. From my point of view it looks ugly, very ugly. Strange colors, low contrasts in menus, orange icons in Nautilus… window buttons on the left side… At the end of the day an usability-horror.

Under the hood Ubuntu is a very stable distribution with recent software. Ubuntu 10.04 is a LTS (Long Term Support) version and is thus suited as a enterprise server. Support for the server variant of Ubuntu is five years. Ubuntu is – like Debian – capable to upgrade to a new major release w/o service interruption.

Managebility

You “can” mirror Debian and Ubuntu repositories locally but it is difficult if you to not like to mirror all architectures available. Unfortunately there is (AFAIK) no software available such as Spacewalk/RHN Satellite to manage your servers.

The best method is to allow each single system installed to talk directly or via proxy to the mirror servers. This is a nightmare for firewall administrators.

To my knowledge there is no convenient way to install Ubuntu over the net. There are rumors that spacewalk and cobbler is going to get Debian/Ubuntu support at some time.

Reliability

Debian and thus Ubuntu has an evidence to be reliable. This also  seems to be true for the current release 10.04. The software came from Debians testing repository but was stabilized during months. Canonical (The sponsor of Ubuntu) has a reputation for its quality management. To use Ubuntu as a server operating system is sane.

Conclusion

As a desktop operating system I’ll avoid Ubuntu, since the usability is focused on dummy-users and not professional Linux users. For server usage you need to ask yourself about your needs. If you are operating Oracle DB’s or other commercial applications you probably want install Red Hat Enterprise Linux (RHEL). For a web server Ununtu is very well suited, even better than RHEL. In two years there will be another LTS variant available and you are free to upgrade online. Reliability is very good, manageability is poor, especially when used in larger companies.

In short: Ubuntu for web servers, RHEL/CentOS for other servers.

As always: Feedback is welcome…

Have fun!

RHEL6 as a web server

Sunday, May 2nd, 2010

New software versions

Today I’m writing about the changes and benefits of RHEL6 as a web server compared to RHEL5. Red Hat is well known for its stable API and ABI over the life-cycle of a major release. For some usage types this is a major problem. Sticking to old version of PHP, MySQL, Tomcat you-name-it-piece-of-software is problematic since web applications are rapidly changing its requirements.

  • Instead of PHP 5.1.6, RHEL6 ships almost up-to-date PHP 5.3.1. Which is good, since web applications such as TYPO3 require PHP 5.3 to be able to install security bug fixes.
  • The Apache httpd comes in Version 2.2.14 instead of 2.2.3. Since Apache is not very actively developed further,  it does not matter anyway.
  • MySQL is shipped with an almost-up-to-date version 5.1.42 vs. 5.0.77. No big deal.
  • Tomcat is being installed with version 6.0.20 instead of the very old 5.5 in RHEL5. This brings quite some benefits for Java web developers.
  • Nothing has changed since RHEL5.5 so far for PostgreSQL.  Since RHEL5.5 Red Hat ships version 8.4 in addition to 8.1.
  • Python got upgraded from 2.3 to 2.6 which probably allows to run more Python based web applications.
  • Unfortunately still no appearance of GraphicsMagick as a replacement for ImageMagick.
  • New: Ships with APC (Alternative PHP Cache). This is useful for LAMP servers with loads of traffic and helps to get response time below critical values.

Unlike other distributions, Red Hat’s default DocumentRoot is still in /var/www instead of /srv/www. From my point of view the /var should be used for libraries and similar stuff, but not for application data. This ends up in creating symlinks like it was before.

From the “I-dont-like-bloated-systems” Departement

Looks like Red Hat made a huge progress in making its system less bloated. In Versions up to RHEL5 you can experience strange package dependencies.

  • PHP and friends: While on RHEL5 a “yum install php” automatically selects PostgreSQL-libs and gmp to install, nothing like this happens on RHEL6.
  • Tomcats dependencies went down from 48 packages to only 15.

Conclusion

At the end of the day, Red Hat made a good job to enable RHEL as a web server again. The fundamental problem is still the same: In two years RHEL6 will be completely outdated and not useful for modern web application, like it is today with RHEL5. Of course you can compile the stuff by yourself, but then you’ll get a maintenance problem.

Red Hat should think about something similar like Debian’s “volatile” repository. It provides upgraded software which would otherwise be useless in a two years or older versions. I’m looking forward for a “Red Hat Volatile” Channel on our satellites.

Feedback is welcome…

Have fun!

A brief test of RHEL 6 Beta 1

Friday, April 23rd, 2010

As promised yesterday, I publish the results of a brief test of RHEL6 Beta 1 and the most important findings. It is my point of view as a system guys daily business. If not stated, this overview is based on a default installation with no customization.

General

  • There are new package groups such as  “Minimal” with 228 Packages and “Basic Server” with 523 Packages. “Basic Server” is the default installation, which means the default click trough installation compared to RHEL5 is much less bloated.
  • The versions of the most important software is quite up-to-date but as expected not on the bleeding edge.
  • Postfix is the default MTA. Finally Red Hat managed to switch away from sendmail like other distributions did it years ago.
  • Bye bye SysV init: As I guesstimated in october 2009 RHEL6 comes with upstart instead of traditional SysV init. (See http://blog.delouw.ch/2009/10/31/ready-to-upstart/). The boot process is much faster compared to RHEL5. Upstart comes with legacy support for traditional runcontrol scripts in /etc/init.d.
  • Still too many services enabled after default install. Generally unneeded services like avahi/mDNS and NFS-related daemons such as  portmap are still enabled by default.

Virtualization

As expected, Xen was removed completely from RHEL6. These is being discussed controversial. Why not providing both virtualization solutions as before? Recently Citrix released Xen4 which works well together with Kernel 2.6.32, the same version as used by RHEL6.

KVM and its friends made a huge step forward. lib-virt, virt-manager and stuff is nearly up-to-date with the upstream versions. Means: The virtualization infrastructure made a lot of progress. Installing RHEL6 as a KVM guest works great. All drivers needed (virtio) are automatically installed.

A major good message to people which are using VMware vPhere 4 is that RHEL 6 comes which native support of vmxnet3 which was obviously backported from Kernel 2.6.33. Vmxnet3 is the driver for VMware’s para-virt NIC which brings quite some performance enhancements and lower CPU usage on the ESX host.

Certifications from ISVs

A quick check (not actually tested) for the requirements for SAP and Oracle shows that those are fulfilled already. We can expect the certification quite soon after GA of RHEL6. [update] Some compatibility RPMs from the mid 1990′s disappeared.  I now need to figure out if they are *really* needed by Oracle and/or SAP[/update]

Integration with Cobbler

Integration with cobbler works like expected, cobbler import –patch=/mnt –name=rhel6 and you are done. For a quick test I just copied the kickstart template from RHEL5 and I’m not sure if this is a good method. A test-install on ESX4 failed, the system hung at the creating of the root-VG. Not sure yet if it is a bug or something is incompatible in the kickstart. [update] The system hung was because of out-of-memory. The test-installation was on a ESX guest with 384Mbyte of memory which is enough according to the documentation but too little in real life. Growing the RAM of the test system to 512Mybte helped, but some packages needed by for SAP have changes names or disappeared.  After changing/removing those RPMs, the installation went smoothly[/update]

Bugs or features?

I detected some oddities where I’m not sure if it is a bug or a feature. We will see whats going on on http://bugzilla.redhat.com.

  • No network configured after default install. At the moment you need to configure it manually (considered a Bug)
  • I detected a major security issue during install, I’m not going to disclose it before a patch is available or more information from Red Hat is made available. I reported it 2010-04-23 ~12:00 on Red Hats bugzilla bugtracker. [update] The bug gots assigned to a Red Engineer after three hours, seems like Red Hat is acting very professional on the case[/update]

Conclusion

After this brief test one can say that RHEL6 will be a really great Linux Distribution for enterprise servers. The beta is already very stable with few bugs detected from my side. My guesstimate is that mid of May 2010 there will be a second public beta released, lets stay tuned, I’ll keep you up-to-date with further findings.

Have fun!

302 Redirects behind SSL-terminating proxies

Thursday, October 29th, 2009

Problem

You have a web site all with SSL. There is a reverse proxy or load balancer that acts as SSL termination point. Behind that reverse proxy you have an Apache web server running plain http.

Your application uses 302 redirects to announce new URLs or whatever the reason is for doing so. Since the web server does not know that https URLs should be announced, the response header is wrong and looks like following:

Location http://www.example.com/your-fancy-url

The browser interprets that location header and send a request to this non-SSL URL instead of https:///www.example.com/your-fancy-url

If your reverse proxy does not know how to handle this, a connection will time-out. How to circumvent this if you have access to the web server but not to the reverse proxy or load balancer? What to do if your load balancer (such as Blue Coat devices) are closed down appliances that are not able to rewrite response headers?

Search engines do obviously not know the answer or I simply did not asked the right question.

Solution

Since Apache version 2.2.4 mod_headers is able to rewrite response headers. Just add the following to your httpd.conf

Header edit Location ^http://(.*)$ https://$1

This configuration statement will solve your problem. Redirects triggered by your back end web servers will be re-rewritten to comply with your SSL terminating reverse proxy/load balancer.

Further reading: mod_headers

Have fun….