Archive for the ‘Red Hat’ Category

« Older Entries

Cross distribution system management with Spacewalk

Tuesday, May 24th, 2011

In a perfect world, all systems in a data centre are running the same Linux operating system, a homogeneous system landscape. In real life things are working differently. Windows systems are out of focus in this post, lets concentrate on Linux systems.

Most companies with a large Linux base are either RHEL shops or using SLES. A lot of RHEL users have some SLES systems running and so are SLES users running some RHEL systems. Some companies have additional systems running Debian.

How to handle those heterogeneous system landscapes? Those real world scenarios? Lets assume a company runs 500 RHEL systems, 20 SLES systems and some 10 Debian systems.

At the moment, for the base software management subscription such Linux users are spending a lot of money for RHN Satellite and SUSE Manager. Additionally there are per-system costs for management, provisioning and other modules. The Debian systems are handled manually. A lot of additional costs for a few out-of-strategy systems.

The solution is Spacewalk, the upstream project of the RHN Satellite which is at the same time the upstream for the recently released SUSE Manager. While SUSE offers support for RHEL systems, Red Hat does not (yet) offer support for SLES systems for RHN Satellite.

In Spacewalk Version 1.4 code contributions from SUSE are included and a student at Brno University of Technology contributed Debian support for Spacewalk as part of his master thesis.

While the support for SUSE is already quite stable, the Debian related code still have some rough edges. No wonder, SUSE is using RPM for its packaging wile Debian has its own packaging system. This makes it much easier for SUSE to get Spacewalk ready for its distribution.

At the moment, one can call the Debian support still as experimental, but the goal for the Spacewalk project is to have it fully functional in future releases.

The goal should be that both of the management system from the major enterprise Linux vendors, Red Hat and SUSE should support each others distribution for its Spacewalk based products. Debian is a niche player in the enterprise Linux environment and should also be supported by both products, RHN Satellite and SUSE manager. Nobody does expected to get system support for those distributions by the competing distribution, but having support for the management of it.

Further readings:
Registering Clients
Deb support in Spacewalk

Have fun!

Tags: , , ,
Posted in Debian, Infrastructure, Linux, Red Hat, SUSE | Comments Off

Implement a high available Cobbler provisioning system

Sunday, April 24th, 2011

A not so well known feature of Cobbler is its replication facility. It allows you to create a high available system provisioning system. The whole set up is straight forward.

Background
Today people tend to NOT backup systems, only data is being backed up. In the case of a system failure, they just re-provisioning the system and automatically configure it with configuration management tools such as Puppet, CFengine or RHN Satellite.

You not only need to have your configuration management system(s) to be redundant, you also need to replicate your Cobbler provisioning systems.

It even works in a set up where you have multiple datacenters for disaster recovery, having one or two Cobbler servers at each site.

Luckily, Cobbler comes with that feature out-of-the-box.

Assumptions

  • Both Cobbler servers are in the same network
  • Your master cobbler has a DNS-entry cobbler-master, your slave is cobbler-slave

Lets do it

Unfortunately, Cobbler replication, at the moment, does not work when SELinux is running in enforcing mode. You need to turn it off on both Cobbler servers.

setenforce 0

Hopefully this will change soon.

For a initial replication just run:

cobbler replicate --master=cobbler-master --sync-all

On the slave server. It is recommended to run this command nightly by a cronjob to ensure you have the latest RPM-packages of your distros in sync. Feel free to run this command manually at every time if you think it is needed to immediately sync the whole Cobbler system.

Update on-the-fly
You can use a Cobbler trigger script that automatically connects to the slave and triggers a replication with the master whenever a system is added or deleted.

I’m using the following script to do so:

#!/bin/bash
#
# This is a Cobbler trigger script that triggers a replication on the slave
# server. You need to create a private/public key-pair on the master and
# add the public key to ~/.ssh/authorized_keys on the slave

SLAVE=cobbler-slave

ssh $SLAVE "cobbler replicate --master=cobbler-master --systems=* --prune"
ssh $SLAVE "cobbler sync"

Put this script in /var/lib/cobbler/triggers/add/system/post and create a symlink in /var/lib/cobbler/triggers/delete/system/post on your master server.

Security
Cobbler handles the file /etc/rsyncd.conf. So far so good. You need to be aware that EVERY host can rsync your distros, kickstarts, snippets etc unless you take some security measures with some firewall and/or other restrictions.

Possibly the simplest way is configuring xinetd.

A sample /etc/xinetd.d/rsync, where 192.0.2.2 is the IP of your slave, would be:

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
#       allows crc checksumming etc.
service rsync
{
        disable         = no
        only_from       = 192.0.2.2
        flags           = IPv6
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

DHCP redundancy
You also need to operate a dhcp server on both Cobbler servers. If you have both Cobbler servers on the same subnet, you may have a look how to configure ISC dhcpd. A good guide (untestet) is at http://www.madboa.com/geek/dhcp-failover/. The lazy ones just turn off dhcp on the slave and activate it as needed.

Conclusion
As I wrote before, Cobbler is just great. Not only “batteries are included”, high availability, disaster safety and other cool stuff is included as well and can be quickly implemented.

Having fun? I rarely needed to re-provision systems because they went foobar and hopefully I will never experience a disaster such a burned-down, exploded or whatever datacenter. Cobbler does not protect you from such disasters, it helps you to recover.

Tags: , ,
Posted in Infrastructure, Red Hat | Comments Off

I got employed by Red Hat

Thursday, April 21st, 2011

This is pretty cool: End of March I signed a contract with Red Hat as a senior Linux consultant. It is not just “another new job”. It is cool for (at least) two reasons: First reason is that Red Hat is not “just another company”, it is Red Hat which is not very comparable to other employers, it is THE Linux and open source company, for me as a open source guy, this is perfect. The second reason is: I’m moving from Zurich in Switzerland to Berlin in Germany.

So, two major changes in my life at the same time. I’m looking forward to the challenges that are waiting for me.

I’ll continue to work at Siemens IT Solutions and Services AG until approx. mid of June and start working at Red Hat at 1st of July.

From May, 09 to May 15. I’ll be the first time in Berlin to have a look at the city and its different suburbs. I’ll also be there to organize some stuff required to settle in Berlin. In the same time, Europe’s biggest Linux conference will be held in Berlin, the “Linux Tag”. I guess I’ll have a lot of fun, and maybe meet some of my future workmates.

It is hard for me to leave my country, I have a lot of friends here. On the other hand, Berlin is just about 1.5h away by plane. As a consultant, I’m travelling a lot. Because of that, it would not be that easy to build up a social network (I mean real-life-stuff, not Facebook) in Berlin.

It also is not easy for me to leave Siemens, I’m involved in a very cool project with the Swiss government (all Systems will be RHEL6) and I also have friends and nice workmates at work which I’m going to leave.

I already know quite some people at Red Hat, they are all nice and I guess some of them will get good friends over the time.

Having fun?

Absolutely guaranteed!

Tags: , ,
Posted in Linux, Personal, Red Hat, Uncategorized | 9 Comments »

I voted for beefy miracle

Thursday, April 7th, 2011

Beefy miracle

 

There is a open poll on voting for a name for Fedora 16. I gave my vote to Beefy Miracle. Why I voted for Beefy Miracle? Because it is cool, geeky, freaky, I’m loving hot dogs and it is something new.

The Fedora distribution is geeky, freaky and open to new stuff.

Having fun? Of course!

Tags:
Posted in Red Hat, Uncategorized | Comments Off

How to harden RHEL systems

Sunday, March 27th, 2011

Some time ago, the NSA released an excellent guide how to harden RHEL5 systems.

Despite of being written for RHEL5, it partially also applies to RHEL6 and newer versions of Fedora. It is also worth looking at it for users of non-RH breed distributions. To be mentioned: Its clearly focused on server systems, not desktops.

Some of the topics are really basic stuff which is already in place as industries “best practices”, other methods are not that well known.

Most of the items can be implemented very easy, others should be reviewed if the complexity is worth the gain of security.

Minimize Software to Minimize Vulnerability is a good starting point. RHEL5 is quite bad on this point, a default installation comes with a complete desktop environment. RHEL6 made a lot of progress on this issue as I wrote about it in a earlier post.

The default file system layout of most Linux distributions is suboptimal. At least /var, /tmp and /home should be on separate file systems. You can enhance the systems security by setting mount options such as noexec, nodev and nosuid where appropriate.

Always set SELinux to Enforcing mode where possible. Since tools like audit2allow and selinux-polgengui enables users to easily create basic policies, its no more rocket science. For further readings and hints about SElinux, have look on Dan Walsh’s Blog.

Check if only needed daemons are running. I. e if you are not using NFS, disable portmapper and friends.

Other things things disabling rhnsd is IMHO not a good idea. Enabling a warning banner for pre-login texts is just clueless.

Conclusion
NSA provides a nice guide which is really worth reading for server administrators. Some topics described in the guide are maybe overkill and complex, while others are easy to implement and maintain. Hopefully NSA will soon update its paper to RHEL6.

It also shows that Linux distributors have room for improvements to provide a better default security.

Have fun!

Tags: , ,
Posted in Linux, Red Hat | 1 Comment »

SUSE Manager based on Fedora Spacewalk

Thursday, March 3rd, 2011

SUSE announced the availability of SUSE manager. Having a closer look to it, one recognizes it is based on Fedora Spacewalk. It is a clone of the Red Hat Satellite.

A few weeks ago I was puzzled to see a post on the spacewalk-devel mailing list. SUSE was contributing some code. What the heck? Now it is clear, they are using Spacewalk as there source for its own product. Spacewalk is no longer just the upstream of RHN Satellite, but also a major tool for managing SLES systems.

The open source way
It is good practice to share knowledge and code between different distributions. SUSE profits from the work Red Hat has done before, and Red Hat profits from the contributions of SUSE. IMHO this is the right way how open source software should work.

The price tag
SUSE claims “SUSE Manager allows you to save up to 50 percent for Linux support”. Really?

Lets have a look to How to buy. The price is exactly the same as for RHN Satellite: USD 13,500. Really the same price tag? Lets dig deeper on features Click on Database support. One would read 

"SUSE Manager provides a built-in Oracle XE database, but can also leverage existing
Oracle 10g or 11g databases, to locally store all data related to the
managed Linux servers."

Means: With the free Oracle XE database delivered with SUSE you can manage just a few systems. If you want to manage more systems, you need to buy a very expensive Oracle License which, last least, doubles the price tag of SUSE Manager.

And Debian? There are some works going on, maybe I’m going to write soon about Spacewalk and what it can do for and with Debian.

Conclusion
Because SUSE was not in a hurry to release its new product, I can not understand why SUSE was not helping the Spacewalk project to get PostgreSQL production ready before releasing it. This would provide its customers (and the spacewalk community) a real benefit.

I hope that SUSE will sustainably contribute code to Spacewalk, it is now in the interest of users of both distributions.

Have fun!

Tags: , , , ,
Posted in Infrastructure, Linux, Red Hat, SUSE | 11 Comments »

Updating a distro in cobbler

Monday, February 28th, 2011

A few weeks ago RHEL 5.6 was released, the installation media was also updated. So it is time to get it into cobbler to deploy the latest dot release when provisioning new systems.

Lets assume your profile name is rhel5-x86_64, you have an existing distro named rhel55-x86_64 and you want to replace it with rhel56-x86_64.

Lets start with importing the new distro:

# Mount the ISO as loop back
mount /some/where/rhel-server-5.6-x86_64-dvd.iso /mnt/rhel56iso -o loop

# Import the Install Media
cobbler import --path=/mnt/rhel56iso --name rhel56-x86_64

Cobbler creates a profile consisting of the name provided at import plus its architecture. We do not want that profile, lets delete it:

cobbler profile remove --name=rhel56-x86_64

The next step is to change the distribution used by the profile

cobbler profile edit --name=rhel5-x86_64 --distro=rhel56-x86_64

If you want to delete the old distro, first check if there is a profile still using this distro, otherwise any childs will be deleted.

cobbler profile find --distro=rhel55-x86_64

If the result is null, it is safe to remove the old distribution. If you want to keep it for a fallback scenario, just omit this step.

cobbler distro remove --name=rhel55-x86_64

Cobbler is just cool stuff :-) Want to know more? Visit https://fedorahosted.org/cobbler/wiki.

Have fun!

Tags: ,
Posted in Infrastructure, Linux, Red Hat | 1 Comment »

A review of RHEV

Sunday, February 27th, 2011

In the past few weeks I had the chance to have a closer look at the current release 2.2. The reason is that I’m working on a project using RHEL6 clients as virtual Desktops. For a proof-of-concept I’ve set up a test environment in the lab. Due to the lack of time I was not able to test every single feature.

After reading some docs, it was amazingly easily and quickly installed.

Test environment
The tests have been made on the following hardware:

  • 2 Xeon servers with a total of 8 cores and 24Gbyte on each host. OS is RHEL5.6. Those hosts are called RHEV-H (H like Hypervisor)
  • 1 Windows 2008R2 server (64bit) with 4Gbyte RAM as RHEV-M (M like management)
  • 1 dedicated NetApp filer serving storage over NFS
  • 1 RHEL 5.6 Server providing “cheap” NFS storage
  • 3 different networks connected to 3 NICs
  • 1 RHEL 5.6 “thin” client with spice-xpi as client
  • 1 Windows 2008R2 Server with spice-activex as client (actually on the management server)

VM’s (All 64bit):

  • 20 Virtual desktops with RHEL 6.0 clients using one core and 2Gbyte or RAM each
  • 2 Virtual servers with RHEL 6.0 server using 2 cores and 2Gbyte of RAM each
  • 2 Virtual servers with RHEL 5.6 server using 2 cores and 2Gbyte of RAM each

Management Portal
The Management interface is the central point of administration. It is not always as intuitive as expected. If you are new to RHEV, you can get confused. A big plus is the search functionality. One can search for storage, VMs, hosts and any other item in the environment.

User Portal
The user portal is very simple, lean and clean. Users see the machines that are assigned to them, they can power on and off the machines and can connect them to powered VMs, and the client appears on the desktop.

The user portal resides on the management server and is just the connection broker. As soon as a user connects to its desktop or server, the connection will be made directly to the spiced qemu-kvm running on one of the hypervisors.

Client
At the moment there are two different clients supported:

  • spice-activex with Internet Explorer 7+ on Windows XP, Vista and 7
  • spice-xpi with Firefox for RHEL5 and RHEL6

If you do not fear the work, you probably can get the spice-xpi client also running on Fedora 13 and other Linux distributions.

The client communicates with the VM over SPICE (Simplified Protocol for Independent Computing Environments). It can forward USB devices as well as sound and other multimedia stuff. For more informations about the protocol, please have a look at http://www.spice-space.org/home.html.

Storage
The first step to set up a RHEV environment is to define and set up the storage.

There are 3 types of storage:

  • ISO, where you upload your install-media for your virtual machines.
  • Data, where the VM images are stored.
  • Export, where I have no clue (yet) what it is good for.

Usually the storage is a NFS or iSCSI server, FC storage is also supported. The storage server creates snapshots of volumes for backing up the stuff.

Hosts (hypervisors)
You can either use RHEV-H as an “embedded” hypervisor which comes as a stripped-down RHEL5 distribution (the ISO is just about 75Mbyte IIRC) or you can set up RHEL5 and use it as a hypervisor.

First I tried the RHEV-H, it is manual work to install it. You need to burn a CDROM put it in your server and enter some stuff like network config etc. In the short time I did not tried to provision RHEV-H with cobbler. According to the documentation one can pass boot parameters to partially automate the installation.

Because I have a cobbler server handy, I decided to use RHEL5 as hypervisors. Just be sure you subscribe the system to the channels “rhel-x86_64-server-vt-5″ and rhel-x86_64-rhev-mgmt-agent-5. You also need to allow root to log in via SSH. You can change this later after the hosts is registered on the RHEV-M server.

After the setup of the RHEL5 servers you just need to tell RHEV-M that there is a new host available. Tell RHEV-M about the root password and all the needed additional software gets installed automatically.

That’s a pretty lean procedure to install and set up hosts. Hopefully there will soon be a way to fully automate this, w/o using MS Powershell.

The technologies on the hosts are the well known stable and mature KVM and qemu-kvm. There are also parts of o-virt being used.

Networking
I do not know how many networks you can add, but I think enough even for large environments. The network configuration part in RHEV-M shows a list of available Ethernet ports on each host. Just assign them a static network and IP address and you’re done. Make sure you define the network you want to add on all hosts in a particular datacenter.

Ensure that firewalls between the users and VM’s networks are configured accordingly to avoid connection problems.

Sizing and performance

  • Memory overcommitment
    Thanks to KSM (Kernel Samepage Merging) one can quite overcommit the memory. Ensure you have a LOT of swap on your hosts. Why? It takes some time until KSM kicks in and frees memory pages. During the collection of those pages, swap space is needed to prevent getting a visit by the OOM killer.

    I once faced a complete crash when putting one host in maintenance mode and all VMs have been migrated to the second host. That’s exactly why you should have LOTs of swap spaces to survive when using memory over-commitment. During the period of KSM searching for same pages, performance is degraded.

    Under normal circumstances, this will never happen since the virtual hosts are load balanced. This means VM’s are distributed to have the optimal performance.

  • CPU overcommitment
    This very depends on your users computing needs. For normal desktops and servers you can overcommit at least 200% of the CPUs and the performance is still fine. For CPU intensive workload it is not recommended to over commit CPUs.
  • Number of hosts
    Its recommended to start with at least three hosts to avoid temporary performance penalties if you need to put a host into maintenance mode. (See also “Memory overcommitment”).

    The performance is surprisingly good, the user experience is nearly the same as on physical desktops. For servers, KVM is already known for its good performance.

Drawbacks
Unfortunately you can not hot-add or hot-remove virtual CPU’s, NICs and storage. Despite of being supported by the underlying technologies used, your VM must be powered off to add or remove resources.

Another drawback is the lack of different storage classes, this makes it actually impossible to use the product in large enterprise environments. Since it makes no sense to back up swap drives, it is waste of backup space.
It is also not possible to have different storage for different needs. Lets say a bulk RAID5 consisting of cheap 2Tbyte SATA disks for operating systems and a RAID10 consisting of fast 15k SAS or FC disks for data and swap. It is also not possible to life-migrate VMs from one storage to another.

As of today, SPICE is only useful on LAN networks. Depending on the work load it can demand lots of bandwidth available. For usual office workload, 1 Mbps per connection is fine. A WAN optimized version is under development and should be released “later this year”.

The two supported clients (spice-xpi and spice-activex) can be problematic when the clients are not in an environment controlled by you. Why? Lots of enterprises are preventing its users to install Active X applications and Firefox Add-On’s for security reasons. It would be better to have some kind of “fat client” which does not need to be installed, or a Java Applet which does the job.

At the moment, it is still required to have a Windows 2008R2 server installed and use MS technologies such as MSSQL, DotNET, Powershell and Active Directory. Because you can not manage RHEV-M with a satellite, one needs to download and install updates manually.

RHEV-M is not only the management server, it is also the connection broker for clients connecting. RHEV-M is a single point of failure. Maybe one can build a MS-Cluster with it, no clue. It is also not possible to self-host RHEV-M on the RHEV-H hosts. Already connected clients should not be disconnected in the case of RHEV-M failing (to be tested).

Red Hat is working on an Linux-Only RHEV-M, as you can read in a comment from a Red Hat employee in a earlier post.

The same comment talks about the WAN problems to be solved in near future.

Conclusion
Frankly: From my point of view, RHEV is not yet “enterprise ready”, this is because of the drawbacks mentioned above, especially the storage shortcomings and RHEV-M being a SPoF.

For smaller environments up to lets say ~50 servers or ~200 desktops it good enough.

Nevertheless: I think RHEV has a huge potential in the future. Compared to VMware ESX, the KVM’s technology is much better and scales better. At the moment VMware’s USP is its management software, not the hypervisor used. As soon as Red Hat ironed out the major drawbacks, I’ll expect a boost for RHEV.

Red Hat is keen to improve the product, I guess a lot of improvements will be annouced in 2011.

Have fun!

Tags: , , , , ,
Posted in Infrastructure, Linux, Red Hat, Virtualization | 3 Comments »

RHEL6.1 and Red Hat is changing its subscription methods

Wednesday, February 16th, 2011

I just got an email with the subject “Opportunity for Red Hat Certified Professionals to test new Red Hat software”.

Quoting the email:

" The new subscription management tools provide a very different user
   experience than today’s Red Hat Network (RHN). We would like to get
   your feedback on the software so that we can improve the tooling before
   RHEL 6.1 is released. As part of this Beta Program, we will be offering
   you a beta version Red Hat Enterprise Linux Personal Subscription. This
   subscription will allow you to access the tooling that will be provided
   as part of the RHEL 6.1 minor release."

In the same email Red Hat offers the audience to have up to 10 systems registered for free:

  "Under  the Personal Subscription provided via the Beta Program, users are able
   to deploy the software on up to 10 personal systems. The Red Hat
   Personal Subscriptions entitle you to access software and software
   updates"

That are actually great news for us as Red Hat certified professionals. But it also opens new questions about the future of RHN, the RHN Satellite and the subscription model of Red Hat in general.

According to the documentation (You need to be at least RHCE and provide your RHCE number to get access to it), with RHEL6.1 registering systems to the RHN will completely change. No more rhn-register it is now the CLI command subscription-manager.

The most important thing that changed is that the username and password of RHN needs to be transmitted just once, afterwards you will get identified by a X509 client certificate.

The only drawback I’ve found was that the command to register a consumer needs to provide the password in clear text on the command line.

And for Satellite users?
As far as I can see, nothing changes, Satellite users can still provision the systems with activation keys, it is still channel based, not product based.

For enterprise users nothing is changing in the next time, the new entitlement and subscription method does only apply to does users NOT using a Satellite server, at least for now and for RHEL6.1.

The Readme also mentions RHN Satellte 5.5 which is not (yet) released. It is quite unclear what is expecting us with Satellite 5.5.

Reading some bugzilla entries, it is clear that there is still some time until RHN Satellite 5.5. will hit the road.

Please: A public Beta for RHN Satellite 5.5
Please Red Hat, provide a public, or at least a semi-public beta (like for RHEL6.1) release, to give your enterprise customers a chance to do the Q&A which was missing on the release of RHN Satellite 5.4.

Having fun? I do actually not care about RHN Network, I’m a Satellite user. Personally I’m having fun with my 10 personal RHEL6.1 subscriptions for free, it allows me to do lots of tests before putting RHEL6.1 into production.

Tags: , , ,
Posted in Linux, Red Hat | 6 Comments »

RHEL 4.9 released

Wednesday, February 16th, 2011

Today, Red Hat released its “service pack” or “maintenance release” of RHEL4. According to Red Hats life cycle policy” it ends the production stage two.

That means: In future only bugs with a high severity will be fixed. The “normal” life cycle of RHEL4 will end in approx. one year.

This means that everyone using RHEL4 systems should think about a migration scenario to RHEL6. Unfortunately, Red Hat does not support OS upgrades, you need to install the systems from scratch. Since RHEL4 was released in February 2005, most of this systems have reached there lifecycle anyway.

Its time to migrate your RHEL4 systems.

For the release notes, please see http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/4/html/4.9_Release_Notes/index.html

Have fun with migrating….

Tags:
Posted in Linux, Red Hat | Comments Off

« Older Entries