Posts Tagged ‘VMware’

A review of RHEV

Sunday, February 27th, 2011

In the past few weeks I had the chance to have a closer look at the current release 2.2. The reason is that I’m working on a project using RHEL6 clients as virtual Desktops. For a proof-of-concept I’ve set up a test environment in the lab. Due to the lack of time I was not able to test every single feature.

After reading some docs, it was amazingly easily and quickly installed.

Test environment
The tests have been made on the following hardware:

  • 2 Xeon servers with a total of 8 cores and 24Gbyte on each host. OS is RHEL5.6. Those hosts are called RHEV-H (H like Hypervisor)
  • 1 Windows 2008R2 server (64bit) with 4Gbyte RAM as RHEV-M (M like management)
  • 1 dedicated NetApp filer serving storage over NFS
  • 1 RHEL 5.6 Server providing “cheap” NFS storage
  • 3 different networks connected to 3 NICs
  • 1 RHEL 5.6 “thin” client with spice-xpi as client
  • 1 Windows 2008R2 Server with spice-activex as client (actually on the management server)

VM’s (All 64bit):

  • 20 Virtual desktops with RHEL 6.0 clients using one core and 2Gbyte or RAM each
  • 2 Virtual servers with RHEL 6.0 server using 2 cores and 2Gbyte of RAM each
  • 2 Virtual servers with RHEL 5.6 server using 2 cores and 2Gbyte of RAM each

Management Portal
The Management interface is the central point of administration. It is not always as intuitive as expected. If you are new to RHEV, you can get confused. A big plus is the search functionality. One can search for storage, VMs, hosts and any other item in the environment.

User Portal
The user portal is very simple, lean and clean. Users see the machines that are assigned to them, they can power on and off the machines and can connect them to powered VMs, and the client appears on the desktop.

The user portal resides on the management server and is just the connection broker. As soon as a user connects to its desktop or server, the connection will be made directly to the spiced qemu-kvm running on one of the hypervisors.

Client
At the moment there are two different clients supported:

  • spice-activex with Internet Explorer 7+ on Windows XP, Vista and 7
  • spice-xpi with Firefox for RHEL5 and RHEL6

If you do not fear the work, you probably can get the spice-xpi client also running on Fedora 13 and other Linux distributions.

The client communicates with the VM over SPICE (Simplified Protocol for Independent Computing Environments). It can forward USB devices as well as sound and other multimedia stuff. For more informations about the protocol, please have a look at http://www.spice-space.org/home.html.

Storage
The first step to set up a RHEV environment is to define and set up the storage.

There are 3 types of storage:

  • ISO, where you upload your install-media for your virtual machines.
  • Data, where the VM images are stored.
  • Export, where I have no clue (yet) what it is good for.

Usually the storage is a NFS or iSCSI server, FC storage is also supported. The storage server creates snapshots of volumes for backing up the stuff.

Hosts (hypervisors)
You can either use RHEV-H as an “embedded” hypervisor which comes as a stripped-down RHEL5 distribution (the ISO is just about 75Mbyte IIRC) or you can set up RHEL5 and use it as a hypervisor.

First I tried the RHEV-H, it is manual work to install it. You need to burn a CDROM put it in your server and enter some stuff like network config etc. In the short time I did not tried to provision RHEV-H with cobbler. According to the documentation one can pass boot parameters to partially automate the installation.

Because I have a cobbler server handy, I decided to use RHEL5 as hypervisors. Just be sure you subscribe the system to the channels “rhel-x86_64-server-vt-5″ and rhel-x86_64-rhev-mgmt-agent-5. You also need to allow root to log in via SSH. You can change this later after the hosts is registered on the RHEV-M server.

After the setup of the RHEL5 servers you just need to tell RHEV-M that there is a new host available. Tell RHEV-M about the root password and all the needed additional software gets installed automatically.

That’s a pretty lean procedure to install and set up hosts. Hopefully there will soon be a way to fully automate this, w/o using MS Powershell.

The technologies on the hosts are the well known stable and mature KVM and qemu-kvm. There are also parts of o-virt being used.

Networking
I do not know how many networks you can add, but I think enough even for large environments. The network configuration part in RHEV-M shows a list of available Ethernet ports on each host. Just assign them a static network and IP address and you’re done. Make sure you define the network you want to add on all hosts in a particular datacenter.

Ensure that firewalls between the users and VM’s networks are configured accordingly to avoid connection problems.

Sizing and performance

  • Memory overcommitment
    Thanks to KSM (Kernel Samepage Merging) one can quite overcommit the memory. Ensure you have a LOT of swap on your hosts. Why? It takes some time until KSM kicks in and frees memory pages. During the collection of those pages, swap space is needed to prevent getting a visit by the OOM killer.

    I once faced a complete crash when putting one host in maintenance mode and all VMs have been migrated to the second host. That’s exactly why you should have LOTs of swap spaces to survive when using memory over-commitment. During the period of KSM searching for same pages, performance is degraded.

    Under normal circumstances, this will never happen since the virtual hosts are load balanced. This means VM’s are distributed to have the optimal performance.

  • CPU overcommitment
    This very depends on your users computing needs. For normal desktops and servers you can overcommit at least 200% of the CPUs and the performance is still fine. For CPU intensive workload it is not recommended to over commit CPUs.
  • Number of hosts
    Its recommended to start with at least three hosts to avoid temporary performance penalties if you need to put a host into maintenance mode. (See also “Memory overcommitment”).

    The performance is surprisingly good, the user experience is nearly the same as on physical desktops. For servers, KVM is already known for its good performance.

Drawbacks
Unfortunately you can not hot-add or hot-remove virtual CPU’s, NICs and storage. Despite of being supported by the underlying technologies used, your VM must be powered off to add or remove resources.

Another drawback is the lack of different storage classes, this makes it actually impossible to use the product in large enterprise environments. Since it makes no sense to back up swap drives, it is waste of backup space.
It is also not possible to have different storage for different needs. Lets say a bulk RAID5 consisting of cheap 2Tbyte SATA disks for operating systems and a RAID10 consisting of fast 15k SAS or FC disks for data and swap. It is also not possible to life-migrate VMs from one storage to another.

As of today, SPICE is only useful on LAN networks. Depending on the work load it can demand lots of bandwidth available. For usual office workload, 1 Mbps per connection is fine. A WAN optimized version is under development and should be released “later this year”.

The two supported clients (spice-xpi and spice-activex) can be problematic when the clients are not in an environment controlled by you. Why? Lots of enterprises are preventing its users to install Active X applications and Firefox Add-On’s for security reasons. It would be better to have some kind of “fat client” which does not need to be installed, or a Java Applet which does the job.

At the moment, it is still required to have a Windows 2008R2 server installed and use MS technologies such as MSSQL, DotNET, Powershell and Active Directory. Because you can not manage RHEV-M with a satellite, one needs to download and install updates manually.

RHEV-M is not only the management server, it is also the connection broker for clients connecting. RHEV-M is a single point of failure. Maybe one can build a MS-Cluster with it, no clue. It is also not possible to self-host RHEV-M on the RHEV-H hosts. Already connected clients should not be disconnected in the case of RHEV-M failing (to be tested).

Red Hat is working on an Linux-Only RHEV-M, as you can read in a comment from a Red Hat employee in a earlier post.

The same comment talks about the WAN problems to be solved in near future.

Conclusion
Frankly: From my point of view, RHEV is not yet “enterprise ready”, this is because of the drawbacks mentioned above, especially the storage shortcomings and RHEV-M being a SPoF.

For smaller environments up to lets say ~50 servers or ~200 desktops it good enough.

Nevertheless: I think RHEV has a huge potential in the future. Compared to VMware ESX, the KVM’s technology is much better and scales better. At the moment VMware’s USP is its management software, not the hypervisor used. As soon as Red Hat ironed out the major drawbacks, I’ll expect a boost for RHEV.

Red Hat is keen to improve the product, I guess a lot of improvements will be annouced in 2011.

Have fun!

Deploying RHEL as ESX guests – Kickstarting or using ESX templates?

Wednesday, October 20th, 2010

Some time ago I asked my self the question if it is better to kickstart systems or working with ESX templates when deploying RHEL as ESX guests. I also had some discussions with friends working in the same industry. I tried it and came to the following conclusion:

Kickstart the systems is the way to go.

Pros:

  • Kickstarted Systems are already up-to-date after installation.
  • Proper SSH host keys. Using ESX templates ends up in having identical SSH host keys, from the security standpoint not usable, they need to be manually re-created.
  • Kickstarting means lean deployment, much less data needs to be transferred.
  • Very fast, kickstarted systems are deployed in ~3min instead of ~10min (depending on I/O and network performance).
  • Systems are being automatically registered @rhn or on a rhn-satellite with the help of cobbler snippets.
  • Better customization.

Cons:

  • The ESX template to used for kickstarting must have no disks configured, otherwise the whole nominal disk size is being transferred over the net.

When kickstarting virtual systems, only the data needed (the RPMs) is transferred. The best way is to have a “empty” ESX template, just with the network defined, but no disks. The reason for that is: ESX creates a checksum of the disk files, even if the disk is empty, the sparse disk files (in the case of “thin provisioning”) will be transferred over the net at its full nominal size.

When using ESX templates, after deploying, one needs to register the system manually and also manually update the system by invoking “yum -y update”. In contrary, kickstarted systems are always up to date automatically. To circumvent this fact, one needs to keep the templates up to date, it is a manual task which can not be automated easily.

Have fun!

Rumors about Novell and Suse Linux

Thursday, September 16th, 2010

There have been a lot of rumors that Novell will be sold. Latest rumors are that the Linux parts of Novell (former SuSE) will be split off and sold separately.

Since that got public, there are even more rumors: Which company will buy the Linux part of Novell?

  • Red Hat: Very unlikely, since the Monopolies and Mergers Commission in the EU and USA will most likely disagree because RHEL and SLES are already some kind of “duopoly”.
  • Microsoft: No way that the Monopolies and Mergers Commission will agree on this. On the other hand, MS has enough manpower to get on market with “MS-Linux” if they want to do so.
  • VMware: Would make sense, they can position them as a counterpart of Red Hat with its KVM initiative. Since VMware is a Windows-Shop, it is unlikely.
  • Oracle: They already have OEL, a clone of REHL, very unlikely
  • Hewlett-Packard: Would make sense because HP-UX is dead and HP does not have a future proof operating system anymore
  • IBM: They have AIX and IBM is involved in business with both, RHEL and SLES and others. I’m unsure if it would make sense to take over Suse. AIX is one of the two Unix Systems that will survive (maybe the only one since most probably Solaris 12 will never appear).

My wild guess is: Hewlett-Packard. Why? Monopolies and Mergers Commission will not be concerned, and HP does not have any competitive Operating System in its assets anymore.

Depending on who is buying the Linux business of Novell, we will have fun or not…

Lets take a breath,

Luc

Experiences with RHEL6 Beta 2.1

Friday, July 23rd, 2010

Like promised I’ll keep you updated on the RHEL6b2.1. The “official name” is not Beta2.1, it is “Beta 2 refresh”. Why not calling it Beta3? Anyway: The good news first: In contrary to the first release of Beta 2, it works fine again! The first release of Beta2 was quite crappy, it was not installable as a KVM guest. This was obviously due to severe bugs in some virtio drivers.

So, what are the news?

1. The bugs in the virtio drivers have been fixed, you can deploy RHEL6 in KVM environments again.
2. The vmware_ballooning driver has been backported.
3. A lot of minor bugs have been fixed, see the announcement.

Especially point two is cool, running RHEL6 in a VMware ESX environment does not necessarily need the vmware-tools installed anymore. RHEL6 now provides all three important vm-ware related drivers: The vmxnet3, vmware_ballooning and pvscsi. At the end of the day, this means one can dismiss the always-hated vmware-tools. A test of the behavior w/o vmware-tools by a ESX specialist is pending.

The alternative of vmware-tools are the open-vm-tools. This would add the benefit of controlled shutdown of the ESX guest with the vCenter tools. Since VMware does not provide (yet) RHEL 6 packages of the open-vm-tools I was unable to test it.

I made the same brief tests as I reported here. It seems that Red Hat is back on track, RHEL6b2.1 is reliable and not far away from being ready for production.

When can we expect a Beta3? Will there even be a next beta, or is Red Hat release a RC1 soon? There is still no published release schedule, all we know is “later this year”.

Anyway: Download Beta2.1 and test it, its a pretty cool release. If you find bugs, report them.

Have fun!

Red Hat’s virtualization strategy has redundancy – Quo vadis?

Thursday, May 27th, 2010

A couple of days there have been some reports that Red Hat will release a commercialized version of deltacloud, an abstraction layer for different kinds of virtualization technologies and clouds such as VMware, RHEV, Amazon EC2 etc.

Red Hat puts a lot of resources on virtualization, they maintain and/or sponsor multiple projects in parallel. The most important from my point of view is libvirt which is as well an abstraction layer for different virtulization technologies such as VMware, KVM, Xen and others. Libvirt and deltacloud are partially redundant.

It is not the only redundancy created by Red Hat. There is also O-virt “competing” with RHEV. Both are not tightly bound to RHN satellite or Spacewalk.

RHEV works with system templates similar to those at VMware. On the other hand: Koan, together with cobbler is a deployment software for virtual hosts and was recently bundled with RHN satellite.

Not all of those Red Hat virtualization projects are working well together. So the question arises: What is the strategy of having such redundancies of projects? Why not integrating all of this projects and glue them together?

Lots of questions…

Have fun!

Kernel questions about RHEL6, ESX support and experiences with F13a3

Monday, March 15th, 2010

Still no official informations

Red Hat is still refusing any questions about the features of RHEL 6 and its Linux Kernel. However: Since Vanilla Kernel 2.6.33 vmxnet3 and pvscsi is supported. Fedora 13 Alpha 3 is shipped with a derivate of Kernel 2.6.33.

I still hope that Red Hat is switching to 2.6.33 or back-porting the VMWare code to its 2.6.32 derivative Kernel as known by RHEL 6 Alpha 3.

Experiences with F13a3 so far

Installing F13a3 on a ESX guest – with RHEL5 as “supported Guest OS”  – and enabled vmxnet “enhanced” plus pvscsci as HBA was a smooth experience. No driver disk was needed, no dirty fixes. Just selecting vmxnet3 as NIC and PVSCSCI as disk HBA. Thats the way RHEL6 should work from my point of view.

RHEV vs. VMWare ESX

Since Red Hat released its visualization solution “RHEV”, VMWare and Red Hat are competitors. Is Red Hat willing to include ESX support in its Enterprise Products? My guess is to not to do so, but I’m open for surprises.

The goals

The goal on the long term is to switch from ESX to KVM. However, if you deployed a large ESX farm already and the management members are members of the “ESX-Church” it will be hard.

The mid-term goal is to get rid of those crappy VMWare tools. The current state of this “Tools” definitively proves that VMWare is a Windows shop and  does not take care about Linux virtualization.

Will we have fun? Depends on EMC and Red Hat….