Posts Tagged ‘KVM’

Automated disk partitioning on virtual machines with Cobbler

Saturday, December 15th, 2012

The default Cobbler Snippets just do simple auto partitioning. For a more sophisticated partition layout you need to know what kind of VM you are going to install. KVMs and RHEVs device name is /dev/vda, Xen uses /dev/xvda and ESX /dev/sda.

Luckily this can be figured out automatically, those different virtualization vendors are using its own MAC prefixes. So we can add two nice small Cobbler snippets to do the job. In this example, I call them hw-detect and partitioning.

hw-detect

#set $mac = $getVar('$mac_address_eth0')
#if $mac
#set $mac_prefix = $mac[0:8]
#if $mac_prefix == "00:1a:4a"
# This is a RHEV virtual machine
#set global $machinetype = 'kvm'

#else if $mac_prefix == "52:54:00"
# This is a KVM/Qemu virtual machine
#set global $machinetype='kvm'

#else if $mac_prefix == "00:16:3e"
# This is a XEN virtual machine
#set global $machinetype='xen'
#
#else if $mac_prefix == "00:50:56"
# This is a ESX virtual machine
#set global $machinetype = 'esx'

#else
# #This is a physical machine
#set global $machinetype = 'physical'
#end if
#end if

partitioning

#if $machinetype == 'kvm'
#set $disk='vda'
#else if $machinetype == 'xen'
#set $disk = 'xvda'
#else
#set $disk = 'sda'
#end if
# Lets install the system on /dev/$disk
part /boot      --fstype ext2 --size=250 --ondisk=$disk
part pv.0       --size=1 --grow --ondisk=$disk

volgroup vg_${name} pv.0

logvol /        --fstype ext4 --name=lv_root    --vgname=vg_${name} --size=4096
logvol /home    --fstype ext4 --name=lv_home    --vgname=vg_${name} --size=512 --fsoption=nosuid,nodev,noexec
logvol /tmp     --fstype ext4 --name=lv_tmp    --vgname=vg_${name} --size=1024 --fsoption=nosuid,nodev,noexec
logvol /var     --fstype ext4 --name=lv_var    --vgname=vg_${name} --size=2048 --fsoption=nosuid,nodev,noexec
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_${name} --size=2048

An additional “feature” of the partitioning Snippet is: It sets up the Volume Group name according to your systems name. This is the unofficial standard since quite some time. It also sets some more secure mount options. Review them carefully if they make sense for you and edit them as needed.

The next step is to configure your kickstart template.

Standalone Cobbler
On a standalone Cobbler server edit /var/lib/cobbler/kickstart/your-kick-start-template.ks

# Detect the used hardware type
$SNIPPET('hw-detect')
# Set up default partitioning
$SNIPPET('partitioning')

Bundled Cobbler
When using cobbler bundled with Spacewalk or Red Hat Satellite, you need to edit the Kickstart profile in the WebUI.


Navigate to Systems -> Kickstart -> Profile. Select the Kickstart profile to be modified -> System Details -> Partitioning.

Copy the two Snippets in /var/lib/cobbler/spacewalk/1, where 1 is representing your OrgId.

Alternatively you can edit them in the WebUI as well.

To check if all is working as expected, add a system to Cobbler using the Command Line Interface and have a look to the rendered Kickstart file. This can be easily done with cobbler system getks --name=blah.

Happy System installing….

Have fun :-)

Kernel 3.5.3 partially broken for virtualization

Wednesday, October 3rd, 2012

Some time ago, Fedora 17 got a Kernel update to 3.5.3-1. Since then, PXE booting virtual machines does not work anymore. It seems that it has not been fixed in the upstream Kernel, but only the 3.5 series of Kernels is affected.

A bug has been filed, but no fix is available. The only solution for now is to stick to Kernel 3.4.5-2. I’ve checked the Fedora annouce mailinglist, looks like there have been no grave bugfixes since then.

The bug only hits when you use PXE boot virtual machines with qemu-kvm. The virtual machine gets just paused, to find out the reason for it, you need to have a closer look to /var/log/libvirt/libvirtd.log. There you can read: “KVM: entry failed, hardware error 0x80000021“.

Someone proposed to use the emulate_invalid_guest_state=y parameter to the kvm_intel module, but according to a Ubuntu bugreport it fails too, but differently.

Hopefully a bug fix will be made available soon.

Having fun? Well, could be worse, could be better.

A review of RHEV

Sunday, February 27th, 2011

In the past few weeks I had the chance to have a closer look at the current release 2.2. The reason is that I’m working on a project using RHEL6 clients as virtual Desktops. For a proof-of-concept I’ve set up a test environment in the lab. Due to the lack of time I was not able to test every single feature.

After reading some docs, it was amazingly easily and quickly installed.

Test environment
The tests have been made on the following hardware:

  • 2 Xeon servers with a total of 8 cores and 24Gbyte on each host. OS is RHEL5.6. Those hosts are called RHEV-H (H like Hypervisor)
  • 1 Windows 2008R2 server (64bit) with 4Gbyte RAM as RHEV-M (M like management)
  • 1 dedicated NetApp filer serving storage over NFS
  • 1 RHEL 5.6 Server providing “cheap” NFS storage
  • 3 different networks connected to 3 NICs
  • 1 RHEL 5.6 “thin” client with spice-xpi as client
  • 1 Windows 2008R2 Server with spice-activex as client (actually on the management server)

VM’s (All 64bit):

  • 20 Virtual desktops with RHEL 6.0 clients using one core and 2Gbyte or RAM each
  • 2 Virtual servers with RHEL 6.0 server using 2 cores and 2Gbyte of RAM each
  • 2 Virtual servers with RHEL 5.6 server using 2 cores and 2Gbyte of RAM each

Management Portal
The Management interface is the central point of administration. It is not always as intuitive as expected. If you are new to RHEV, you can get confused. A big plus is the search functionality. One can search for storage, VMs, hosts and any other item in the environment.

User Portal
The user portal is very simple, lean and clean. Users see the machines that are assigned to them, they can power on and off the machines and can connect them to powered VMs, and the client appears on the desktop.

The user portal resides on the management server and is just the connection broker. As soon as a user connects to its desktop or server, the connection will be made directly to the spiced qemu-kvm running on one of the hypervisors.

Client
At the moment there are two different clients supported:

  • spice-activex with Internet Explorer 7+ on Windows XP, Vista and 7
  • spice-xpi with Firefox for RHEL5 and RHEL6

If you do not fear the work, you probably can get the spice-xpi client also running on Fedora 13 and other Linux distributions.

The client communicates with the VM over SPICE (Simplified Protocol for Independent Computing Environments). It can forward USB devices as well as sound and other multimedia stuff. For more informations about the protocol, please have a look at http://www.spice-space.org/home.html.

Storage
The first step to set up a RHEV environment is to define and set up the storage.

There are 3 types of storage:

  • ISO, where you upload your install-media for your virtual machines.
  • Data, where the VM images are stored.
  • Export, where I have no clue (yet) what it is good for.

Usually the storage is a NFS or iSCSI server, FC storage is also supported. The storage server creates snapshots of volumes for backing up the stuff.

Hosts (hypervisors)
You can either use RHEV-H as an “embedded” hypervisor which comes as a stripped-down RHEL5 distribution (the ISO is just about 75Mbyte IIRC) or you can set up RHEL5 and use it as a hypervisor.

First I tried the RHEV-H, it is manual work to install it. You need to burn a CDROM put it in your server and enter some stuff like network config etc. In the short time I did not tried to provision RHEV-H with cobbler. According to the documentation one can pass boot parameters to partially automate the installation.

Because I have a cobbler server handy, I decided to use RHEL5 as hypervisors. Just be sure you subscribe the system to the channels “rhel-x86_64-server-vt-5″ and rhel-x86_64-rhev-mgmt-agent-5. You also need to allow root to log in via SSH. You can change this later after the hosts is registered on the RHEV-M server.

After the setup of the RHEL5 servers you just need to tell RHEV-M that there is a new host available. Tell RHEV-M about the root password and all the needed additional software gets installed automatically.

That’s a pretty lean procedure to install and set up hosts. Hopefully there will soon be a way to fully automate this, w/o using MS Powershell.

The technologies on the hosts are the well known stable and mature KVM and qemu-kvm. There are also parts of o-virt being used.

Networking
I do not know how many networks you can add, but I think enough even for large environments. The network configuration part in RHEV-M shows a list of available Ethernet ports on each host. Just assign them a static network and IP address and you’re done. Make sure you define the network you want to add on all hosts in a particular datacenter.

Ensure that firewalls between the users and VM’s networks are configured accordingly to avoid connection problems.

Sizing and performance

  • Memory overcommitment
    Thanks to KSM (Kernel Samepage Merging) one can quite overcommit the memory. Ensure you have a LOT of swap on your hosts. Why? It takes some time until KSM kicks in and frees memory pages. During the collection of those pages, swap space is needed to prevent getting a visit by the OOM killer.

    I once faced a complete crash when putting one host in maintenance mode and all VMs have been migrated to the second host. That’s exactly why you should have LOTs of swap spaces to survive when using memory over-commitment. During the period of KSM searching for same pages, performance is degraded.

    Under normal circumstances, this will never happen since the virtual hosts are load balanced. This means VM’s are distributed to have the optimal performance.

  • CPU overcommitment
    This very depends on your users computing needs. For normal desktops and servers you can overcommit at least 200% of the CPUs and the performance is still fine. For CPU intensive workload it is not recommended to over commit CPUs.
  • Number of hosts
    Its recommended to start with at least three hosts to avoid temporary performance penalties if you need to put a host into maintenance mode. (See also “Memory overcommitment”).

    The performance is surprisingly good, the user experience is nearly the same as on physical desktops. For servers, KVM is already known for its good performance.

Drawbacks
Unfortunately you can not hot-add or hot-remove virtual CPU’s, NICs and storage. Despite of being supported by the underlying technologies used, your VM must be powered off to add or remove resources.

Another drawback is the lack of different storage classes, this makes it actually impossible to use the product in large enterprise environments. Since it makes no sense to back up swap drives, it is waste of backup space.
It is also not possible to have different storage for different needs. Lets say a bulk RAID5 consisting of cheap 2Tbyte SATA disks for operating systems and a RAID10 consisting of fast 15k SAS or FC disks for data and swap. It is also not possible to life-migrate VMs from one storage to another.

As of today, SPICE is only useful on LAN networks. Depending on the work load it can demand lots of bandwidth available. For usual office workload, 1 Mbps per connection is fine. A WAN optimized version is under development and should be released “later this year”.

The two supported clients (spice-xpi and spice-activex) can be problematic when the clients are not in an environment controlled by you. Why? Lots of enterprises are preventing its users to install Active X applications and Firefox Add-On’s for security reasons. It would be better to have some kind of “fat client” which does not need to be installed, or a Java Applet which does the job.

At the moment, it is still required to have a Windows 2008R2 server installed and use MS technologies such as MSSQL, DotNET, Powershell and Active Directory. Because you can not manage RHEV-M with a satellite, one needs to download and install updates manually.

RHEV-M is not only the management server, it is also the connection broker for clients connecting. RHEV-M is a single point of failure. Maybe one can build a MS-Cluster with it, no clue. It is also not possible to self-host RHEV-M on the RHEV-H hosts. Already connected clients should not be disconnected in the case of RHEV-M failing (to be tested).

Red Hat is working on an Linux-Only RHEV-M, as you can read in a comment from a Red Hat employee in a earlier post.

The same comment talks about the WAN problems to be solved in near future.

Conclusion
Frankly: From my point of view, RHEV is not yet “enterprise ready”, this is because of the drawbacks mentioned above, especially the storage shortcomings and RHEV-M being a SPoF.

For smaller environments up to lets say ~50 servers or ~200 desktops it good enough.

Nevertheless: I think RHEV has a huge potential in the future. Compared to VMware ESX, the KVM’s technology is much better and scales better. At the moment VMware’s USP is its management software, not the hypervisor used. As soon as Red Hat ironed out the major drawbacks, I’ll expect a boost for RHEV.

Red Hat is keen to improve the product, I guess a lot of improvements will be annouced in 2011.

Have fun!

Spice and RHEV, a RHCE goes MCSE

Tuesday, January 11th, 2011

I’m currently working in a project which includes some virtual Linux desktops. The desktop of choice is RHEL6.

How to bring a Linux desktop via WAN to a thin client? VNC -> are you nuts? Remote X11 over SSH -> WAN = no go. NX -> another vendor involved. SPICE -> Spicy! But: Spice over WAN? To be tested…

SPICE is the protocol used by RHEV (Red Hat Enterprise Virtualization). Some time ago I had the chance to test this stuff @Red Hat in Munich. The experience was nice, it is comparable to vSphere, but it only works with MS Internet explorer due to ActiveX and .Net stuff.

The management software needs to be installed on a Windows 2008R2 server. The database to be used is – you guess it – MS SQL. Users are authenticated either by Active Directory (Not generic LDAP!) or local Windows Users. Holy cow!

At first, when I got this product presented by Red Hat I was LOL. Now, it seems that I need to refresh my Windows knowledge because it seems to be the only product capable to provide enterprise ready Linux desktop virtualization. I’m crying :-(

At least the hypervisor used is not MS HyperV, it is KVM based on RHEL5, to replaced with RHEL6 in the future.

There is some light at the end of the tunnel: Red Hat is working on a replacement of the Windows-bound stuff. It will be replaced with some JBOSS and Java stuff. The database will probably be PostgreSQL. It will take some time to develop it before it will be ready for production.

Since Red Hat is opensourcing all (or most) of its products, it would be great to get in touch with the upstream project (release early, release often).

In meantime I need to build up knowledge about Windows Server 2008R2, Active Directory, MS SQL Server and DotNet.

Having fun?

Rumors about Novell and Suse Linux

Thursday, September 16th, 2010

There have been a lot of rumors that Novell will be sold. Latest rumors are that the Linux parts of Novell (former SuSE) will be split off and sold separately.

Since that got public, there are even more rumors: Which company will buy the Linux part of Novell?

  • Red Hat: Very unlikely, since the Monopolies and Mergers Commission in the EU and USA will most likely disagree because RHEL and SLES are already some kind of “duopoly”.
  • Microsoft: No way that the Monopolies and Mergers Commission will agree on this. On the other hand, MS has enough manpower to get on market with “MS-Linux” if they want to do so.
  • VMware: Would make sense, they can position them as a counterpart of Red Hat with its KVM initiative. Since VMware is a Windows-Shop, it is unlikely.
  • Oracle: They already have OEL, a clone of REHL, very unlikely
  • Hewlett-Packard: Would make sense because HP-UX is dead and HP does not have a future proof operating system anymore
  • IBM: They have AIX and IBM is involved in business with both, RHEL and SLES and others. I’m unsure if it would make sense to take over Suse. AIX is one of the two Unix Systems that will survive (maybe the only one since most probably Solaris 12 will never appear).

My wild guess is: Hewlett-Packard. Why? Monopolies and Mergers Commission will not be concerned, and HP does not have any competitive Operating System in its assets anymore.

Depending on who is buying the Linux business of Novell, we will have fun or not…

Lets take a breath,

Luc

Experiences with RHEL6 Beta 2.1

Friday, July 23rd, 2010

Like promised I’ll keep you updated on the RHEL6b2.1. The “official name” is not Beta2.1, it is “Beta 2 refresh”. Why not calling it Beta3? Anyway: The good news first: In contrary to the first release of Beta 2, it works fine again! The first release of Beta2 was quite crappy, it was not installable as a KVM guest. This was obviously due to severe bugs in some virtio drivers.

So, what are the news?

1. The bugs in the virtio drivers have been fixed, you can deploy RHEL6 in KVM environments again.
2. The vmware_ballooning driver has been backported.
3. A lot of minor bugs have been fixed, see the announcement.

Especially point two is cool, running RHEL6 in a VMware ESX environment does not necessarily need the vmware-tools installed anymore. RHEL6 now provides all three important vm-ware related drivers: The vmxnet3, vmware_ballooning and pvscsi. At the end of the day, this means one can dismiss the always-hated vmware-tools. A test of the behavior w/o vmware-tools by a ESX specialist is pending.

The alternative of vmware-tools are the open-vm-tools. This would add the benefit of controlled shutdown of the ESX guest with the vCenter tools. Since VMware does not provide (yet) RHEL 6 packages of the open-vm-tools I was unable to test it.

I made the same brief tests as I reported here. It seems that Red Hat is back on track, RHEL6b2.1 is reliable and not far away from being ready for production.

When can we expect a Beta3? Will there even be a next beta, or is Red Hat release a RC1 soon? There is still no published release schedule, all we know is “later this year”.

Anyway: Download Beta2.1 and test it, its a pretty cool release. If you find bugs, report them.

Have fun!

Red Hat’s virtualization strategy has redundancy – Quo vadis?

Thursday, May 27th, 2010

A couple of days there have been some reports that Red Hat will release a commercialized version of deltacloud, an abstraction layer for different kinds of virtualization technologies and clouds such as VMware, RHEV, Amazon EC2 etc.

Red Hat puts a lot of resources on virtualization, they maintain and/or sponsor multiple projects in parallel. The most important from my point of view is libvirt which is as well an abstraction layer for different virtulization technologies such as VMware, KVM, Xen and others. Libvirt and deltacloud are partially redundant.

It is not the only redundancy created by Red Hat. There is also O-virt “competing” with RHEV. Both are not tightly bound to RHN satellite or Spacewalk.

RHEV works with system templates similar to those at VMware. On the other hand: Koan, together with cobbler is a deployment software for virtual hosts and was recently bundled with RHN satellite.

Not all of those Red Hat virtualization projects are working well together. So the question arises: What is the strategy of having such redundancies of projects? Why not integrating all of this projects and glue them together?

Lots of questions…

Have fun!

KVM supports live migration between CPUs with different features

Saturday, May 15th, 2010

The video is a bit old, it is from November 2008. But it is still quite interesting to see and discuss about it. With KVM you can upgrade your farm of servers easy, it does not matter if the new servers have CPUs with new features or not. I’m not sure if you can do this with ESX, I guess not, you probably need to migrate them shut down.

Have fun!

Kernel questions about RHEL6, ESX support and experiences with F13a3

Monday, March 15th, 2010

Still no official informations

Red Hat is still refusing any questions about the features of RHEL 6 and its Linux Kernel. However: Since Vanilla Kernel 2.6.33 vmxnet3 and pvscsi is supported. Fedora 13 Alpha 3 is shipped with a derivate of Kernel 2.6.33.

I still hope that Red Hat is switching to 2.6.33 or back-porting the VMWare code to its 2.6.32 derivative Kernel as known by RHEL 6 Alpha 3.

Experiences with F13a3 so far

Installing F13a3 on a ESX guest – with RHEL5 as “supported Guest OS”  – and enabled vmxnet “enhanced” plus pvscsci as HBA was a smooth experience. No driver disk was needed, no dirty fixes. Just selecting vmxnet3 as NIC and PVSCSCI as disk HBA. Thats the way RHEL6 should work from my point of view.

RHEV vs. VMWare ESX

Since Red Hat released its visualization solution “RHEV”, VMWare and Red Hat are competitors. Is Red Hat willing to include ESX support in its Enterprise Products? My guess is to not to do so, but I’m open for surprises.

The goals

The goal on the long term is to switch from ESX to KVM. However, if you deployed a large ESX farm already and the management members are members of the “ESX-Church” it will be hard.

The mid-term goal is to get rid of those crappy VMWare tools. The current state of this “Tools” definitively proves that VMWare is a Windows shop and  does not take care about Linux virtualization.

Will we have fun? Depends on EMC and Red Hat….

Where the heck is RHEL6?

Sunday, February 28th, 2010

Release cycle slowed down

In the past Red Hat has released a new version of its Red Hat Enterprise Linux (RHEL) roughly every two years. RHEL5 was released on march 2007. Compared to the past release cycle, RHEL6 is overdue since one year.

Official information

There is only little known about the upcoming features of RHEL6. On the Red Hat Summit 2009, there was a presentation held by Tim Burke which gives just some hints that RHEL6 is actually approaching, see http://www.redhat.com/f/pdf/summit/tburke_1050_rhel_roadmap.pdf. Quoting a note on the slide about RHEL6: Note: this information is high level planning projection and does not constitute formal product commitment.

My conclusion is that Red Hat seems to be unsure about the features planned for its upcoming Enterprise Product.

Another interesting quote from the same presentation is: RHEL6 feature previews – appearing in Fedora 11 & 12. Meanwhile, almost a half year later, Fedora 13 is approaching and still no sign of RHEL6, no schedule, no official feature list. Looking at the feature list if Fedora 13 https://fedoraproject.org/wiki/Releases/13/FeatureList, nothing special so far. It seems that the pace of development has been slowed down a bit to put more energy into stabilizing F11/F12 to RHEL6.

Inofficial information

When carefully watching git commit logs and bugzilla entries, there are some small traces of RHEL6.

There is almost no information leaking for the topic. The only valuable unofficial information is from bug #562766 which was reported by a Red Hat employee on 2010-02-08.  This bug states RHEL6 Alpha3!  Quoting a comment from the same employee: Upgrading rhel6.0 kernel to 2.6.32-14.el6 fixes the issue.

this brings me to a wild guess for a release schedule:

  • February 2010: Alpha3
  • March 2010: Beta1
  • April 2010: Beta2
  • May or June 2010: GA [Update: End of June/Early July seems to be more likely, since the Red Hat Summit will be held June 22-25 2010]

My wish list for RHEL6

  • Kernel based on version 2.6.33 instead of 2.6.32 as in Alpha3, since there are a lot of improvements when using RHEL as a VMware ESX guest.
  • Default installation with a smaller footprint
  • Cleanup of insane package dependencies
  • BusLogic drivers included as the vanilla Kernel ships it since years

The question remains

Where the heck is RHEL6? One reason could be that the focus on RHEL6 seems to be virtualization and system management. Since approximately two years, in this domain the pace of the development had increased a lot, maybe too much. Think about KVM, libvirt, virt-manager, o-virt. All of those projects are sponsored by Red Hat and included in F12. So one of the reason of the late release of RHEL6 can be problems in stabilizing those virtualization products to be enterprise-ready.

Why Red Hat makes its customers angry with late releases and no roadmap

First of all, RHEL products have a life-cycle of seven years. RHEL5 was released on march 2007. Assuming RHEL6 will be GA on May 2010. Add a few months before it is supported by ISVs such as SAP, Oracle etc. Customers can begin with deploying RHEL6 on lets say August 2010. Until then, RHEL5 has almost reached half of its life-cycle: 3 1/2 years. Means: A SAP system deployed on July 2010 is out of support some 3 years and nine months later. For an enterprise product this not acceptable! Red Hat should think about a life-cycle like “Next-Release plus five years“, this would make system deployment and company-internal life-cycle management easier.

Keeping its customers in the dark with no official roadmap at all is just bad behavior and indeed not customer friendly.