Archive for the ‘Virtualization’ Category

Automated disk partitioning on virtual machines with Cobbler

Saturday, December 15th, 2012

The default Cobbler Snippets just do simple auto partitioning. For a more sophisticated partition layout you need to know what kind of VM you are going to install. KVMs and RHEVs device name is /dev/vda, Xen uses /dev/xvda and ESX /dev/sda.

Luckily this can be figured out automatically, those different virtualization vendors are using its own MAC prefixes. So we can add two nice small Cobbler snippets to do the job. In this example, I call them hw-detect and partitioning.

hw-detect

#set $mac = $getVar('$mac_address_eth0')
#if $mac
#set $mac_prefix = $mac[0:8]
#if $mac_prefix == "00:1a:4a"
# This is a RHEV virtual machine
#set global $machinetype = 'kvm'

#else if $mac_prefix == "52:54:00"
# This is a KVM/Qemu virtual machine
#set global $machinetype='kvm'

#else if $mac_prefix == "00:16:3e"
# This is a XEN virtual machine
#set global $machinetype='xen'
#
#else if $mac_prefix == "00:50:56"
# This is a ESX virtual machine
#set global $machinetype = 'esx'

#else
# #This is a physical machine
#set global $machinetype = 'physical'
#end if
#end if

partitioning

#if $machinetype == 'kvm'
#set $disk='vda'
#else if $machinetype == 'xen'
#set $disk = 'xvda'
#else
#set $disk = 'sda'
#end if
# Lets install the system on /dev/$disk
part /boot      --fstype ext2 --size=250 --ondisk=$disk
part pv.0       --size=1 --grow --ondisk=$disk

volgroup vg_${name} pv.0

logvol /        --fstype ext4 --name=lv_root    --vgname=vg_${name} --size=4096
logvol /home    --fstype ext4 --name=lv_home    --vgname=vg_${name} --size=512 --fsoption=nosuid,nodev,noexec
logvol /tmp     --fstype ext4 --name=lv_tmp    --vgname=vg_${name} --size=1024 --fsoption=nosuid,nodev,noexec
logvol /var     --fstype ext4 --name=lv_var    --vgname=vg_${name} --size=2048 --fsoption=nosuid,nodev,noexec
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_${name} --size=2048

An additional “feature” of the partitioning Snippet is: It sets up the Volume Group name according to your systems name. This is the unofficial standard since quite some time. It also sets some more secure mount options. Review them carefully if they make sense for you and edit them as needed.

The next step is to configure your kickstart template.

Standalone Cobbler
On a standalone Cobbler server edit /var/lib/cobbler/kickstart/your-kick-start-template.ks

# Detect the used hardware type
$SNIPPET('hw-detect')
# Set up default partitioning
$SNIPPET('partitioning')

Bundled Cobbler
When using cobbler bundled with Spacewalk or Red Hat Satellite, you need to edit the Kickstart profile in the WebUI.


Navigate to Systems -> Kickstart -> Profile. Select the Kickstart profile to be modified -> System Details -> Partitioning.

Copy the two Snippets in /var/lib/cobbler/spacewalk/1, where 1 is representing your OrgId.

Alternatively you can edit them in the WebUI as well.

To check if all is working as expected, add a system to Cobbler using the Command Line Interface and have a look to the rendered Kickstart file. This can be easily done with cobbler system getks --name=blah.

Happy System installing….

Have fun :-)

RHEV 3.1 – an overview about the new features

Sunday, December 9th, 2012
RHEV-M

RHEV-M

Recently Red Hat announced the public availability of RHEV 3.1.

Finally, no more Windows needed for the whole software stack :-)

In 3.0, the new webadmin interface was already inncluded, as a tech preview and had its problems. Now with 3.1 its working great and looks neat. In contrary to 3.0, it is now listening on the standard ports 80 and 443. This will probably help users in organizations with strict proxy policies and setting.

So what else is new?

The supported number of virtual CPUs in a guest is now ridiculous 160, and RAM per guest is at ridiculous two Terabytes. But this are the least import updates.

Especially on the storage side, a lot of effort has been done and long missing features integrated.

From my point of view, the most important new feature is the possibility to have disks from more than one Storage Domain attached to a virtual machine. This would allow to install the Operating system to cheap SATA storage while data disks are super fast SSDs.

There is also support for live snapshots, but snapshots are (as on other platforms) kind of problematic because they are COW (Copy-On-Write). This can lead to I/O performance problems. Snapshots are a cool feature for i.e. taking a snapshot before updating software etc. Be sure you remove the snapshot afterwards if you want to keep a good I/O performance.

You now can use DirectLUN directly from the GUI without the usage of hooks. DirectLUN allows to attach FibreChannel and iSCSI LUNs directly to a Virtual Machine. This is great when you want to use shared filesystems such as GFS.

Another nice feature is Live Storage Migration which is a technical preview, means: Unsupported for the moment. It probably will be supported in a later version. Storage live migration is a nice feature when you need to free up some space on a storage domain and you can not shut down a VM. Be sure to power-cycle the VM in question as soon as your SLA allows it, to get rid of the Snapshot (COW here again).

If you want to script stuff or you are too lazy to open a brower, there is now a CLI available. Have a look to the documentation.

If you want to integrate RHEV deeper into your existing infrastructure, such as RHN Satellite, Cobbler, Your-super-duper-CMDB or IaaS/PaaS broker, there are two different APIs available. For the XML lovers, there is the previously known RestAPI which has some performance improvements. For the XML haters, there is now a native Python API which allows to to access RHEV entities directly as objects in your Python code. For both APIs, have a look to the Documentation.

I personally like the Python API, because a lot of other Red Hat infrastructure products come with Python APIs. So it is very easy to integrate those software pieces.

Under the hood, it is now powered by JBoss EAP6 instead of version 5. To be able to connect to standard ports 80 and 443, there is an Apache httpd with mod_proxy_ajp.

Have fun :-)

Kernel 3.5.3 partially broken for virtualization

Wednesday, October 3rd, 2012

Some time ago, Fedora 17 got a Kernel update to 3.5.3-1. Since then, PXE booting virtual machines does not work anymore. It seems that it has not been fixed in the upstream Kernel, but only the 3.5 series of Kernels is affected.

A bug has been filed, but no fix is available. The only solution for now is to stick to Kernel 3.4.5-2. I’ve checked the Fedora annouce mailinglist, looks like there have been no grave bugfixes since then.

The bug only hits when you use PXE boot virtual machines with qemu-kvm. The virtual machine gets just paused, to find out the reason for it, you need to have a closer look to /var/log/libvirt/libvirtd.log. There you can read: “KVM: entry failed, hardware error 0x80000021“.

Someone proposed to use the emulate_invalid_guest_state=y parameter to the kvm_intel module, but according to a Ubuntu bugreport it fails too, but differently.

Hopefully a bug fix will be made available soon.

Having fun? Well, could be worse, could be better.

A review of RHEV

Sunday, February 27th, 2011

In the past few weeks I had the chance to have a closer look at the current release 2.2. The reason is that I’m working on a project using RHEL6 clients as virtual Desktops. For a proof-of-concept I’ve set up a test environment in the lab. Due to the lack of time I was not able to test every single feature.

After reading some docs, it was amazingly easily and quickly installed.

Test environment
The tests have been made on the following hardware:

  • 2 Xeon servers with a total of 8 cores and 24Gbyte on each host. OS is RHEL5.6. Those hosts are called RHEV-H (H like Hypervisor)
  • 1 Windows 2008R2 server (64bit) with 4Gbyte RAM as RHEV-M (M like management)
  • 1 dedicated NetApp filer serving storage over NFS
  • 1 RHEL 5.6 Server providing “cheap” NFS storage
  • 3 different networks connected to 3 NICs
  • 1 RHEL 5.6 “thin” client with spice-xpi as client
  • 1 Windows 2008R2 Server with spice-activex as client (actually on the management server)

VM’s (All 64bit):

  • 20 Virtual desktops with RHEL 6.0 clients using one core and 2Gbyte or RAM each
  • 2 Virtual servers with RHEL 6.0 server using 2 cores and 2Gbyte of RAM each
  • 2 Virtual servers with RHEL 5.6 server using 2 cores and 2Gbyte of RAM each

Management Portal
The Management interface is the central point of administration. It is not always as intuitive as expected. If you are new to RHEV, you can get confused. A big plus is the search functionality. One can search for storage, VMs, hosts and any other item in the environment.

User Portal
The user portal is very simple, lean and clean. Users see the machines that are assigned to them, they can power on and off the machines and can connect them to powered VMs, and the client appears on the desktop.

The user portal resides on the management server and is just the connection broker. As soon as a user connects to its desktop or server, the connection will be made directly to the spiced qemu-kvm running on one of the hypervisors.

Client
At the moment there are two different clients supported:

  • spice-activex with Internet Explorer 7+ on Windows XP, Vista and 7
  • spice-xpi with Firefox for RHEL5 and RHEL6

If you do not fear the work, you probably can get the spice-xpi client also running on Fedora 13 and other Linux distributions.

The client communicates with the VM over SPICE (Simplified Protocol for Independent Computing Environments). It can forward USB devices as well as sound and other multimedia stuff. For more informations about the protocol, please have a look at http://www.spice-space.org/home.html.

Storage
The first step to set up a RHEV environment is to define and set up the storage.

There are 3 types of storage:

  • ISO, where you upload your install-media for your virtual machines.
  • Data, where the VM images are stored.
  • Export, where I have no clue (yet) what it is good for.

Usually the storage is a NFS or iSCSI server, FC storage is also supported. The storage server creates snapshots of volumes for backing up the stuff.

Hosts (hypervisors)
You can either use RHEV-H as an “embedded” hypervisor which comes as a stripped-down RHEL5 distribution (the ISO is just about 75Mbyte IIRC) or you can set up RHEL5 and use it as a hypervisor.

First I tried the RHEV-H, it is manual work to install it. You need to burn a CDROM put it in your server and enter some stuff like network config etc. In the short time I did not tried to provision RHEV-H with cobbler. According to the documentation one can pass boot parameters to partially automate the installation.

Because I have a cobbler server handy, I decided to use RHEL5 as hypervisors. Just be sure you subscribe the system to the channels “rhel-x86_64-server-vt-5″ and rhel-x86_64-rhev-mgmt-agent-5. You also need to allow root to log in via SSH. You can change this later after the hosts is registered on the RHEV-M server.

After the setup of the RHEL5 servers you just need to tell RHEV-M that there is a new host available. Tell RHEV-M about the root password and all the needed additional software gets installed automatically.

That’s a pretty lean procedure to install and set up hosts. Hopefully there will soon be a way to fully automate this, w/o using MS Powershell.

The technologies on the hosts are the well known stable and mature KVM and qemu-kvm. There are also parts of o-virt being used.

Networking
I do not know how many networks you can add, but I think enough even for large environments. The network configuration part in RHEV-M shows a list of available Ethernet ports on each host. Just assign them a static network and IP address and you’re done. Make sure you define the network you want to add on all hosts in a particular datacenter.

Ensure that firewalls between the users and VM’s networks are configured accordingly to avoid connection problems.

Sizing and performance

  • Memory overcommitment
    Thanks to KSM (Kernel Samepage Merging) one can quite overcommit the memory. Ensure you have a LOT of swap on your hosts. Why? It takes some time until KSM kicks in and frees memory pages. During the collection of those pages, swap space is needed to prevent getting a visit by the OOM killer.

    I once faced a complete crash when putting one host in maintenance mode and all VMs have been migrated to the second host. That’s exactly why you should have LOTs of swap spaces to survive when using memory over-commitment. During the period of KSM searching for same pages, performance is degraded.

    Under normal circumstances, this will never happen since the virtual hosts are load balanced. This means VM’s are distributed to have the optimal performance.

  • CPU overcommitment
    This very depends on your users computing needs. For normal desktops and servers you can overcommit at least 200% of the CPUs and the performance is still fine. For CPU intensive workload it is not recommended to over commit CPUs.
  • Number of hosts
    Its recommended to start with at least three hosts to avoid temporary performance penalties if you need to put a host into maintenance mode. (See also “Memory overcommitment”).

    The performance is surprisingly good, the user experience is nearly the same as on physical desktops. For servers, KVM is already known for its good performance.

Drawbacks
Unfortunately you can not hot-add or hot-remove virtual CPU’s, NICs and storage. Despite of being supported by the underlying technologies used, your VM must be powered off to add or remove resources.

Another drawback is the lack of different storage classes, this makes it actually impossible to use the product in large enterprise environments. Since it makes no sense to back up swap drives, it is waste of backup space.
It is also not possible to have different storage for different needs. Lets say a bulk RAID5 consisting of cheap 2Tbyte SATA disks for operating systems and a RAID10 consisting of fast 15k SAS or FC disks for data and swap. It is also not possible to life-migrate VMs from one storage to another.

As of today, SPICE is only useful on LAN networks. Depending on the work load it can demand lots of bandwidth available. For usual office workload, 1 Mbps per connection is fine. A WAN optimized version is under development and should be released “later this year”.

The two supported clients (spice-xpi and spice-activex) can be problematic when the clients are not in an environment controlled by you. Why? Lots of enterprises are preventing its users to install Active X applications and Firefox Add-On’s for security reasons. It would be better to have some kind of “fat client” which does not need to be installed, or a Java Applet which does the job.

At the moment, it is still required to have a Windows 2008R2 server installed and use MS technologies such as MSSQL, DotNET, Powershell and Active Directory. Because you can not manage RHEV-M with a satellite, one needs to download and install updates manually.

RHEV-M is not only the management server, it is also the connection broker for clients connecting. RHEV-M is a single point of failure. Maybe one can build a MS-Cluster with it, no clue. It is also not possible to self-host RHEV-M on the RHEV-H hosts. Already connected clients should not be disconnected in the case of RHEV-M failing (to be tested).

Red Hat is working on an Linux-Only RHEV-M, as you can read in a comment from a Red Hat employee in a earlier post.

The same comment talks about the WAN problems to be solved in near future.

Conclusion
Frankly: From my point of view, RHEV is not yet “enterprise ready”, this is because of the drawbacks mentioned above, especially the storage shortcomings and RHEV-M being a SPoF.

For smaller environments up to lets say ~50 servers or ~200 desktops it good enough.

Nevertheless: I think RHEV has a huge potential in the future. Compared to VMware ESX, the KVM’s technology is much better and scales better. At the moment VMware’s USP is its management software, not the hypervisor used. As soon as Red Hat ironed out the major drawbacks, I’ll expect a boost for RHEV.

Red Hat is keen to improve the product, I guess a lot of improvements will be annouced in 2011.

Have fun!

Spice and RHEV, a RHCE goes MCSE

Tuesday, January 11th, 2011

I’m currently working in a project which includes some virtual Linux desktops. The desktop of choice is RHEL6.

How to bring a Linux desktop via WAN to a thin client? VNC -> are you nuts? Remote X11 over SSH -> WAN = no go. NX -> another vendor involved. SPICE -> Spicy! But: Spice over WAN? To be tested…

SPICE is the protocol used by RHEV (Red Hat Enterprise Virtualization). Some time ago I had the chance to test this stuff @Red Hat in Munich. The experience was nice, it is comparable to vSphere, but it only works with MS Internet explorer due to ActiveX and .Net stuff.

The management software needs to be installed on a Windows 2008R2 server. The database to be used is – you guess it – MS SQL. Users are authenticated either by Active Directory (Not generic LDAP!) or local Windows Users. Holy cow!

At first, when I got this product presented by Red Hat I was LOL. Now, it seems that I need to refresh my Windows knowledge because it seems to be the only product capable to provide enterprise ready Linux desktop virtualization. I’m crying :-(

At least the hypervisor used is not MS HyperV, it is KVM based on RHEL5, to replaced with RHEL6 in the future.

There is some light at the end of the tunnel: Red Hat is working on a replacement of the Windows-bound stuff. It will be replaced with some JBOSS and Java stuff. The database will probably be PostgreSQL. It will take some time to develop it before it will be ready for production.

Since Red Hat is opensourcing all (or most) of its products, it would be great to get in touch with the upstream project (release early, release often).

In meantime I need to build up knowledge about Windows Server 2008R2, Active Directory, MS SQL Server and DotNet.

Having fun?

Deploying RHEL as ESX guests – Kickstarting or using ESX templates?

Wednesday, October 20th, 2010

Some time ago I asked my self the question if it is better to kickstart systems or working with ESX templates when deploying RHEL as ESX guests. I also had some discussions with friends working in the same industry. I tried it and came to the following conclusion:

Kickstart the systems is the way to go.

Pros:

  • Kickstarted Systems are already up-to-date after installation.
  • Proper SSH host keys. Using ESX templates ends up in having identical SSH host keys, from the security standpoint not usable, they need to be manually re-created.
  • Kickstarting means lean deployment, much less data needs to be transferred.
  • Very fast, kickstarted systems are deployed in ~3min instead of ~10min (depending on I/O and network performance).
  • Systems are being automatically registered @rhn or on a rhn-satellite with the help of cobbler snippets.
  • Better customization.

Cons:

  • The ESX template to used for kickstarting must have no disks configured, otherwise the whole nominal disk size is being transferred over the net.

When kickstarting virtual systems, only the data needed (the RPMs) is transferred. The best way is to have a “empty” ESX template, just with the network defined, but no disks. The reason for that is: ESX creates a checksum of the disk files, even if the disk is empty, the sparse disk files (in the case of “thin provisioning”) will be transferred over the net at its full nominal size.

When using ESX templates, after deploying, one needs to register the system manually and also manually update the system by invoking “yum -y update”. In contrary, kickstarted systems are always up to date automatically. To circumvent this fact, one needs to keep the templates up to date, it is a manual task which can not be automated easily.

Have fun!

Experiences with RHEL6 Beta 2.1

Friday, July 23rd, 2010

Like promised I’ll keep you updated on the RHEL6b2.1. The “official name” is not Beta2.1, it is “Beta 2 refresh”. Why not calling it Beta3? Anyway: The good news first: In contrary to the first release of Beta 2, it works fine again! The first release of Beta2 was quite crappy, it was not installable as a KVM guest. This was obviously due to severe bugs in some virtio drivers.

So, what are the news?

1. The bugs in the virtio drivers have been fixed, you can deploy RHEL6 in KVM environments again.
2. The vmware_ballooning driver has been backported.
3. A lot of minor bugs have been fixed, see the announcement.

Especially point two is cool, running RHEL6 in a VMware ESX environment does not necessarily need the vmware-tools installed anymore. RHEL6 now provides all three important vm-ware related drivers: The vmxnet3, vmware_ballooning and pvscsi. At the end of the day, this means one can dismiss the always-hated vmware-tools. A test of the behavior w/o vmware-tools by a ESX specialist is pending.

The alternative of vmware-tools are the open-vm-tools. This would add the benefit of controlled shutdown of the ESX guest with the vCenter tools. Since VMware does not provide (yet) RHEL 6 packages of the open-vm-tools I was unable to test it.

I made the same brief tests as I reported here. It seems that Red Hat is back on track, RHEL6b2.1 is reliable and not far away from being ready for production.

When can we expect a Beta3? Will there even be a next beta, or is Red Hat release a RC1 soon? There is still no published release schedule, all we know is “later this year”.

Anyway: Download Beta2.1 and test it, its a pretty cool release. If you find bugs, report them.

Have fun!

What is possibly going into RHEL6 GA and what is not

Tuesday, June 1st, 2010

As I wrote different times before, RHEL6 is going to have a Kernel based on upstreams 2.6.32 Kernel. Meanwhile Linus Torvalds and his fellows released 2.6.34. Since then – from a System Engineers Point of view – there have some “minor” changes which are affecting the daily work in enterprise environments.

I think that Red Hat is aware that RHEL6 is one of its most important releases made so far. RHEL6 Beta-Testers have acknowledged that this is one of the best Linux distributions made so far.

So lets have a look to http://bit.ly/98yNsk (https://bugzilla.redhat.com search for RHEL6 select all states, sort by Bug-ID and having RFE (Request For Enhancement) in Summary).

Unrar
I requested to add “unrar” to RHEL, unfortunatly they refused because of the strange license of unrar. This is really not understandable, because *ALL* major Linux distros such as SLES, Debian, Ubuntu are providing a package for it. Red Hat think (and they are right) it is a “unfree” license. From my point of view it does not hurt because nobody is forced to use its libs in own software. Unfortunately SAP distributes a lot of software components in RAR-compressed files, this is a problem.

virtio net/vhost net speed enhancements from upstream kernel
This was reported as bug #593158 and later appeared as #595287. Since Red Hat is keen to improve virtualization things, I think this is going to GA.

DRBD
DRBD was getting into upstream Kernel 2.6.33. DRBD (Distributed Replicated Block Device) is some kind of RAID-1 over TCP/IP and is rock solid since years. From my point of view it is the best invention since sliced bread when it comes to cluster technologies. It is widely used, also on RHEL. Have a look to Florians Haas’ comment about support, and further to Alan Robertson’s comment. While Florian is working at Linbit (the developer company of DRBD) points to support problems existing on current releases on RHEL, Alan is a “Urgestein” (sorry, cant find a English word for it, it is meant in a very positive manner) of Linux clustering likes too to have DRBD in RHEL6. Quite a lot of people are included in the bugs CC list (as I’m writing 37 people). This brings quite some preasure on Red Hat to include DRBD in RHEL6. @Red Hat: Do it! include DRBD! If not as a “supported” product, deliver it and find a way with Linbit for the support.

Getting rid of the crappy VMware-tools
For people urged to use VMWares ESX stuff as virtalization technology, there is another important thing that changed: In 2.6.34 upstream Kernel, Linus Torvalds accepted VMWares ballooning driver (vmmemctl). In 2.6.33 Linus accepted VMWares vmxnet3 and pvscsci drivers which have been already backported to RH’s Kernel 2.6.32-EL. So, also backporting vmmemctl is *THE* chance to get rid of those crappy VMWare Tools. For companies relying on ESX this would be a *VERY* important feature. I’ll made a service request (SR 2021028) @Red Hat and will file a RFE-Bug at bugzilla ASAP. Please vote for it!

Other stuff
There are other RFE’s pending. Most of them are not really important for enterprise computing (my point of view). Mostly this RFE’s are about virtualization and bound to libvirt. Most of these RFE’s seems to be trivial and are on status “ON_QA” which means they are most probably included in RHEL6.

What is your favorit RFE-Bug? Please let me know…

Have fun!

Luc

Red Hat’s virtualization strategy has redundancy – Quo vadis?

Thursday, May 27th, 2010

A couple of days there have been some reports that Red Hat will release a commercialized version of deltacloud, an abstraction layer for different kinds of virtualization technologies and clouds such as VMware, RHEV, Amazon EC2 etc.

Red Hat puts a lot of resources on virtualization, they maintain and/or sponsor multiple projects in parallel. The most important from my point of view is libvirt which is as well an abstraction layer for different virtulization technologies such as VMware, KVM, Xen and others. Libvirt and deltacloud are partially redundant.

It is not the only redundancy created by Red Hat. There is also O-virt “competing” with RHEV. Both are not tightly bound to RHN satellite or Spacewalk.

RHEV works with system templates similar to those at VMware. On the other hand: Koan, together with cobbler is a deployment software for virtual hosts and was recently bundled with RHN satellite.

Not all of those Red Hat virtualization projects are working well together. So the question arises: What is the strategy of having such redundancies of projects? Why not integrating all of this projects and glue them together?

Lots of questions…

Have fun!

KVM supports live migration between CPUs with different features

Saturday, May 15th, 2010

The video is a bit old, it is from November 2008. But it is still quite interesting to see and discuss about it. With KVM you can upgrade your farm of servers easy, it does not matter if the new servers have CPUs with new features or not. I’m not sure if you can do this with ESX, I guess not, you probably need to migrate them shut down.

Have fun!