Posts Tagged ‘ESX’

Automated disk partitioning on virtual machines with Cobbler

Saturday, December 15th, 2012

The default Cobbler Snippets just do simple auto partitioning. For a more sophisticated partition layout you need to know what kind of VM you are going to install. KVMs and RHEVs device name is /dev/vda, Xen uses /dev/xvda and ESX /dev/sda.

Luckily this can be figured out automatically, those different virtualization vendors are using its own MAC prefixes. So we can add two nice small Cobbler snippets to do the job. In this example, I call them hw-detect and partitioning.

hw-detect

#set $mac = $getVar('$mac_address_eth0')
#if $mac
#set $mac_prefix = $mac[0:8]
#if $mac_prefix == "00:1a:4a"
# This is a RHEV virtual machine
#set global $machinetype = 'kvm'

#else if $mac_prefix == "52:54:00"
# This is a KVM/Qemu virtual machine
#set global $machinetype='kvm'

#else if $mac_prefix == "00:16:3e"
# This is a XEN virtual machine
#set global $machinetype='xen'
#
#else if $mac_prefix == "00:50:56"
# This is a ESX virtual machine
#set global $machinetype = 'esx'

#else
# #This is a physical machine
#set global $machinetype = 'physical'
#end if
#end if

partitioning

#if $machinetype == 'kvm'
#set $disk='vda'
#else if $machinetype == 'xen'
#set $disk = 'xvda'
#else
#set $disk = 'sda'
#end if
# Lets install the system on /dev/$disk
part /boot      --fstype ext2 --size=250 --ondisk=$disk
part pv.0       --size=1 --grow --ondisk=$disk

volgroup vg_${name} pv.0

logvol /        --fstype ext4 --name=lv_root    --vgname=vg_${name} --size=4096
logvol /home    --fstype ext4 --name=lv_home    --vgname=vg_${name} --size=512 --fsoption=nosuid,nodev,noexec
logvol /tmp     --fstype ext4 --name=lv_tmp    --vgname=vg_${name} --size=1024 --fsoption=nosuid,nodev,noexec
logvol /var     --fstype ext4 --name=lv_var    --vgname=vg_${name} --size=2048 --fsoption=nosuid,nodev,noexec
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_${name} --size=2048

An additional “feature” of the partitioning Snippet is: It sets up the Volume Group name according to your systems name. This is the unofficial standard since quite some time. It also sets some more secure mount options. Review them carefully if they make sense for you and edit them as needed.

The next step is to configure your kickstart template.

Standalone Cobbler
On a standalone Cobbler server edit /var/lib/cobbler/kickstart/your-kick-start-template.ks

# Detect the used hardware type
$SNIPPET('hw-detect')
# Set up default partitioning
$SNIPPET('partitioning')

Bundled Cobbler
When using cobbler bundled with Spacewalk or Red Hat Satellite, you need to edit the Kickstart profile in the WebUI.


Navigate to Systems -> Kickstart -> Profile. Select the Kickstart profile to be modified -> System Details -> Partitioning.

Copy the two Snippets in /var/lib/cobbler/spacewalk/1, where 1 is representing your OrgId.

Alternatively you can edit them in the WebUI as well.

To check if all is working as expected, add a system to Cobbler using the Command Line Interface and have a look to the rendered Kickstart file. This can be easily done with cobbler system getks --name=blah.

Happy System installing….

Have fun :-)

A review of RHEV

Sunday, February 27th, 2011

In the past few weeks I had the chance to have a closer look at the current release 2.2. The reason is that I’m working on a project using RHEL6 clients as virtual Desktops. For a proof-of-concept I’ve set up a test environment in the lab. Due to the lack of time I was not able to test every single feature.

After reading some docs, it was amazingly easily and quickly installed.

Test environment
The tests have been made on the following hardware:

  • 2 Xeon servers with a total of 8 cores and 24Gbyte on each host. OS is RHEL5.6. Those hosts are called RHEV-H (H like Hypervisor)
  • 1 Windows 2008R2 server (64bit) with 4Gbyte RAM as RHEV-M (M like management)
  • 1 dedicated NetApp filer serving storage over NFS
  • 1 RHEL 5.6 Server providing “cheap” NFS storage
  • 3 different networks connected to 3 NICs
  • 1 RHEL 5.6 “thin” client with spice-xpi as client
  • 1 Windows 2008R2 Server with spice-activex as client (actually on the management server)

VM’s (All 64bit):

  • 20 Virtual desktops with RHEL 6.0 clients using one core and 2Gbyte or RAM each
  • 2 Virtual servers with RHEL 6.0 server using 2 cores and 2Gbyte of RAM each
  • 2 Virtual servers with RHEL 5.6 server using 2 cores and 2Gbyte of RAM each

Management Portal
The Management interface is the central point of administration. It is not always as intuitive as expected. If you are new to RHEV, you can get confused. A big plus is the search functionality. One can search for storage, VMs, hosts and any other item in the environment.

User Portal
The user portal is very simple, lean and clean. Users see the machines that are assigned to them, they can power on and off the machines and can connect them to powered VMs, and the client appears on the desktop.

The user portal resides on the management server and is just the connection broker. As soon as a user connects to its desktop or server, the connection will be made directly to the spiced qemu-kvm running on one of the hypervisors.

Client
At the moment there are two different clients supported:

  • spice-activex with Internet Explorer 7+ on Windows XP, Vista and 7
  • spice-xpi with Firefox for RHEL5 and RHEL6

If you do not fear the work, you probably can get the spice-xpi client also running on Fedora 13 and other Linux distributions.

The client communicates with the VM over SPICE (Simplified Protocol for Independent Computing Environments). It can forward USB devices as well as sound and other multimedia stuff. For more informations about the protocol, please have a look at http://www.spice-space.org/home.html.

Storage
The first step to set up a RHEV environment is to define and set up the storage.

There are 3 types of storage:

  • ISO, where you upload your install-media for your virtual machines.
  • Data, where the VM images are stored.
  • Export, where I have no clue (yet) what it is good for.

Usually the storage is a NFS or iSCSI server, FC storage is also supported. The storage server creates snapshots of volumes for backing up the stuff.

Hosts (hypervisors)
You can either use RHEV-H as an “embedded” hypervisor which comes as a stripped-down RHEL5 distribution (the ISO is just about 75Mbyte IIRC) or you can set up RHEL5 and use it as a hypervisor.

First I tried the RHEV-H, it is manual work to install it. You need to burn a CDROM put it in your server and enter some stuff like network config etc. In the short time I did not tried to provision RHEV-H with cobbler. According to the documentation one can pass boot parameters to partially automate the installation.

Because I have a cobbler server handy, I decided to use RHEL5 as hypervisors. Just be sure you subscribe the system to the channels “rhel-x86_64-server-vt-5″ and rhel-x86_64-rhev-mgmt-agent-5. You also need to allow root to log in via SSH. You can change this later after the hosts is registered on the RHEV-M server.

After the setup of the RHEL5 servers you just need to tell RHEV-M that there is a new host available. Tell RHEV-M about the root password and all the needed additional software gets installed automatically.

That’s a pretty lean procedure to install and set up hosts. Hopefully there will soon be a way to fully automate this, w/o using MS Powershell.

The technologies on the hosts are the well known stable and mature KVM and qemu-kvm. There are also parts of o-virt being used.

Networking
I do not know how many networks you can add, but I think enough even for large environments. The network configuration part in RHEV-M shows a list of available Ethernet ports on each host. Just assign them a static network and IP address and you’re done. Make sure you define the network you want to add on all hosts in a particular datacenter.

Ensure that firewalls between the users and VM’s networks are configured accordingly to avoid connection problems.

Sizing and performance

  • Memory overcommitment
    Thanks to KSM (Kernel Samepage Merging) one can quite overcommit the memory. Ensure you have a LOT of swap on your hosts. Why? It takes some time until KSM kicks in and frees memory pages. During the collection of those pages, swap space is needed to prevent getting a visit by the OOM killer.

    I once faced a complete crash when putting one host in maintenance mode and all VMs have been migrated to the second host. That’s exactly why you should have LOTs of swap spaces to survive when using memory over-commitment. During the period of KSM searching for same pages, performance is degraded.

    Under normal circumstances, this will never happen since the virtual hosts are load balanced. This means VM’s are distributed to have the optimal performance.

  • CPU overcommitment
    This very depends on your users computing needs. For normal desktops and servers you can overcommit at least 200% of the CPUs and the performance is still fine. For CPU intensive workload it is not recommended to over commit CPUs.
  • Number of hosts
    Its recommended to start with at least three hosts to avoid temporary performance penalties if you need to put a host into maintenance mode. (See also “Memory overcommitment”).

    The performance is surprisingly good, the user experience is nearly the same as on physical desktops. For servers, KVM is already known for its good performance.

Drawbacks
Unfortunately you can not hot-add or hot-remove virtual CPU’s, NICs and storage. Despite of being supported by the underlying technologies used, your VM must be powered off to add or remove resources.

Another drawback is the lack of different storage classes, this makes it actually impossible to use the product in large enterprise environments. Since it makes no sense to back up swap drives, it is waste of backup space.
It is also not possible to have different storage for different needs. Lets say a bulk RAID5 consisting of cheap 2Tbyte SATA disks for operating systems and a RAID10 consisting of fast 15k SAS or FC disks for data and swap. It is also not possible to life-migrate VMs from one storage to another.

As of today, SPICE is only useful on LAN networks. Depending on the work load it can demand lots of bandwidth available. For usual office workload, 1 Mbps per connection is fine. A WAN optimized version is under development and should be released “later this year”.

The two supported clients (spice-xpi and spice-activex) can be problematic when the clients are not in an environment controlled by you. Why? Lots of enterprises are preventing its users to install Active X applications and Firefox Add-On’s for security reasons. It would be better to have some kind of “fat client” which does not need to be installed, or a Java Applet which does the job.

At the moment, it is still required to have a Windows 2008R2 server installed and use MS technologies such as MSSQL, DotNET, Powershell and Active Directory. Because you can not manage RHEV-M with a satellite, one needs to download and install updates manually.

RHEV-M is not only the management server, it is also the connection broker for clients connecting. RHEV-M is a single point of failure. Maybe one can build a MS-Cluster with it, no clue. It is also not possible to self-host RHEV-M on the RHEV-H hosts. Already connected clients should not be disconnected in the case of RHEV-M failing (to be tested).

Red Hat is working on an Linux-Only RHEV-M, as you can read in a comment from a Red Hat employee in a earlier post.

The same comment talks about the WAN problems to be solved in near future.

Conclusion
Frankly: From my point of view, RHEV is not yet “enterprise ready”, this is because of the drawbacks mentioned above, especially the storage shortcomings and RHEV-M being a SPoF.

For smaller environments up to lets say ~50 servers or ~200 desktops it good enough.

Nevertheless: I think RHEV has a huge potential in the future. Compared to VMware ESX, the KVM’s technology is much better and scales better. At the moment VMware’s USP is its management software, not the hypervisor used. As soon as Red Hat ironed out the major drawbacks, I’ll expect a boost for RHEV.

Red Hat is keen to improve the product, I guess a lot of improvements will be annouced in 2011.

Have fun!

Deploying RHEL as ESX guests – Kickstarting or using ESX templates?

Wednesday, October 20th, 2010

Some time ago I asked my self the question if it is better to kickstart systems or working with ESX templates when deploying RHEL as ESX guests. I also had some discussions with friends working in the same industry. I tried it and came to the following conclusion:

Kickstart the systems is the way to go.

Pros:

  • Kickstarted Systems are already up-to-date after installation.
  • Proper SSH host keys. Using ESX templates ends up in having identical SSH host keys, from the security standpoint not usable, they need to be manually re-created.
  • Kickstarting means lean deployment, much less data needs to be transferred.
  • Very fast, kickstarted systems are deployed in ~3min instead of ~10min (depending on I/O and network performance).
  • Systems are being automatically registered @rhn or on a rhn-satellite with the help of cobbler snippets.
  • Better customization.

Cons:

  • The ESX template to used for kickstarting must have no disks configured, otherwise the whole nominal disk size is being transferred over the net.

When kickstarting virtual systems, only the data needed (the RPMs) is transferred. The best way is to have a “empty” ESX template, just with the network defined, but no disks. The reason for that is: ESX creates a checksum of the disk files, even if the disk is empty, the sparse disk files (in the case of “thin provisioning”) will be transferred over the net at its full nominal size.

When using ESX templates, after deploying, one needs to register the system manually and also manually update the system by invoking “yum -y update”. In contrary, kickstarted systems are always up to date automatically. To circumvent this fact, one needs to keep the templates up to date, it is a manual task which can not be automated easily.

Have fun!

Red Hat’s virtualization strategy has redundancy – Quo vadis?

Thursday, May 27th, 2010

A couple of days there have been some reports that Red Hat will release a commercialized version of deltacloud, an abstraction layer for different kinds of virtualization technologies and clouds such as VMware, RHEV, Amazon EC2 etc.

Red Hat puts a lot of resources on virtualization, they maintain and/or sponsor multiple projects in parallel. The most important from my point of view is libvirt which is as well an abstraction layer for different virtulization technologies such as VMware, KVM, Xen and others. Libvirt and deltacloud are partially redundant.

It is not the only redundancy created by Red Hat. There is also O-virt “competing” with RHEV. Both are not tightly bound to RHN satellite or Spacewalk.

RHEV works with system templates similar to those at VMware. On the other hand: Koan, together with cobbler is a deployment software for virtual hosts and was recently bundled with RHN satellite.

Not all of those Red Hat virtualization projects are working well together. So the question arises: What is the strategy of having such redundancies of projects? Why not integrating all of this projects and glue them together?

Lots of questions…

Have fun!

A brief test of RHEL 6 Beta 1

Friday, April 23rd, 2010

As promised yesterday, I publish the results of a brief test of RHEL6 Beta 1 and the most important findings. It is my point of view as a system guys daily business. If not stated, this overview is based on a default installation with no customization.

General

  • There are new package groups such as  “Minimal” with 228 Packages and “Basic Server” with 523 Packages. “Basic Server” is the default installation, which means the default click trough installation compared to RHEL5 is much less bloated.
  • The versions of the most important software is quite up-to-date but as expected not on the bleeding edge.
  • Postfix is the default MTA. Finally Red Hat managed to switch away from sendmail like other distributions did it years ago.
  • Bye bye SysV init: As I guesstimated in october 2009 RHEL6 comes with upstart instead of traditional SysV init. (See http://blog.delouw.ch/2009/10/31/ready-to-upstart/). The boot process is much faster compared to RHEL5. Upstart comes with legacy support for traditional runcontrol scripts in /etc/init.d.
  • Still too many services enabled after default install. Generally unneeded services like avahi/mDNS and NFS-related daemons such as  portmap are still enabled by default.

Virtualization

As expected, Xen was removed completely from RHEL6. These is being discussed controversial. Why not providing both virtualization solutions as before? Recently Citrix released Xen4 which works well together with Kernel 2.6.32, the same version as used by RHEL6.

KVM and its friends made a huge step forward. lib-virt, virt-manager and stuff is nearly up-to-date with the upstream versions. Means: The virtualization infrastructure made a lot of progress. Installing RHEL6 as a KVM guest works great. All drivers needed (virtio) are automatically installed.

A major good message to people which are using VMware vPhere 4 is that RHEL 6 comes which native support of vmxnet3 which was obviously backported from Kernel 2.6.33. Vmxnet3 is the driver for VMware’s para-virt NIC which brings quite some performance enhancements and lower CPU usage on the ESX host.

Certifications from ISVs

A quick check (not actually tested) for the requirements for SAP and Oracle shows that those are fulfilled already. We can expect the certification quite soon after GA of RHEL6. [update] Some compatibility RPMs from the mid 1990′s disappeared.  I now need to figure out if they are *really* needed by Oracle and/or SAP[/update]

Integration with Cobbler

Integration with cobbler works like expected, cobbler import –path=/mnt –name=rhel6 and you are done. For a quick test I just copied the kickstart template from RHEL5 and I’m not sure if this is a good method. A test-install on ESX4 failed, the system hung at the creating of the root-VG. Not sure yet if it is a bug or something is incompatible in the kickstart. [update] The system hung was because of out-of-memory. The test-installation was on a ESX guest with 384Mbyte of memory which is enough according to the documentation but too little in real life. Growing the RAM of the test system to 512Mybte helped, but some packages needed by for SAP have changes names or disappeared.  After changing/removing those RPMs, the installation went smoothly[/update]

Bugs or features?

I detected some oddities where I’m not sure if it is a bug or a feature. We will see whats going on on http://bugzilla.redhat.com.

  • No network configured after default install. At the moment you need to configure it manually (considered a Bug)
  • I detected a major security issue during install, I’m not going to disclose it before a patch is available or more information from Red Hat is made available. I reported it 2010-04-23 ~12:00 on Red Hats bugzilla bugtracker. [update] The bug gots assigned to a Red Engineer after three hours, seems like Red Hat is acting very professional on the case[/update]

Conclusion

After this brief test one can say that RHEL6 will be a really great Linux Distribution for enterprise servers. The beta is already very stable with few bugs detected from my side. My guesstimate is that mid of May 2010 there will be a second public beta released, lets stay tuned, I’ll keep you up-to-date with further findings.

Have fun!

Kernel questions about RHEL6, ESX support and experiences with F13a3

Monday, March 15th, 2010

Still no official informations

Red Hat is still refusing any questions about the features of RHEL 6 and its Linux Kernel. However: Since Vanilla Kernel 2.6.33 vmxnet3 and pvscsi is supported. Fedora 13 Alpha 3 is shipped with a derivate of Kernel 2.6.33.

I still hope that Red Hat is switching to 2.6.33 or back-porting the VMWare code to its 2.6.32 derivative Kernel as known by RHEL 6 Alpha 3.

Experiences with F13a3 so far

Installing F13a3 on a ESX guest – with RHEL5 as “supported Guest OS”  – and enabled vmxnet “enhanced” plus pvscsci as HBA was a smooth experience. No driver disk was needed, no dirty fixes. Just selecting vmxnet3 as NIC and PVSCSCI as disk HBA. Thats the way RHEL6 should work from my point of view.

RHEV vs. VMWare ESX

Since Red Hat released its visualization solution “RHEV”, VMWare and Red Hat are competitors. Is Red Hat willing to include ESX support in its Enterprise Products? My guess is to not to do so, but I’m open for surprises.

The goals

The goal on the long term is to switch from ESX to KVM. However, if you deployed a large ESX farm already and the management members are members of the “ESX-Church” it will be hard.

The mid-term goal is to get rid of those crappy VMWare tools. The current state of this “Tools” definitively proves that VMWare is a Windows shop and  does not take care about Linux virtualization.

Will we have fun? Depends on EMC and Red Hat….