Using Ansible to automate oVirt and RHV environments

Bored of clicking in the WebUI of RHV or oVirt? Automate it with Ansible! Set up a complete virtualization environment within a few minutes.

Some time ago, Ansible includes a module for orchestrating RHV environments. It allows you to automate the setup of such an environment as well as automating daily tasks.

Preparation

Of course, Ansible can not automate all tasks, you need to set up a few things manually. Lets assume you want your oVirt-engine or RHV-manager running outside of the RHV environment which has some benefits when it comes to system management.

  • Setup of at least two hypervisor machines with RHEL7 latest
  • Setup of the RHV-M machine with RHEL7 latest
  • Having the appropriate Redhat Subscriptions
  • A machine with Ansible 2.3 installed

Set up the inventory file

Ensure you have a inventory file like the following in place,i.e. in /etc/ansible/hosts

[rhv]
        rhv-m.example.com

[hypervisors]
        hv1.example.com
        hv2.example.com

Helper files

ovirt-engine-vars.yml

engine_url: https://rhv-m.example.com/ovirt-engine/api
username: admin@internal
password: redhat
engine_cafile: /etc/pki/ovirt-engine/ca.pem
datacenter: Default
cluster: Default

rhsm_user: user@example.com
rhsm_pass: secret

Please adjust the following example answer file for your environment.

rhv-setup.conf

# action=setup                                                                                                        
[environment:default]                                                                                                 
OVESETUP_DIALOG/confirmSettings=bool:True                                                                                            
OVESETUP_CONFIG/applicationMode=str:both                                                                                             
OVESETUP_CONFIG/remoteEngineSetupStyle=none:None                                                                                     
OVESETUP_CONFIG/sanWipeAfterDelete=bool:False                                                                                        
OVESETUP_CONFIG/storageIsLocal=bool:False                                                                                            
OVESETUP_CONFIG/firewallManager=none:None                                                                                            
OVESETUP_CONFIG/remoteEngineHostRootPassword=none:None                                                                               
OVESETUP_CONFIG/firewallChangesReview=none:None                                                                                      
OVESETUP_CONFIG/updateFirewall=bool:False                                                                                            
OVESETUP_CONFIG/remoteEngineHostSshPort=none:None                                                                                    
OVESETUP_CONFIG/fqdn=str:rhv-m.example.com                                                                                        
OVESETUP_CONFIG/storageType=none:None                                                                                                        
OSETUP_RPMDISTRO/requireRollback=none:None                                                                                                   
OSETUP_RPMDISTRO/enableUpgrade=none:None                                                                                                     
OVESETUP_PROVISIONING/postgresProvisioningEnabled=bool:True                                                                                  
OVESETUP_APACHE/configureRootRedirection=bool:True                                                                                           
OVESETUP_APACHE/configureSsl=bool:True                                                                                                         
OVESETUP_DB/secured=bool:False
OVESETUP_DB/fixDbConfiguration=none:None
OVESETUP_DB/user=str:engine
OVESETUP_DB/dumper=str:pg_custom
OVESETUP_DB/database=str:engine
OVESETUP_DB/fixDbViolations=none:None
OVESETUP_DB/engineVacuumFull=none:None
OVESETUP_DB/host=str:localhost
OVESETUP_DB/port=int:5432
OVESETUP_DB/filter=none:None
OVESETUP_DB/restoreJobs=int:2
OVESETUP_DB/securedHostValidation=bool:False
OVESETUP_ENGINE_CORE/enable=bool:True
OVESETUP_CORE/engineStop=none:None
OVESETUP_SYSTEM/memCheckEnabled=bool:True
OVESETUP_SYSTEM/nfsConfigEnabled=bool:False
OVESETUP_PKI/organization=str:example.com
OVESETUP_PKI/renew=none:None
OVESETUP_CONFIG/isoDomainName=none:None
OVESETUP_CONFIG/engineHeapMax=str:1955M
OVESETUP_CONFIG/ignoreVdsgroupInNotifier=none:None
OVESETUP_CONFIG/adminPassword=str:redhat
OVESETUP_CONFIG/isoDomainACL=none:None
OVESETUP_CONFIG/isoDomainMountPoint=none:None
OVESETUP_CONFIG/engineDbBackupDir=str:/var/lib/ovirt-engine/backups
OVESETUP_CONFIG/engineHeapMin=str:1955M
OVESETUP_DWH_CORE/enable=bool:True
OVESETUP_DWH_CONFIG/scale=str:1
OVESETUP_DWH_CONFIG/dwhDbBackupDir=str:/var/lib/ovirt-engine-dwh/backups
OVESETUP_DWH_DB/secured=bool:False
OVESETUP_DWH_DB/restoreBackupLate=bool:True
OVESETUP_DWH_DB/disconnectExistingDwh=none:None
OVESETUP_DWH_DB/host=str:localhost
OVESETUP_DWH_DB/user=str:ovirt_engine_history
OVESETUP_DWH_DB/dumper=str:pg_custom
OVESETUP_DWH_DB/database=str:ovirt_engine_history
OVESETUP_DWH_DB/performBackup=none:None
OVESETUP_DWH_DB/port=int:5432
OVESETUP_DWH_DB/filter=none:None
OVESETUP_DWH_DB/restoreJobs=int:2
OVESETUP_DWH_DB/securedHostValidation=bool:False
OVESETUP_DWH_PROVISIONING/postgresProvisioningEnabled=bool:True
OVESETUP_CONFIG/imageioProxyConfig=bool:True
OVESETUP_RHEVM_DIALOG/confirmUpgrade=bool:True
OVESETUP_VMCONSOLE_PROXY_CONFIG/vmconsoleProxyConfig=bool:True
OVESETUP_CONFIG/websocketProxyConfig=bool:True

Prepare your machines

The first Playbook ensures your machines are subscribed to RHSM and the needed repos are made available.

install_rhv.yml

---
- hosts: rhv,hypervisors
  vars_files:
    - ovirt-engine-vars.yml
  
  tasks:
  - name: Register the machines to RHSM
    redhat_subscription:
      state: present
      username: "{{ rhsm_user }}"
      password: "{{ rhsm_pass }}"
      pool: '^(Red Hat Enterprise Server|Red Hat Virtualization)$'

  - name: Disable all repos
    command: subscription-manager repos --disable=*

- hosts: hypervisors
  tasks:
    - name: Enable required repositories
      command: subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhv-4-mgmt-agent-rpms
 
- hosts: rhv
  tasks:

    - name: Enable required repositories
      command: subscription-manager repos --enable=jb-eap-7-for-rhel-7-server-rpms --enable=rhel-7-server-rhv-4-tools-rpms --enable=rhel-7-server-rhv-4.1-rpms --enable=rhel-7-server-supplementary-rpms --enable=rhel-7-server-rpms

    - name: Copy Answer File
      copy:
        src: rhv-setup.conf
        dest: /tmp/rhv-setup.conf

    - name: Run RHV setup
      shell: |
        engine-setup --config-append=/tmp/rhv-setup.conf

Run the playbook

user@ansible playbooks]$ ansible-playbook -k install_rhv.yml 
SSH password: 

PLAY [rhv,hypervisors] ************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [rhv-m.example.com]
ok: [hv1.example.com]
ok: [hv2.example.com]

TASK [Register the machines to RHSM] **********************************************************************************************************
ok: [hv1.example.com]
ok: [hv2.example.com]
ok: [rhv-m.example.com]

TASK [Disable all repos] **********************************************************************************************************************
changed: [rhv-m.example.com]
changed: [hv2example.com]
changed: [hv1.example.com]

PLAY [hypervisors] ****************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [hv1.example.com]
ok: [hv2.example.com]

TASK [Enable required repositories] ***********************************************************************************************************
changed: [hv1.example.com]
changed: [hv2.example.com]

PLAY [rhv] ************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [rhv-m.example.com]

TASK [Enable required repositories] ***********************************************************************************************************
changed: [rhv-m.example.com]

TASK [Copy Answer File] ***********************************************************************************************************************
ok: [rhv-m.example.com]

TASK [Run RHV setup] **************************************************************************************************************************
changed: [rhv-m.example.com]

PLAY RECAP ************************************************************************************************************************************
hv1.example.com : ok=5    changed=2    unreachable=0    failed=0   
hv2.example.com : ok=5    changed=2    unreachable=0    failed=0   
rhv-m.example.com       : ok=7    changed=3    unreachable=0    failed=0   

[user@ansible playbooks]$ 

Deploy your environment

Your environment is now ready to set up all the required stuff such as data centers, clusters, networks, storage etc.

rhv-deploy.yml

---
- name: Deploy RHV environment
  hosts: rhv

  vars_files: 
    - ovirt-engine-vars.yml

  pre_tasks:
  - name: Log in
    ovirt_auth:
      url: "{{ engine_url }}"
      username: "{{ username }}"
      password: "{{ password }}"
      ca_file: "{{ engine_cafile }}"
    tags:
      - always

  tasks:

  - name: ensure Datacenter "{{ datacenter }}" is existing
    ovirt_datacenters:
      auth: "{{ ovirt_auth }}"
      name: "{{ datacenter }}"
      comment: "Our primary DC"
      compatibility_version: 4.1
      quota_mode: enabled
      local: False

  - name: Ensure Cluster "{{ cluster }}" is existing
    ovirt_clusters:
      auth: "{{ ovirt_auth }}"
      name: "{{ cluster }}"
      data_center: "{{ datacenter }}"
      description: "Default Cluster 1"
      cpu_type: "Intel Haswell-noTSX Family"
      switch_type: legacy
      compatibility_version: 4.1
      gluster: false
      ballooning: false
      ha_reservation: true
      memory_policy: server
      rng_sources:
        - random

  - name: Ensure logical network VLAN101 exists
    ovirt_networks:
      auth: "{{ ovirt_auth }}"
      data_center: "{{ datacenter }}"
      name: vlan101
      vlan_tag: 101
      clusters:
        - name: "{{ cluster }}"
          assigned: True
          required: False

  - name: ensure host hv1 is joined
    ovirt_hosts:
      auth: "{{ ovirt_auth }}"
      cluster: "{{ cluster }}"
      name: hv1
      address: 192.168.100.112
      password: redhat

  - name: ensure host hv2 is joined
    ovirt_hosts:
      auth: "{{ ovirt_auth }}"
      cluster: "{{ cluster }}"
      name: hv2
      address: 192.168.100.20
      password: redhat

  - name: Assign Networks to host 
    ovirt_host_networks:
      auth: "{{ ovirt_auth }}"
      state: present
      name: "{{ item }}"
      interface: eth1
      save: True
      networks: 
        - name: vlan101
    with_items:
      - hv1
      - hv2


  - name: Enable Power Management for host1    
    ovirt_host_pm:
      auth: "{{ ovirt_auth }}"
      name: hv1
      address: 10.10.10.10
      options:
        lanplus=true
      username: admin
      password: secret
      type: ipmilan

  - name: Enable Power Management for host1
    ovirt_host_pm:
      auth: "{{ ovirt_auth }}"
      name: hv2
      address: 10.10.10.11
      options:
        lanplus=true
      username: admin
      password: secret
      type: ipmilan

  - name: Create VM datastore
    ovirt_storage_domains:
      auth: "{{ ovirt_auth }}"
      name: vms
      host: "hv2"
      data_center: "{{ datacenter }}"
      nfs:
        address: nfs.example.com
        path: /exports/rhv/vms

  - name: Create export NFS storage domain
    ovirt_storage_domains:
      auth: "{{ ovirt_auth }}"
      name: export
      host: "hv2"
      domain_function: export
      data_center: "{{ datacenter }}"
      nfs:
        address: nfs.example.com
        path: /exports/rhv/export

  - name: Create ISO NFS storage domain
    ovirt_storage_domains:
      auth: "{{ ovirt_auth }}"
      name: iso
      host: "hv2"
      domain_function: iso
      data_center: "{{ datacenter }}"
      nfs:
        address: nfs.example.com
        path: /exports/rhv/iso

Run the playbook

user@ansible playbooks]$ ansible-playbook -k rhv-deploy.yml
SSH password: 

PLAY [Deplay RHV environment] *****************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [rhv-m.example.com]

TASK [Log in] *********************************************************************************************************************************
ok: [rhv-m.example.com]

TASK [ensure Datacenter "Default" is existing] ************************************************************************************************
changed: [rhv-m.example.com]

TASK [Ensure Cluster "Default" is existing] ***************************************************************************************************
changed: [rhv-m.example.com]

TASK [Ensure logical network VLAN101 exists] **************************************************************************************************
changed: [rhv-m.example.com]

TASK [ensure host hv1 is joined] ****************************************************************************************************
changed: [rhv-m.example.com]

TASK [ensure host hv2 is joined] ****************************************************************************************************
changed: [rhv-m.example.com]

TASK [Assign Networks to host] ****************************************************************************************************************
ok: [rhv-m.example.com] => (item=hv1)
ok: [rhv-m.example.com] => (item=hv2)

TASK [Enable Power Management for host1] ******************************************************************************************************
changed: [rhv-m.example.com]

TASK [Enable Power Management for host1] ******************************************************************************************************
changed: [rhv-m.example.com]

TASK [Create VM datastore] ********************************************************************************************************************
changed: [rhv-m.example.com]

TASK [Create export NFS storage domain] *******************************************************************************************************
changed: [rhv-m.example.com]

TASK [Create ISO NFS storage domain] **********************************************************************************************************
changed: [rhv-m.example.com]

PLAY RECAP ************************************************************************************************************************************
rhv-m.example.com       : ok=13   changed=10   unreachable=0    failed=0   

[user@ansible playbooks]$ 

Further readings

Conclusion

With the help of Ansible you can automate a lot of boring tasks in a convenient way. You may even merge the two playbooks into one, be aware that the RHV-M setup will fail if its already set up.

Have fun 🙂

Providing SRV and TXT records for Kerberos and LDAP with dnsmasq

What if you have an application such as OVirt/RHEV-M that relies on DNS services records and you dont have the possibility to add them to the DNS servers because the DNS admins do not like to do its job?

Fake them! DNSMasq is your friend 🙂 Install dnsmasq on the server in question and configure /etc/resolv.conf to query first dnsmask on localhost.

yum -y install dnsmasq
chkconfig dnsmasq on

Assuming your subdomain is called example.com and your ldap and kerberos providers are ipa1.example.com and ipa2.example.com, configure dnsmasq as following:

cat << EOF >> /etc/dnsmasq.conf
srv-host =_kerberos._udp.example.com,ipa1.example.com,88
srv-host =_kerberos._udp.example.com,ipa2.example.com,88
srv-host =_kerberos._tcp.example.com,ipa1.example.com,88
srv-host =_kerberos._tcp.example.com,ipa2.example.com,88
srv-host =_kerberos-master._tcp.example.com,ipa1.example.com,88
srv-host =_kerberos-master._tcp.example.com,ipa2.example.com,88
srv-host =_kerberos-master._udp.example.com,ipa1.example.com,88
srv-host =_kerberos-master._udp.example.com,ipa2.example.com,88
srv-host =_kpasswd._tcp.example.com,ipa1.example.com,88
srv-host =_kpasswd._tcp.example.com,ipa2.example.com,88
srv-host =_kpasswd._udp.example.com,ipa1.example.com,88
srv-host =_kpasswd._udp.example.com,ipa2.example.com,88
srv-host =_ldap._tcp.example.com,ipa1.example.com,389
srv-host =_ldap._tcp.example.com,ipa2.example.com,389
txt-record=_kerberos.example.com,"EXAMPLE.COM"
EOF

Add the follwing line to /etc/resolv.conf and make sure 127.0.0.1 is the first DNS server to be queried.

nameserver 127.0.0.1

Start dnsmasq and have fun 🙂

service dnsmask start

Automated disk partitioning on virtual machines with Cobbler

The default Cobbler Snippets just do simple auto partitioning. For a more sophisticated partition layout you need to know what kind of VM you are going to install. KVMs and RHEVs device name is /dev/vda, Xen uses /dev/xvda and ESX /dev/sda.

Luckily this can be figured out automatically, those different virtualization vendors are using its own MAC prefixes. So we can add two nice small Cobbler snippets to do the job. In this example, I call them hw-detect and partitioning.

hw-detect

#set $mac = $getVar('$mac_address_eth0')
#if $mac
#set $mac_prefix = $mac[0:8]
#if $mac_prefix == "00:1a:4a"
# This is a RHEV virtual machine
#set global $machinetype = 'kvm'

#else if $mac_prefix == "52:54:00"
# This is a KVM/Qemu virtual machine
#set global $machinetype='kvm'

#else if $mac_prefix == "00:16:3e"
# This is a XEN virtual machine
#set global $machinetype='xen'
#
#else if $mac_prefix == "00:50:56"
# This is a ESX virtual machine
#set global $machinetype = 'esx'

#else
# #This is a physical machine
#set global $machinetype = 'physical'
#end if
#end if

partitioning

#if $machinetype == 'kvm'
#set $disk='vda'
#else if $machinetype == 'xen'
#set $disk = 'xvda'
#else
#set $disk = 'sda'
#end if
# Lets install the system on /dev/$disk
part /boot      --fstype ext2 --size=250 --ondisk=$disk
part pv.0       --size=1 --grow --ondisk=$disk

volgroup vg_${name} pv.0

logvol /        --fstype ext4 --name=lv_root    --vgname=vg_${name} --size=4096
logvol /home    --fstype ext4 --name=lv_home    --vgname=vg_${name} --size=512 --fsoption=nosuid,nodev,noexec
logvol /tmp     --fstype ext4 --name=lv_tmp    --vgname=vg_${name} --size=1024 --fsoption=nosuid,nodev,noexec
logvol /var     --fstype ext4 --name=lv_var    --vgname=vg_${name} --size=2048 --fsoption=nosuid,nodev,noexec
logvol swap     --fstype swap --name=lv_swap    --vgname=vg_${name} --size=2048

An additional “feature” of the partitioning Snippet is: It sets up the Volume Group name according to your systems name. This is the unofficial standard since quite some time. It also sets some more secure mount options. Review them carefully if they make sense for you and edit them as needed.

The next step is to configure your kickstart template.

Standalone Cobbler
On a standalone Cobbler server edit /var/lib/cobbler/kickstart/your-kick-start-template.ks

# Detect the used hardware type
$SNIPPET('hw-detect')
# Set up default partitioning
$SNIPPET('partitioning')

Bundled Cobbler
When using cobbler bundled with Spacewalk or Red Hat Satellite, you need to edit the Kickstart profile in the WebUI.


Navigate to Systems -> Kickstart -> Profile. Select the Kickstart profile to be modified -> System Details -> Partitioning.

Copy the two Snippets in /var/lib/cobbler/spacewalk/1, where 1 is representing your OrgId.

Alternatively you can edit them in the WebUI as well.

To check if all is working as expected, add a system to Cobbler using the Command Line Interface and have a look to the rendered Kickstart file. This can be easily done with cobbler system getks --name=blah.

Happy System installing….

Have fun 🙂

RHEV 3.1 – an overview about the new features

RHEV-M

RHEV-M

Recently Red Hat announced the public availability of RHEV 3.1.

Finally, no more Windows needed for the whole software stack 🙂

In 3.0, the new webadmin interface was already inncluded, as a tech preview and had its problems. Now with 3.1 its working great and looks neat. In contrary to 3.0, it is now listening on the standard ports 80 and 443. This will probably help users in organizations with strict proxy policies and setting.

So what else is new?

The supported number of virtual CPUs in a guest is now ridiculous 160, and RAM per guest is at ridiculous two Terabytes. But this are the least import updates.

Especially on the storage side, a lot of effort has been done and long missing features integrated.

From my point of view, the most important new feature is the possibility to have disks from more than one Storage Domain attached to a virtual machine. This would allow to install the Operating system to cheap SATA storage while data disks are super fast SSDs.

There is also support for live snapshots, but snapshots are (as on other platforms) kind of problematic because they are COW (Copy-On-Write). This can lead to I/O performance problems. Snapshots are a cool feature for i.e. taking a snapshot before updating software etc. Be sure you remove the snapshot afterwards if you want to keep a good I/O performance.

You now can use DirectLUN directly from the GUI without the usage of hooks. DirectLUN allows to attach FibreChannel and iSCSI LUNs directly to a Virtual Machine. This is great when you want to use shared filesystems such as GFS.

Another nice feature is Live Storage Migration which is a technical preview, means: Unsupported for the moment. It probably will be supported in a later version. Storage live migration is a nice feature when you need to free up some space on a storage domain and you can not shut down a VM. Be sure to power-cycle the VM in question as soon as your SLA allows it, to get rid of the Snapshot (COW here again).

If you want to script stuff or you are too lazy to open a brower, there is now a CLI available. Have a look to the documentation.

If you want to integrate RHEV deeper into your existing infrastructure, such as RHN Satellite, Cobbler, Your-super-duper-CMDB or IaaS/PaaS broker, there are two different APIs available. For the XML lovers, there is the previously known RestAPI which has some performance improvements. For the XML haters, there is now a native Python API which allows to to access RHEV entities directly as objects in your Python code. For both APIs, have a look to the Documentation.

I personally like the Python API, because a lot of other Red Hat infrastructure products come with Python APIs. So it is very easy to integrate those software pieces.

Under the hood, it is now powered by JBoss EAP6 instead of version 5. To be able to connect to standard ports 80 and 443, there is an Apache httpd with mod_proxy_ajp.

Have fun 🙂