Using Ansible to automate oVirt and RHV environments

Bored of clicking in the WebUI of RHV or oVirt? Automate it with Ansible! Set up a complete virtualization environment within a few minutes.

Some time ago, Ansible includes a module for orchestrating RHV environments. It allows you to automate the setup of such an environment as well as automating daily tasks.

Preparation

Of course, Ansible can not automate all tasks, you need to set up a few things manually. Lets assume you want your oVirt-engine or RHV-manager running outside of the RHV environment which has some benefits when it comes to system management.

  • Setup of at least two hypervisor machines with RHEL7 latest
  • Setup of the RHV-M machine with RHEL7 latest
  • Having the appropriate Redhat Subscriptions
  • A machine with Ansible 2.3 installed

Set up the inventory file

Ensure you have a inventory file like the following in place,i.e. in /etc/ansible/hosts

[rhv]
        rhv-m.example.com

[hypervisors]
        hv1.example.com
        hv2.example.com

Helper files

ovirt-engine-vars.yml

engine_url: https://rhv-m.example.com/ovirt-engine/api
username: admin@internal
password: redhat
engine_cafile: /etc/pki/ovirt-engine/ca.pem
datacenter: Default
cluster: Default

rhsm_user: user@example.com
rhsm_pass: secret

Please adjust the following example answer file for your environment.

rhv-setup.conf

# action=setup                                                                                                        
[environment:default]                                                                                                 
OVESETUP_DIALOG/confirmSettings=bool:True                                                                                            
OVESETUP_CONFIG/applicationMode=str:both                                                                                             
OVESETUP_CONFIG/remoteEngineSetupStyle=none:None                                                                                     
OVESETUP_CONFIG/sanWipeAfterDelete=bool:False                                                                                        
OVESETUP_CONFIG/storageIsLocal=bool:False                                                                                            
OVESETUP_CONFIG/firewallManager=none:None                                                                                            
OVESETUP_CONFIG/remoteEngineHostRootPassword=none:None                                                                               
OVESETUP_CONFIG/firewallChangesReview=none:None                                                                                      
OVESETUP_CONFIG/updateFirewall=bool:False                                                                                            
OVESETUP_CONFIG/remoteEngineHostSshPort=none:None                                                                                    
OVESETUP_CONFIG/fqdn=str:rhv-m.example.com                                                                                        
OVESETUP_CONFIG/storageType=none:None                                                                                                        
OSETUP_RPMDISTRO/requireRollback=none:None                                                                                                   
OSETUP_RPMDISTRO/enableUpgrade=none:None                                                                                                     
OVESETUP_PROVISIONING/postgresProvisioningEnabled=bool:True                                                                                  
OVESETUP_APACHE/configureRootRedirection=bool:True                                                                                           
OVESETUP_APACHE/configureSsl=bool:True                                                                                                         
OVESETUP_DB/secured=bool:False
OVESETUP_DB/fixDbConfiguration=none:None
OVESETUP_DB/user=str:engine
OVESETUP_DB/dumper=str:pg_custom
OVESETUP_DB/database=str:engine
OVESETUP_DB/fixDbViolations=none:None
OVESETUP_DB/engineVacuumFull=none:None
OVESETUP_DB/host=str:localhost
OVESETUP_DB/port=int:5432
OVESETUP_DB/filter=none:None
OVESETUP_DB/restoreJobs=int:2
OVESETUP_DB/securedHostValidation=bool:False
OVESETUP_ENGINE_CORE/enable=bool:True
OVESETUP_CORE/engineStop=none:None
OVESETUP_SYSTEM/memCheckEnabled=bool:True
OVESETUP_SYSTEM/nfsConfigEnabled=bool:False
OVESETUP_PKI/organization=str:example.com
OVESETUP_PKI/renew=none:None
OVESETUP_CONFIG/isoDomainName=none:None
OVESETUP_CONFIG/engineHeapMax=str:1955M
OVESETUP_CONFIG/ignoreVdsgroupInNotifier=none:None
OVESETUP_CONFIG/adminPassword=str:redhat
OVESETUP_CONFIG/isoDomainACL=none:None
OVESETUP_CONFIG/isoDomainMountPoint=none:None
OVESETUP_CONFIG/engineDbBackupDir=str:/var/lib/ovirt-engine/backups
OVESETUP_CONFIG/engineHeapMin=str:1955M
OVESETUP_DWH_CORE/enable=bool:True
OVESETUP_DWH_CONFIG/scale=str:1
OVESETUP_DWH_CONFIG/dwhDbBackupDir=str:/var/lib/ovirt-engine-dwh/backups
OVESETUP_DWH_DB/secured=bool:False
OVESETUP_DWH_DB/restoreBackupLate=bool:True
OVESETUP_DWH_DB/disconnectExistingDwh=none:None
OVESETUP_DWH_DB/host=str:localhost
OVESETUP_DWH_DB/user=str:ovirt_engine_history
OVESETUP_DWH_DB/dumper=str:pg_custom
OVESETUP_DWH_DB/database=str:ovirt_engine_history
OVESETUP_DWH_DB/performBackup=none:None
OVESETUP_DWH_DB/port=int:5432
OVESETUP_DWH_DB/filter=none:None
OVESETUP_DWH_DB/restoreJobs=int:2
OVESETUP_DWH_DB/securedHostValidation=bool:False
OVESETUP_DWH_PROVISIONING/postgresProvisioningEnabled=bool:True
OVESETUP_CONFIG/imageioProxyConfig=bool:True
OVESETUP_RHEVM_DIALOG/confirmUpgrade=bool:True
OVESETUP_VMCONSOLE_PROXY_CONFIG/vmconsoleProxyConfig=bool:True
OVESETUP_CONFIG/websocketProxyConfig=bool:True

Prepare your machines

The first Playbook ensures your machines are subscribed to RHSM and the needed repos are made available.

install_rhv.yml

---
- hosts: rhv,hypervisors
  vars_files:
    - ovirt-engine-vars.yml
  
  tasks:
  - name: Register the machines to RHSM
    redhat_subscription:
      state: present
      username: "{{ rhsm_user }}"
      password: "{{ rhsm_pass }}"
      pool: '^(Red Hat Enterprise Server|Red Hat Virtualization)$'

  - name: Disable all repos
    command: subscription-manager repos --disable=*

- hosts: hypervisors
  tasks:
    - name: Enable required repositories
      command: subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhv-4-mgmt-agent-rpms
 
- hosts: rhv
  tasks:

    - name: Enable required repositories
      command: subscription-manager repos --enable=jb-eap-7-for-rhel-7-server-rpms --enable=rhel-7-server-rhv-4-tools-rpms --enable=rhel-7-server-rhv-4.1-rpms --enable=rhel-7-server-supplementary-rpms --enable=rhel-7-server-rpms

    - name: Copy Answer File
      copy:
        src: rhv-setup.conf
        dest: /tmp/rhv-setup.conf

    - name: Run RHV setup
      shell: |
        engine-setup --config-append=/tmp/rhv-setup.conf

Run the playbook

user@ansible playbooks]$ ansible-playbook -k install_rhv.yml 
SSH password: 

PLAY [rhv,hypervisors] ************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [rhv-m.example.com]
ok: [hv1.example.com]
ok: [hv2.example.com]

TASK [Register the machines to RHSM] **********************************************************************************************************
ok: [hv1.example.com]
ok: [hv2.example.com]
ok: [rhv-m.example.com]

TASK [Disable all repos] **********************************************************************************************************************
changed: [rhv-m.example.com]
changed: [hv2example.com]
changed: [hv1.example.com]

PLAY [hypervisors] ****************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [hv1.example.com]
ok: [hv2.example.com]

TASK [Enable required repositories] ***********************************************************************************************************
changed: [hv1.example.com]
changed: [hv2.example.com]

PLAY [rhv] ************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [rhv-m.example.com]

TASK [Enable required repositories] ***********************************************************************************************************
changed: [rhv-m.example.com]

TASK [Copy Answer File] ***********************************************************************************************************************
ok: [rhv-m.example.com]

TASK [Run RHV setup] **************************************************************************************************************************
changed: [rhv-m.example.com]

PLAY RECAP ************************************************************************************************************************************
hv1.example.com : ok=5    changed=2    unreachable=0    failed=0   
hv2.example.com : ok=5    changed=2    unreachable=0    failed=0   
rhv-m.example.com       : ok=7    changed=3    unreachable=0    failed=0   

[user@ansible playbooks]$ 

Deploy your environment

Your environment is now ready to set up all the required stuff such as data centers, clusters, networks, storage etc.

rhv-deploy.yml

---
- name: Deploy RHV environment
  hosts: rhv

  vars_files: 
    - ovirt-engine-vars.yml

  pre_tasks:
  - name: Log in
    ovirt_auth:
      url: "{{ engine_url }}"
      username: "{{ username }}"
      password: "{{ password }}"
      ca_file: "{{ engine_cafile }}"
    tags:
      - always

  tasks:

  - name: ensure Datacenter "{{ datacenter }}" is existing
    ovirt_datacenters:
      auth: "{{ ovirt_auth }}"
      name: "{{ datacenter }}"
      comment: "Our primary DC"
      compatibility_version: 4.1
      quota_mode: enabled
      local: False

  - name: Ensure Cluster "{{ cluster }}" is existing
    ovirt_clusters:
      auth: "{{ ovirt_auth }}"
      name: "{{ cluster }}"
      data_center: "{{ datacenter }}"
      description: "Default Cluster 1"
      cpu_type: "Intel Haswell-noTSX Family"
      switch_type: legacy
      compatibility_version: 4.1
      gluster: false
      ballooning: false
      ha_reservation: true
      memory_policy: server
      rng_sources:
        - random

  - name: Ensure logical network VLAN101 exists
    ovirt_networks:
      auth: "{{ ovirt_auth }}"
      data_center: "{{ datacenter }}"
      name: vlan101
      vlan_tag: 101
      clusters:
        - name: "{{ cluster }}"
          assigned: True
          required: False

  - name: ensure host hv1 is joined
    ovirt_hosts:
      auth: "{{ ovirt_auth }}"
      cluster: "{{ cluster }}"
      name: hv1
      address: 192.168.100.112
      password: redhat

  - name: ensure host hv2 is joined
    ovirt_hosts:
      auth: "{{ ovirt_auth }}"
      cluster: "{{ cluster }}"
      name: hv2
      address: 192.168.100.20
      password: redhat

  - name: Assign Networks to host 
    ovirt_host_networks:
      auth: "{{ ovirt_auth }}"
      state: present
      name: "{{ item }}"
      interface: eth1
      save: True
      networks: 
        - name: vlan101
    with_items:
      - hv1
      - hv2


  - name: Enable Power Management for host1    
    ovirt_host_pm:
      auth: "{{ ovirt_auth }}"
      name: hv1
      address: 10.10.10.10
      options:
        lanplus=true
      username: admin
      password: secret
      type: ipmilan

  - name: Enable Power Management for host1
    ovirt_host_pm:
      auth: "{{ ovirt_auth }}"
      name: hv2
      address: 10.10.10.11
      options:
        lanplus=true
      username: admin
      password: secret
      type: ipmilan

  - name: Create VM datastore
    ovirt_storage_domains:
      auth: "{{ ovirt_auth }}"
      name: vms
      host: "hv2"
      data_center: "{{ datacenter }}"
      nfs:
        address: nfs.example.com
        path: /exports/rhv/vms

  - name: Create export NFS storage domain
    ovirt_storage_domains:
      auth: "{{ ovirt_auth }}"
      name: export
      host: "hv2"
      domain_function: export
      data_center: "{{ datacenter }}"
      nfs:
        address: nfs.example.com
        path: /exports/rhv/export

  - name: Create ISO NFS storage domain
    ovirt_storage_domains:
      auth: "{{ ovirt_auth }}"
      name: iso
      host: "hv2"
      domain_function: iso
      data_center: "{{ datacenter }}"
      nfs:
        address: nfs.example.com
        path: /exports/rhv/iso

Run the playbook

user@ansible playbooks]$ ansible-playbook -k rhv-deploy.yml
SSH password: 

PLAY [Deplay RHV environment] *****************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [rhv-m.example.com]

TASK [Log in] *********************************************************************************************************************************
ok: [rhv-m.example.com]

TASK [ensure Datacenter "Default" is existing] ************************************************************************************************
changed: [rhv-m.example.com]

TASK [Ensure Cluster "Default" is existing] ***************************************************************************************************
changed: [rhv-m.example.com]

TASK [Ensure logical network VLAN101 exists] **************************************************************************************************
changed: [rhv-m.example.com]

TASK [ensure host hv1 is joined] ****************************************************************************************************
changed: [rhv-m.example.com]

TASK [ensure host hv2 is joined] ****************************************************************************************************
changed: [rhv-m.example.com]

TASK [Assign Networks to host] ****************************************************************************************************************
ok: [rhv-m.example.com] => (item=hv1)
ok: [rhv-m.example.com] => (item=hv2)

TASK [Enable Power Management for host1] ******************************************************************************************************
changed: [rhv-m.example.com]

TASK [Enable Power Management for host1] ******************************************************************************************************
changed: [rhv-m.example.com]

TASK [Create VM datastore] ********************************************************************************************************************
changed: [rhv-m.example.com]

TASK [Create export NFS storage domain] *******************************************************************************************************
changed: [rhv-m.example.com]

TASK [Create ISO NFS storage domain] **********************************************************************************************************
changed: [rhv-m.example.com]

PLAY RECAP ************************************************************************************************************************************
rhv-m.example.com       : ok=13   changed=10   unreachable=0    failed=0   

[user@ansible playbooks]$ 

Further readings

Conclusion

With the help of Ansible you can automate a lot of boring tasks in a convenient way. You may even merge the two playbooks into one, be aware that the RHV-M setup will fail if its already set up.

Have fun 🙂

Signing Linux Kernel Modules and enforce to load only signed Modules

Introduction

With the enforcement of loading only signed Linux Kernel Modules you can greatly enhance the security of your Systems.

There are basically two methods of enforcement: Secure (UEFI) Boot and the other is a grub parameter. When using Secure boot you can sign own (or 3rd party) Kernel modules by yourself and add your public key as a MOK (Machine Owner Key) in UEFI. When not using Secure Boot, you can not load self signed modules due to the lack of a capability of storing MOKs. At least you can prevent loading unsigned Modules.

Unfortunately I was unable to test Secure Boot with a KVM virtual machine, the MOK was not added. Also on hardware it does not seem to work on all machines. I failed with my Lenovo T450s Notebook. Finally I succeeded with my Workstation with a Gigabyte Z97-D3H motherboard using Fedora 25. If someone has a solution with virtual machines, please let me know.

About Secure boot

Basically it is a chain of trust with x509 certificates. UEFI Firmware -> Shim First-Stage Bootloader -> Grub Second Stage Bootloader -> Kernel -> Modules.

This adds complexity. If something goes wrong it’s not always easy to figure out where and why it goes wrong.

Secure Boot is not without some controversy, its dominated by Microsoft, only Microsoft can sign bootloaders. Yes, the Shim bootloader is signed by Microsoft. If Microsoft decides to no longer sign Shim (or any non-MS loader), the whole Linux landscape is not able to use Secure boot anymore. As of today, most UEFI Firmware let users choose to turn off Secure Boot, how about that in the future?

Creating a dummy Kernel Module

First you need to build a unsigned Kernel module. A “Hello Wold” is good enough

Install the required RPMs

yum -y install kernel-devel.x86_64 gcc keyutils mokutil.x86_64

hello.c

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Luc de Louw");
MODULE_DESCRIPTION("Hello World Linux Kernel Module");

static int __init hello_init(void)
{
    printk(KERN_INFO "Hello world!\n");
    return 0;
}

static void __exit hello_exit(void)
{
    printk(KERN_INFO "Unloading Hello world.\n");
}

module_init(hello_init);
module_exit(hello_exit);

Makefile

obj-m += hello.o

all:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

install:
        cp hello.ko  /lib/modules/$(shell uname -r)/extra
        depmod

Building

make && make install

Testing

modprobe hello

To remove the module afterwards run

rmmod hello

Set up the enforcement of loading only signed modules

This is only needed on machines without secure boot.

Add module.sig_enforce=1 to GRUB_CMDLINE_LINUX in /etc/default/grub

/etc/default/grub

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg_rhel7test/lv_root rd.lvm.lv=vg_rhel7test/lv_swap module.sig_enforce=1"
GRUB_DISABLE_RECOVERY="true"

The next step is to update the GRUB configuration. Please check if you are using UEFI or BIOS on your system first.

On systems with UEFI

[root@rhel7uefi ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

On BIOS systems

[root@rhel7test ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot your system.

Testing

modprobe hello

The Module will not load. You will see an error message instead:

modprobe: ERROR: could not insert 'hello': Required key not available

Signing the module

Needless to say that this must be done on a protected system and not on production servers.

First you need to create an OpenSSL config file like this:

x509.conf

cat >>/tmp/x509.conf <<EOF
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
prompt = no
string_mask = utf8only
x509_extensions = extensions

[ req_distinguished_name ]
O = Example, Inc.
CN = Example, Inc. Kernel signing key
emailAddress = jdoe@example.com

[ extensions ]
basicConstraints=critical,CA:FALSE
keyUsage=digitalSignature
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid
EOF

Generating the Keypair

[root@rhel7uefi ~]# openssl req -x509 -new -nodes -utf8 -sha256 -days 99999 -batch -config /tmp/x509.conf -outform DER -out pubkey.der -keyout priv.key 

Adding the Public key as a MOK (Machine Owner Key)

Note: This does only work on systems with UEFI, on BIOS machines you will get an error.

[root@rhel7uefi ~]# mokutil --import pubkey.der

You will be prompted for a password that will be used for the second part of the MOK enrollment. Reboot your machine, the shim UEFI Key Manager will appear. After waiting 10sec the system continues to boot the normal system.

You can list the enrolled keys with

root@rhel7uefi ~]# mokutil --list-enrolled

Signing the Module

After successfully enroll the MOK you can sign and test the Kernel Module.

First lets have a look at the Module

root@rhel7uefi ~]# modinfo hello
filename:       /lib/modules/3.10.0-514.16.1.el7.x86_64/extra/hello.ko
description:    Hello World Linux Kernel Module
author:         Luc de Louw
license:        GPL
rhelversion:    7.3
srcversion:     4A5235839200E8580493A17
depends:        
vermagic:       3.10.0-514.16.1.el7.x86_64 SMP mod_unload modversions 
[root@rhel7uefi ~]# 

Sign it.

[root@rhel7uefi ~]# /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 priv.key pubkey.der /lib/modules/$(uname -r)/extra/hello.ko

Lets have a look to the module again.

[root@rhel7uefi ~]# modinfo hello.ko
filename: /root/hello.ko
description: Hello World Linux Kernel Module
author: Luc de Louw
license: GPL
rhelversion: 7.3
srcversion: 4A5235839200E8580493A17
depends:
vermagic: 3.10.0-514.16.1.el7.x86_64 SMP mod_unload modversions
signer: Example, Inc. Kernel signing key
sig_key: 71:F7:AA:48:60:A0:B5:D9:D8:A8:1D:A4:6F:92:30:DF:87:35:81:19
sig_hashalgo: sha256
[root@rhel7uefi ~]#

Now you should be able to load your module.

[root@rhel7uefi ~]# modprobe hello

If something went wrong, you will see an error message such as

modprobe: ERROR: could not insert 'hello': Required key not available

Syslog and Journald are more verbose:

Request for unknown module key 'Example, Inc. Kernel signing key: 22e37ef0c0784c7a2c1e2690dc8b27c75533b29d' err -11

Further reading

Conclusion

If your hardware works with secure boot, you can easily enhance security and keep the flexibility to load 3rd party Kernel modules by signing them.

On virtual machines you can make use of signing enforcement which prevents to load any 3rd party module. This may, or may not be a problem.

A major drawback I see is scalability. It may be okay to manually enroll keys on a few workstations or notebooks. On a larger enterprise scale I see problems. For really large environments, you can probably talk with the hardware vendor to include the MOK (Machine Owner Key) factory preinstalled.

Have fun 🙂

Setting up a 6in4 tunnel with Fedora

Why using IPv6 Tunnels anyway?

Today, most Internet access providers are IPv6 enabled. However, unfortunately the majority of them do not provide a static /64 prefix, you will get it dynamically assigned. Some providers can assign you a static prefix for a surcharge.

That’s useless if you want to ensure end-to-end connectivity with your Gadgets at home.

Choosing a tunnel provider

Since 2004 I had my own IPv6 prefix from SixXS. Pretty sad that they are shutting down its services on 2017-06-06.

Time to look for an alternative. Wikipedia has a list of public tunnel brokers. Most brokers are providing only PoP’s in one country. For most users, the only option left is Hurricane Electric which offers tunnels to PoP’s on three continents in various cities.

Setup in Fedora

The whole setup is rather simple, there is just one thing you should keep in mind. The provided Client IPv6 Address is not in the same subnet as the Routed /64. You easily copy-paste the wrong address and you will end up in a nice routing loop. The difference is i.e. 2001:470:6c:something vs. 2001:470:6d:something, only the one character of difference. It was taking me more that an hour to figure out 😉

Tunnel configuration

Create a new interface for the tunnel.

cat >>/etc/sysconfig/network-scripts/ifcfg-he-ipv6 <<EOF
DEVICE=he-ipv6
TYPE=sit
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
# The IPv4 address depends on the PoP you choose
IPV6TUNNELIPV4=216.66.86.114
# That is the IPv6 address of the client, not from the routed prefix
IPV6ADDR=2001:db8:dead:beef::2/64
EOF

LAN interface configuration

In my case I use a bridge to be able to provide IPv6 connectivity not only for the LAN but for Wifi and VPN as well.

cat >>/etc/sysconfig/network-scripts/ifcfg-br0 <<EOF
DEVICE=br0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=192.168.100.1
NETMASK=255.255.255.0
IPV6_AUTOCONF=no
IPV6INIT=yes
IPV6TO4INIT=no
# That is a random IP from your routed /64 prefix. Usually just use the first one
IPV6ADDR=2001:db8:cafe:1::1/64
EOF

 

Enable IPv6 routing

echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf

Setting the default device for IPv6 routing

echo "IPV6_DEFAULTDEV=he-ipv6"  >> /etc/sysconfig/network

Setting up the Route Advertisement Daemon (RADVD

There are several ways of how to configure the clients with an IPv6 address. DHCP6, Static manual configuration and the most easy way is to use RADVD which tells the clients which prefix to use (prefix + fffe + MAC). The client itself adds the MAC address on top of the prefix.

Your clients will always get the same IPv6 address, this may be a privacy problem for you or not. In contrary to SixXS, Hurricane Electric does not provide your name and address to whois, only the city and the ZIP code is made public.

Install radvd if not yet done

router:~# dnf install radvd
router:~# systemctl enable radvd.service

Configuration for the example of the prefix 2001:db8:cafe:1/64

cat >> /etc/radvd.conf << EOF
 interface br0
 {
        AdvSendAdvert on;
        MinRtrAdvInterval 30;
        MaxRtrAdvInterval 100;
        AdvLinkMTU 1480;
        prefix 2001:db8:cafe:1::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
 
} ;
EOF

After restarting your network your done, have fun with IPv6 🙂

Setting up DNS

I’m not going into the details here. I’m using FreeIPA for DNS management, DNS entries are created automatically when you enroll your clients. The only thing you need to do is adding the prefix to be able to do reverse lookups.

[root@ipa1 ~]# ipa dnszone-add --name-from-ip=2001:db8:cafe::/64 --dynamic-update=true 
Zone name [0.0.0.0.e.f.a.c.8.b.d.0.1.0.0.2.ip6.arpa.]: 
  Zone name: 0.0.0.0.e.f.a.c.8.b.d.0.1.0.0.2.ip6.arpa.
  Active zone: TRUE
  Authoritative nameserver: ipa1.example.com.
  Administrator e-mail address: hostmaster
  SOA serial: 1490512663
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  BIND update policy: grant EXAMPLE.COM krb5-subdomain 0.0.0.0.e.f.a.c.8.b.d.0.1.0.0.2.ip6.arpa. PTR;
  Dynamic update: TRUE
  Allow query: any;
  Allow transfer: none;
[root@ipa1 ~]# 

Reverse DNS delegation

Hurricane Electric allows you to delegate DNS lookups of your prefix to your DNS server(s). Make use of that is good practice.

Read further

Fancy stuff

If you finished setting up all your services such as DNS, HTTP, SMTP etc. with IPv6, get “certified” at https://ipv6.he.net/certification/cert-main.php and get a fancy batch like this: IPv6 Certification Badge for ldelouw

Have fun! 🙂

Audit your systems for security compliance with OpenSCAP

OpenSCAP logoIntroduction to (Open)SCAP

SCAP stands for Security Content Automation Protocol. It is an open standard which defines methods for security policy compliance, vulnerability management and measurement etc. This article focuses on the operating system compliance part of SCAP.

It comes originally from the US National Institute of Standards and Technology (NIST) to provide a way for US government agencies to audit its systems for regulatory compliance.

OpenSCAP is a NIST validated open source implementation of SCAP.

Why should I make use of OpenSCAP anyway?

Lot of people will ask this question to them self, in particular System Administrators and Engineers since they are not IT Security Officers.

The simple answer is that you just sit down with the IT Security Officer once and define which systems need to be compliant to what regulatory, With OpenSCAP you can always ensure the systems are configured according the the policy (or policies).

Organizations that need to be compliant according to a official policy will sooner or later facing an external security audit. I experienced that several times, its a nightmare. If you can proof that your systems are scanned regularly with the SCAP standard, you will be very well prepared, an external auditor will not bug you for a long time.

Abbreviations, abbreviations, abbreviations

Its obvious, government agencies love abbreviations 😉 Lets explain the two most important ones.

XCCDF

Extensible Configuration Checklist Description Format. This files, i.e. /usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml contain descriptions used for auditing a system against compliance to a policy.

This files are usually included in your distribution and are updated if needed.

OVAL

Open Vulnerability and Assessment Language. Its used to detect vulnerabilities and patches.

Since vulnerabilities and patches are popping up very quickly they need to be downloaded and distributed to all systems to be audited on a regular base (i.e. daily).

OVAL files can be downloaded as listed below:

Organizations using System Management Tools such as Red Hat Satellite or SUSE Magager will not profit from OVAL patch scans as those products will report which patches have been applied or not by themself. Nevertheless, additional OVAL scans add the benefit of vulnerability scanning regardless of installed patches.

More Abbreviations

More abbreviations and a short description of them can be found here: https://www.open-scap.org/resources/acronyms/

OpenSCAP Scap Security Guide (SSG)

There are a lot of regulations out there. Government of some countries releases policies and sometimes SCAP content for some Operating Systems, mostly RHEL and Windows. The SSG Project works on collecting and implementing content for this policies for the operating systems as well as for some other software such as JBoss. Included in the scap-security-guide are the most important US Government and PCI-DSS for RHEL. Only available for Debian at the moment is the content for the French ANSSI DAT-NT28.

The only Linux distributions I’m aware of that provides packages for scap-security-guide are RHEL and Fedora. However, upstream there is some content for more distributions available. I really hope that all important and fine distributions such as SLES, Debian and Ubuntu will jump on the bandwagon.

Regulations covered by OpenSCAP SSG

Here a list of what is available for the most important Linux distributions.

Red Hat Enterprise Linux 7

  • PCI-DSS (Payment Card Industry – Data Security Standard), Commercial – USA
  • C2S (Commercial Cloud Services), Government – USA
  • USGCB/STIG (United States Government Configuration Baseline/Security Technical Implementation Guide), Government – USA
  • CNSSI 1253 (Committee on National Security Systems), Government – USA
  • CJIS (Criminal Justice Information Services), Government – USA

Debian and Ubuntu

Officially there is nothing available. Its is currently under development, see https://github.com/OpenSCAP/scap-security-guide/tree/master/Ubuntu/16.04 and https://github.com/OpenSCAP/scap-security-guide/tree/master/Debian/8.

As of 2017-03-04 compiling fails.

  • ANSSI DAT-NT28 (Agence nationale de la sécurité des systèmes d’information), Government – France

Suse Linux Entrprise Server

Suse does not provide the scap-security-guide package and there is no XCCDF content for regulatory compliance checks delivered by Suse. However, some basic tests are available. It is not clear if Suse has some plans to join the scap-security-guide community, would be nice to see that. SLES customers can open a support case at https://scc.suse.com/login and ask for enhancement.

Using SCAP content without scap-security-guide

You can make use of SCAP content without the OpenSCAP security guide. Its rather complex and not covered in this article.

Installing the required packages

RHEL 7

[root@server ~]# yum -y install scap-security-guide

All required dependencies will be installed as well

Debian and Ubuntu

root@ubuntu:~# aptitude install python-openscap

All required dependencies will be installed as well

SLES12sp2

sles12sp2:~ # zypper install openscap openscap-content openscap-extra-probes openscap-utils

All required dependencies will be installed as well

Tailoring profiles

For most users it is probably too much to secure its systems according to military standards which includes turning off USB support and the like.

The most important civil regulatory by far is PCI-DSS. Each company handling kind of Credit- or Debitcard data must obey the current standard. As of writing this article this is version 3.2.

PCI-DSS is a de-facto standard in Enterprise Linux environments.

Of course it makes sense for all kind of companies to secure its systems. On systems which are not exposed, security policies can be more relaxed.

Also good to know is that some tests simply do not apply to your system. I.e. if you are using a centralized identity management software such as Redhat IdM with IPA or Microsoft Active Directory then the central instance will take care about the password policies, not the particular system to be audited.

Installation of the SCAP Workbench

The Scap Workbench is available in RHEL to be installed by yum, a binary for Windows and Mac OS is available as well. Needless to say that the source code is available.

Downloads: https://github.com/OpenSCAP/scap-workbench/releases

Usage

In the following examples, we disable the check for AIDE.

SCAP-Workbench Screencast

SCAP-Workbench Screencast

You can save the tailoring file as a single XML file or even better safe it as an RPM for easy distribution to all your systems.

Scanning

The usage is the same on all tested Linux distributions. Be aware, XCCDF scanning makes no sense w/o any SCAP content. If your distribution does not provide you the necessary data, 3rd party providers may.

RHEL 7 comes with the scap-workbench which is GUI that allows you to scan the local or remote systems via SSH. The scap-workbench is a nice tool to scan a handful of servers manually but not to scan a whole zoo of servers.

You also can scan your systems with the CLI on the host itself. Kind of automation can be done with i.e with Ansible.

Manual Scan

The oscap info command gives you an overview which profiles are available.

[root@server ~]# oscap info /usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml
Document type: XCCDF Checklist
Checklist version: 1.1
Imported: 2017-02-14T13:33:08
Status: draft
Generated: 2017-02-14
Resolved: true
Profiles:
        standard
        pci-dss
        C2S
        rht-ccp
        common
        stig-rhel7-workstation-upstream
        stig-rhel7-server-gui-upstream
        stig-rhel7-server-upstream
        ospp-rhel7-server
        nist-cl-il-al
        cjis-rhel7-server
Referenced check files:
        ssg-rhel7-oval.xml
                system: http://oval.mitre.org/XMLSchema/oval-definitions-5
        ssg-rhel7-ocil.xml
                system: http://scap.nist.gov/schema/ocil/2
        http://www.redhat.com/security/data/oval/Red_Hat_Enterprise_Linux_7.xml
                system: http://oval.mitre.org/XMLSchema/oval-definitions-5
[root@server ~]# 

Lets choose pci-dss and start a scan:

[root@server ~]# oscap xccdf eval --profile pci-dss --results scan.xml --report scan.html /usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml
Title   Ensure Red Hat GPG Key Installed
Rule    ensure_redhat_gpgkey_installed
Ident   CCE-26957-1
Result  pass

Title   Ensure gpgcheck Enabled In Main Yum Configuration
Rule    ensure_gpgcheck_globally_activated
Ident   CCE-26989-4
Result  pass
[Lot of Output immited]

The parameter –results saves the result in a HTML file.

Automated scanning with Redhat Satellite 6

Users of Redhat Satellite 6 can schedule scans of large server farms. The screenshots shows you how compliance tests can be presented to a IT Security Officer.

Compliance Report

Compliance Overview

The Compliance report shows a overview of hosts and a brief look at how many test have been failed.

Compliance Report Detail view

Compliance Report Detail view

The Compliance report detail shows which test have been failed. It also provides a description of each topic.

Host details

Host details

The detail view of a host shows that this host is not compliant. In this case, security errata must be applied and the host must be reconfigured to get compliant to the security policy.

Alternatives to OpenSCAP

There are a few alternatives to OpenSCAP as listed by the NIST’s Security Content Automation Protocol Validated Products.

Further reading