Installing RHEL 8 on Hetzner root servers

Hetzner is a very popular provider for so called root servers and VPS (Virtual private Servers) located in Germany with datacenters in Germany and Finnland. They are quite affordable and have good services as well.

The default installation image, sorry Hetzner, is crap (i.e. no logical volumes). The rescue system is not only a nice tool to recover from botched system configurations, it also comes with an image installer which allows users to install a custom system. The default selection is Arch, CentOS 7.6, some Debian and Ubuntu versions but no CentOS8 or even RHEL of any version.

Game over? No, you can create an own image, just put it on a web server and configure the installation accordingly.

Create your image

I just installed a VM from the ISO image that can be downloaded here.

Select the “Minimal” installation, register the machine at Red Hat with subscription-manager register, attach the RHEL OS with subscription-manager attach –auto and install tar and mdadm.

Caution! You need to put the image on a public available web space! Unregister your system and delete the SSH host keys!

subscription-manager unregister
subscription-manager clean
rm -rf /etc/ssh/ssh_host_*

An other important fact is that the Hetzner installer only knows about CentOS and only about version 7. On CentOS8 and RHEL8, the dracut binary is located elsewhere than on RHEL7, you need to create a Symlink.

[root@localhost /]# ln -s /usr/bin/dracut /sbin/dracut

For the same reason you need to name the tarball CentOS-<something>

The next step is to create the tarball. Important to know is that the tarball must not contain /dev, /proc and /sys folders.

[root@localhost /]# tar cJvf CentOS-80-el-x86_64-minimal.tar.xz --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/CentOS-80-el-x86_64-minimal.tar.xz /

Copy the resulting tarball to a public web space:

[root@localhost /]# scp CentOS-80-el-x86_64-minimal.tar.xz blog.delouw.ch:/var/www/html/blog.delouw.ch/pub

Yes, you can use my image if you like: http://blog.delouw.ch/pub/CentOS-80-el-x86_64-minimal.tar.xz

Reboot your server into the rescue mode

Point your browser to https://robot.your-server.de/server, select your server, click on “Rescue” on the server menue, select “64 bit” and click on “Activate Rescue System”. The root password will be shown which will be needed to log in to the system.

Configure the installer

Configuring the installer is straight forward. Create a file called config.txt with the following content:

DRIVE1 /dev/sda
DRIVE2 /dev/sdb
SWRAID 1
SWRAIDLEVEL 1
BOOTLOADER grub
HOSTNAME localhost.localdomain
PART /boot ext2     1024M
PART lvm   rhel       all

LV rhel   root   /       xfs     20G
LV rhel   swap   swap    swap     16G
LV rhel   tmp    /tmp    xfs      1G
LV rhel   home   /home   xfs      4G
LV rhel   var   /var   xfs      4G
LV rhel   var_log   /var/log   xfs      4G
LV rhel   var_log_audit   /var/log/audit   xfs      4G

IMAGE http://blog.delouw.ch/pub/CentOS-80-el-x86_64-minimal.tar.xz

The partitioning is according to PCI-DSS. Note that the installer can not set the mount options needed by PCI-DSS (and a lot of other regularities).

Side note: The Hetzner installer refuses to install the image if /boot is of type xfs. You need to choose ext2,3 or ext4. I’m using ext2 as this is the most efficient file system, fsck on unclean shutdown for 1GByte takes just a few seconds, it does not matter.

Run the installer

Just run it and expect an error….

installimage -a -c config.txt

Hetzner Installer

The installer tries to install some packages which will, of course, fail as the system is not registered to RHSM yet.

[14:26:11] # Running some centos specific functions
[14:26:11] # chroot: chkconfig iptables off
[14:26:11] :   error reading information on service iptables: No such file or directory
[14:26:11] # chroot: chkconfig ip6tables off
[14:26:11] :   error reading information on service ip6tables: No such file or directory
[14:26:11] # chroot: chkconfig postfix off
[14:26:11] :   error reading information on service postfix: No such file or directory
[14:26:11] # Testing and setup of cpanel image
[14:26:11] # chroot: yum check-update
[14:26:12] :   Updating Subscription Management repositories.
[14:26:12] :   Unable to read consumer identity
[14:26:12] :   This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
[14:26:12] :   Error: There are no enabled repos.
[14:26:12] # chroot: yum -y install polkit
[14:26:12] :   Updating Subscription Management repositories.
[14:26:12] :   Unable to read consumer identity
[14:26:12] :   This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
[14:26:13] :   Error: There are no enabled repos.
[14:26:13] => FAILED
[14:26:13] :   report install.conf to rz-admin: 1281870
[14:26:13] :   report debug.txt to rz-admin: ok
[14:26:13] cleaning up

You can just ignore that error messages, it does not matter.

Finalizing

Reboot the system and log in with the root password provided for the rescue system when activating the rescue system at https://robot.your-server.de/server.

Change the password (needless to say?)

Continue to configure the system according your needs.

Have fun 🙂

Migrating from CentOS8 to RHEL8

There are various reasons why to migrate from CentOS to RHEL. Quicker access to bugfixes and new minor releases as well as having a fully commercially supported system. Unfortunately most providers do not have an option to install RHEL but CentOS instead.

There are different tutorial on the net how to migrate from RHEL to CentOS but almost no information about the other way round. It is quite simple and at the end of the day you have only Red Hat Packages installed.

In 2012 I wrote an article about Migrating from CentOS6 to RHEL6. Subsequently I’ve posted a similar article for RHEL 7: Migrating from CentOS7 to RHEL7. Now its time for an update.

Disclaimer

Some of the procedures can be destructive for your system and/or your data. I’m not taking any responsibility for any damage casue. Take a full backup of your system before even thinking about trying this procedure!

Also an important to note is that such a procedure is not supported by Redhat.

Requirements

There are only two things you need

  • A valid RHEL subscription obtained from Redhats online store
  • A RHEL8 ISO-Image which corresponds with your current CentOS minor release (or newer) which can be downloaded at Redhat downloads

Preparations

Be sure you activated your subscription.

Mount the ISO image on your CentOS8 machine:

[root@centos8 ~]# mount /dev/cdrom /mnt -o loop

Go to /mnt/BaseOS/Packages and install the packages we need:

[root@centos8 Packages]# rpm -e subscription-manager-rhsm-certificates --nodeps
[root@centos8 Packages]# yum localinstall subscription-manager-1*
[root@centos8 Packages]# yum localinstall subscription-manager-rhsm-certificates-1*

(Re)Move your CentOS repos
To avoid conflicts between CentOS and Redhat Repositories you need to get rid of them. Remove them or just keep a copy.

[root@centos8 Packages]# mkdir /etc/yum.repos.d.centos
[root@centos8 Packages]# mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d.centos

Force-remove the centos-release and yum RPMs

[root@centos8 Packages]# rpm -e centos-release --nodeps
[root@centos8 Packages]# yum localinstall redhat-release-8*

Unmount the ISO
This is important, otherwise yum reinstall will fail!

[root@centos8 ~]# umount /mnt

Register your system

To get access to RHEL repositories, you need to register your system. The username “user@example.com” must be replaced with your username. The ID is a randomly generated UUID.

[root@centos8 ~]# subscription-manager register
Registering to: subscription.rhsm.redhat.com:443/subscription
Username: user@example.com
Password: 
The system has been registered with ID: 8aa7ef30-e834-4889-9e07-c5e8ce31e681 
The registered system name is: centos8.example.com
[root@centos8 ~]# 

Attach the system to a subscription

Usually it is just good enough to auto-attach the subscription needed.

[root@centos8 ~]# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64
Status:       Subscribed

[root@centos8 ~]#

Review enabled repositories

Sometimes you dont want to use all the repos provided. The simplest way is just to disable all and re-enable those you need.

[root@centos8 ~]# yum repolist
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                 3.0 MB/s | 9.3 MB     00:03    
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                    2.9 MB/s | 8.1 MB     00:02    
repo id                                              repo name                                                                     status
rhel-8-for-x86_64-appstream-rpms                     Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                      6,031
rhel-8-for-x86_64-baseos-rpms                        Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                         2,193
[root@centos8 ~]# 

Adjust with subscription manager repos –disable|enable as needed.

Changing the Distribution

Now we have all requirements met, lets reinstall the packages.

[root@centos8 ~]# yum -y reinstall "*" 

Cleanup

Now we need to manually clean up the CentOS specific packages that have not been reinstalled.

[root@centos8 ~]# rpm -qa --queryformat "%{NAME} %{VENDOR}\n" | grep -i centos | cut -d' ' -f1
containers-common
centos-logos
glibc
libfprint
containernetworking-plugins
python3-dnf-plugin-spacewalk
platform-python
glibc-common
python3-rhnlib
slirp4netns
gpgme
rhn-client-tools
dnf-plugin-spacewalk
podman
python3-libs
qemu-guest-agent
glibc-langpack-en
container-selinux
python3-gpg
runc
buildah
fuse-overlayfs
python3-rhn-client-tools
oci-systemd-hook
[root@centos8 ~]# 

Most of this packages are not problematic to remove and reinstall. Unfortunately this is not the case with the glibc packages. Do not try to remove them with –nodeps, you system will stop working. The problem is that the CentOS delivered packages seems to be newer than that provided by RHEL. This can be solved by downgrading the packages.

That may be the same with other packages as well.

[root@centos8 ~]# yum downgrade glibc

At the end, there will be one package left: python3-gpg. Due to some dependencies it can only be reinstalled after the rest of the packages are reinstalled/downgraded.

[root@centos8 ~]# yum reinstall python3-gpg

Check if there are still RPMs of vendor “Centos” installed:

[root@centos7 ~]# rpm -qa --queryformat "%{NAME} %{VENDOR}\n" | grep -i centos | cut -d' ' -f1

This should return nothing, almost all is now RHEL8. The only traces left are the previously installed Kernels. They will get deleted over time when installing (updating) new Kernels.

In my case I just used CentOS8 “Server” installation. If you plan to go down this road, please clone the system first for testing before migrating the actual system.

Support by Redhat

Will the converted machine be supported after this procedure? Well, officially it is not supported, but if there are no traces of CentOS left on the machine…

As mentioned before, better install RHEL in the first place 🙂

Using Data Deduplication and Compression with VDO on RHEL 7 and 8

Storage deduplication technology has been on the market for quite some time now. Unfortunately all of the implementations have been vendor specific proprietary software. With VDO, there is now an open source Linux native solution available.

Red hat has introduced VDO (Virtual Data Optimizer) in RHEL 7.5, a storage deduplication technology bough with Permabit in 2017. Of course it has been open sourced since then.

In contrast to ZFS which provides the same functionality on the file system level, VDO is an inline data reduction which works on block device level, it is file system agnostic.

Use cases

There are basically two major use cases: VM Storage and Object Storage Backends.

VM Storage

The main use case is storage for virtual machines where a lot of data is redundant, i.e. the base operating system of the VMs. This allows to deduplicate the data on disk on a large scale, think about 100 VMs where the operating system takes about 5Gbyte each will be reduced to approx. 5 Gbyte instead of 500 Gbyte.

Typically VM storage can be over committed by factor 10.

Object- and Block storage backends

As a backend for CEPH and Glusterfs, it is recommended to not over commit more than factor 3. The reason for the lower over commitment is that the storage administrator usually does not know what kind of data will be stored on it.

Availability

VDO is available since RHEL 7.5, it is included in the base subscription. At the moment it is not available for Fedora (yet).

The source code is available on github:

At the moment the Kernel code is not yet in the upstream Mainline Kernel, it is ongoing work to get it into the Mainstream Kernel.

Typical setup

Physical disk -> VDO -> Volumegroup -> Logical volume -> file system.

Block device can be a physical disk (or a partition on it), multi path device, LUKS disk, or a software RAID device (md or LVM RAID).

Restrictions

You can not use LVM cache, LVM snapshots and thin provisioned logical volumes on top of VDO. Theoretically you can use LUKS on top of VDO, but it makes no sense because there is nothing to deduplicate. Needless to say that VDO on top of a VDO device does not make any sense as well. Be aware that you can not make use of partitioning or (LVM) Raid on top of VDO devices, all that things should be done in the underlying layer of VDO.

When using SAN, check if your storage box already does deduplication. In this case VDO is useless for you.

Installation

Its straight forward:

[root@vdotest ~]# yum -y install vdo kmod-kvdo

Create the VDO volume

In this test case, I attached a 110Gbyte disk, created a 100 GByte partition and will over commit it by factor 10.

Warning! As of writing this article, never use a whole physical disk, use a partition instead and leave some spare space in the disk to avoid data loss! (see further below)

[root@vdotest ~]# vdo create --name=vdo1 --device=/dev/vdb1 --vdoLogicalSize=1T

Creating volume group, logical volume and file system on top of the VDO volume

[root@vdotest ~]# pvcreate /dev/mapper/vdo1
[root@vdotest ~]# vgcreate vg_vdo /dev/mapper/vdo1
[root@vdotest ~]# lvcreate -n lv_vdo vg_vdo -L 900G
[root@vdotest ~]# mkfs.xfs -K /dev/vg_vdo/lv_vdo
[root@vdotest ~]# echo "/dev/mapper/vg_vdo-lv_vdo       /mnt    xfs     defaults,x-systemd.requires=vdo.service 0 0" >> /etc/fstab

Display the whole stack

[root@vdotest ~]# lsblk /dev/vdb
NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vdb                 252:16   0  110G  0 disk 
└─vdb1              252:17   0  100G  0 part 
  └─vdo1            253:7    0    1T  0 vdo  
    └─vg_vdo-lv_vdo 253:8    0  900G  0 lvm  /mnt
[root@vdotest ~]#

Populate the disk with data

The ideal test for VDO is to put some real-life VM-Images to the file system on top of it. In this case I scp’ed three IPA server and some instances to that file system. This kind of systems are all quite similar, the disk space saved is tremendous. The total size of the vm images is 105G

Lets have a look:

[root@vdotest ~]# df -h /mnt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg_vdo-lv_vdo  900G  105G  800G  12% /mnt
[root@vdotest ~]# 
[root@vdotest ~]# ll -h /mnt
total 101G
-rw-------. 1 root root 21G Dec 17 09:41 ipa1.lab.delouw.ch.qcow2
-rw-------. 1 root root 21G Dec 17 09:47 ipa1.ldelouw.ch
-rw-------. 1 root root 21G Dec 17 09:54 ipa2.ldelouw.ch
-rw-r--r--. 1 root root 21G Dec 17 09:58 ipaclient-rhel6.home.delouw.ch
-rw-------. 1 root root 21G Dec 17 10:03 ipatest.delouw.ch.qcow2
[root@vdotest ~]#

Lets use the vdostats utility to display the actually used storage on disk:

[root@vdotest ~]# vdostats --si
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo1        107.4G     15.2G     92.2G  14%           89%
[root@vdotest ~]#

Performance Tuning

There are a lot of parameters that can be changed. Unfortunately the documentation available at the moment is rudimentary, thus its more a guesswork than facts.

  • Number of worker threads of different kind
  • Enable or Disable compression

On machines with a lot of CPUs using more threads than the defaults can dramatically boost performance. man 8 vdo gives a glimpse of the different parameters related to threads.

Compression is a quite expensive operation. On top of that, depending on the kind of data you are storing, it does not make much sense to use compression (Well, deduplication is kind of compression as well).

Pitfalls

Be aware! With every storage deduplication solution there comes a big pitfall: The logical volume on top of VDO shows free disk space while the actual disk space on the physical disk can be (almost) exhausted. You need to carefully monitor the actual disk usage.

The fill grade can rapidly change if the data to be stored contains a lot of non-deduplicatable and/or compressible data. A good example is virtual machine images containing a LUKS encrypted disk, In such a case, use LUKS on the storage, not on the VM level.

Even if you update one virtual machine, the delta to other machine images will grow and less physical space is available.

VDO comes with a few Nagios plugins which are very useful for alerting administrators in the cause the available physical disk is filling up. They are located in /usr/share/doc/vdo/examples/nagios

According to df -h, on my test system there is still 800 Gbyte available. What happens if I store my 700 Gbyte Satellite 6 image? The data is mostly RPMs which are already compressed quite well. Lets see….

After a transfer of approx 155 Gbyte, the physical disk got full and the file system is inaccessible. I was hitting the worst case that can happen: Complete and unrecoverable data loss.

The df command shows some 241 Gbyte free.

[root@vdotest ~]# df -h |grep mnt
/dev/mapper/vg_vdo-lv_vdo                900G  241G  660G  27% /mnt
[root@vdotest ~]#

The vdostat command tells a different story, like expected.

[root@vdotest ~]# vdostats --si
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo1        107.4G    107.4G      0.0B 100%           59%
[root@vdotest ~]# 

When attempting to access the data, there will be an I/O error.

[root@vdotest ~]# ll -h /mnt
ls: cannot access /mnt: Input/output error
[root@vdotest ~]# 

Thats bad. I mean really bad. The device is not accessible anymore.

xfs_repair does not work. Do not attempt to make use of the -L option! Your file system will be gone.

Recovering from a full physical disk

Lets resize the partition instead. First unmount the file system

[root@vdotest ~]# umount /mnt

Delete and recreate the partition using fdisk

[root@vdotest ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): d
Selected partition 1
Partition 1 is deleted

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
First sector (2048-41943039, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): 
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@vdotest ~]# partprobe
[root@vdotest ~]#
[root@vdotest ~]# vdo growPhysical -n vdo1

Run a file system check.

Now you are able to mount the file system again and your data is available again.

Documentation

Red Hat maintains a nice documentation about storage administration, VDO is covered by an own chapter. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/storage_administration_guide/#vdo

Conclusion

The technology is very interesting and will kick some ass. Storage deduplication will be more and more important, with VDO there is now a Linux native solution for that.

At the moment it is quite dangerous to use VDO in production. Filling up a physical disk without spare space is an unrecoverable error, a complete data loss. That means: Always create the VDO device on top of a partition that is not using the whole disk or another device that can grow in size to prevent data loss.

If you plan to use VDO in production make sure you have a proper monitoring in place that alerts quite ahead of time to be able to take corrective action.

Nevertheless: Its cool stuff and I’m sure the current situation will be fixed soon.