Renew Letsencrypt certificates for Red Hat Satellite 6 and Capsule

Letsencrypt certificates are only valid for just three months. The procedure to renew x509 certificates in Red Hat Satellite 6 is not so straight forward and its even more complex for Capsule servers.

In an earlier post I was writing about how to set up a Satellite 6 and a capsule using Letsencrypt certificates. This post is a follow up on that.

Be aware: You must follow this procedure before the certificate expires or the Satellite simply stops working, all clients will refuse to communicate with the Red Hat Satellite Server. This is a security feature, not a bug.

Obtain the Satellite certificate

This is straight forward, stop the httpd and just use certbot, you must use the -d parameter because for the capsule it will fail.

systemctl stop httpd && certbot renew -d sat6.example.com

Install the renewed certificate into Satellite

You need to run the Satellite installer to make your certificate active:

satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" --certs-server-key "/etc/letsencrypt/live/sat6.example.com/privkey.pem" --certs-server-ca-cert "/root/ca-cert.pem" --certs-update-server --certs-update-server-ca

Keep the output as you need the oauth key and secret for the capsule.

Obtain the new certifcate for the capsule

This step must be done on the Satellite, not on the Caspule.

The only way to obtain a cert for a server different than than the target is to make use of the DNS challenge.

certbot -d capsule.example.com --manual --preferred-challenges dns certonly

It will ask you to create a DNS TXT entry as a challenge to ensure you are in control of the domain. When your DNS entry is ready, hit enter.

Create the tarball with the certifcates for the Capsule

capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar  "~/$CAPSULE-certs.tar" --server-cert "/etc/letsencrypt/live/capsule.example.com/cert.pem" --server-key "/etc/letsencrypt/live/capsule.example.com/privkey.pem" --server-ca-cert "/root/capsule.example.com/bundle-ca-cert.pem" --certs-update-server

The next step is to copy the tarball to your capsule:

scp /root/capsule.example.com-certs.tar capsule.example.com:

Install the new certificate on the Capsule

This step must be done on the Capsule server

satellite-installer --scenario capsule\
                      --foreman-proxy-content-parent-fqdn           "sat6.example.com"\
                      --foreman-proxy-register-in-foreman           "true"\
                      --foreman-proxy-foreman-base-url              "https://sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "capsule.exmple.com"\
                      --foreman-proxy-oauth-consumer-key            "The Key"\
                      --foreman-proxy-oauth-consumer-secret         "The Secret"\
                      --foreman-proxy-content-certs-tar             "/root/capsule.example.com-certs.tar"\
                      --puppet-server-foreman-url                   "https://sat6.example.com"

Feedback welcome…

Have fun 🙂

OpenID and SAML authentication with Keycloak and FreeIPA

Not every web application can handle Kerberos SSO, but some provide OpenID and/or SAML. There is how Keycloak comes into the game. You can use Keycloak to federate users from different sources. This guide shows how to integrate Keyclock and FreeIPA to authenticate users in WordPress. On clients that are enrolled in IPA, this even works without a password, a Kerberos ticket is good enough to log in.

What is Keycloak

Keycloak is the upstream project for Red Hat SSO. It is a JBoss application that can federate users from various LDAP servers such as 389-Server, OpenLDAP and also MS Active Directory. It provides Single Sign On (SSO) for web application capabilities with OpenID and SAML2.

A very nice feature is the capability of using Kerberos tickets from clients that makes password based authentication obsolete.

Requirements

I’ll describe how to set up the commercially supported products provided by Red Hat, namely RHEL8 and Red Hat SSO. It is expected to work as well with the upstream projects, but please be aware that upstream products never provide formal commercial support.

  • A base installation of RHEL8
  • A subscription for RHEL8 and JBoss EAP
  • A configured and working FreeIPA/Red Hat IdM environment (optional)
  • An instance of WordPress or any other OpenID enabled Webapplication (optional)

The system requirements for a very basic setup are rather small. 2 Gbyte of RAM and 50 Gbyte of disk is more than enough.

Be aware that Red Hat SSO comes with a basic Database called H2. That is not suited for larger production environment. For production environments, user PostgreSQL instead. For better scalability and availability you also should consider to create a cluster of SSO instances using the same shared Database. External Database and Clustering is out of scope in this document, it may be covered in a later article.

This setup is also using a Letsencrypt x509 certificate and makes use of an Apache HTTP based reverse Proxy for better handling of certificates and access control.

Installation

Ensure you have the following yum repositories available:

  • JBoss Enterprise Application Platform 7.2 RHEL 8 RPMs x86_64
  • Red Hat CodeReady Linux Builder for RHEL 8 x86_64 RPMs x86_64 8
  • Single Sign-On 7.3 for RHEL 8 x86_64 RPMs x86_64
subscription-manager repos --enable=jb-eap-7.2-for-rhel-8-x86_64-rpms --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms --enable=codeready-builder-for-rhel-8-x86_64-rpms

The next step is to install the yum packages needed

yum install rh-sso* httpd mod_ssl socat

Install the acme shell script for Letsencrypt certificate handling:

curl https://get.acme.sh | sh

Enable firewall

It is recommended to make use of an host based firewall, its simple:

# HTTP is used for letsencrypt only
firewall-cmd --add-service=http --permanent

# Needed for the reverse proxy
firewall-cmd --add-service=https --permanent
firewall-cmd --reload

Reverse Proxy configuration

Apply the following patch to make Red Hat SSO aware of the proxy usage:

--- /etc/opt/rh/rh-sso7/keycloak/standalone/standalone.xml.orig 2019-04-02 03:31:07.480115492 +0000
+++ /etc/opt/rh/rh-sso7/keycloak/standalone/standalone.xml      2019-04-02 03:32:45.946964803 +0000
@@ -464,7 +464,8 @@
         <subsystem xmlns="urn:jboss:domain:undertow:7.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other">
             <buffer-cache name="default"/>
             <server name="default-server">
-                <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
+                <!-- <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/> -->
+               <http-listener name="default" socket-binding="http" proxy-address-forwarding="true" redirect-socket="proxy-https" />
                 <https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
                 <host name="default-host" alias="localhost">
                     <location name="/" handler="welcome-content"/>
@@ -575,6 +576,8 @@
         <socket-binding name="https" port="${jboss.https.port:8443}"/>
         <socket-binding name="txn-recovery-environment" port="4712"/>
         <socket-binding name="txn-status-manager" port="4713"/>
+       <!-- added for reverse proxy -->
+       <socket-binding name="proxy-https" port="443"/>
         <outbound-socket-binding name="mail-smtp">
             <remote-destination host="localhost" port="25"/>
         </outbound-socket-binding>

Enable and start the Apache HTTPd

systemctl enable httpd
systemctl start httpd

Obtain a certificate

acme.sh --issue -d sso.example.com -w /var/www/html

Install the certificate

/root/.acme.sh/acme.sh --install-cert -d sso.example.com \
--cert-file      /etc/pki/tls/certs/sso.example.cert  \
--key-file       /etc/pki/tls/private/sso.example.com.key  \
--fullchain-file /etc/pki/tls/certs/fullchain.pem

Configure Apache

Edit /etc/httpd/conf.d/ssl.conf and change the certifcate configuration to point to the Letsencrypt certificates:

SSLCertificateFile /etc/pki/tls/certs/fullchain.pem
SSLCertificateKeyFile /etc/pki/tls/private/sso.example.com.key

Reverse Proxy config

ProxyPreserveHost On
SSLProxyEngine On
SSLProxyCheckPeerCN on
SSLProxyCheckPeerExpire on
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/

Ensure Apache is allowed to do network connections:

setsebool httpd_can_network_connect on -P

Restart Apache HTTPd

systemctl restart httpd

Final steps for Red Hat SSO

Enable and start Red Hat SSO

systemctl enable rh-sso7.service
systemctl start rh-sso7.service

To be able to login in into SSO, you need to create a local user.

/opt/rh/rh-sso7/root/usr/share/keycloak/bin/add-user-keycloak.sh -u admin

You are now able to log in to Red Hat SSO with your favorite browser.

Integration with Red Hat IdM

Ensure your SSO server is enrolled in the IPA domain. There is some preparation work to do such as creating a Kerbros Service Principal for the HTTP server and fetch the Kerberos Keytab.

Create the Kerbros Service Pricipal

ipa service-add HTTP/sso.example.com

Fetch the Keytab

ipa-getkeytab -p HTTP/sso.example.com -s ipa1.example.com -k /etc/krb5-keycloak.keytab

Set correct permissions for the Keytab

chown root /etc/krb5-keycloak.keytab
chgrp jboss /etc/krb5-keycloak.keytab
chmod 640 /etc/krb5-keycloak.keytab

User federation

User federation with IPA is the second important step. It is slightly different to the nomal LDAP federation.

Point your bowser to https://sso.example.com/auth/admin/master/console/#/realms/master/user-federation and click on “Add provider” and select LDAP. Fill out the form as follow:

“Edit Mode” READ_ONLY
“Vendor” Red Hat Directory Server
“Username LDAP Attribute” uid
“RDN LDAP attribute” uid
“UUID LDAP attribute” ipaUniqueID
“User Object Class” inetOrgPerson, organizationalPerson
“Connection URL” ldaps://ipa1.example.com
“Users DN” cn=users,cn=accounts,dc=example,dc=com
“Authentication Type” simple
“Bind DN” uid=binduser,cn=sysaccounts,cn=etc,dc=example,dc=com
“Bind Credential” your super secret password

“Allow Kerberos authentication” to On
“Kerberos Realm” EXAMPLE.COMA
“Server Principal” HTTP/sso.example.com
“Keytab” /etc/krb5-keycloak.keytab
“Use Kerberos For Password Authentication” On

Or have a look at the screenshot

SSO-IdM Federation

The next step is more or less cosmetic, the mapping of attributes. Go to the newly created federation provider and click in th “Mappers” tab, click on “First Name” and change “LDAP Attibute” to “givenName”.

Thats it.

Registering a client

Point your browser to https://sso.example.com/auth/admin/master/console/#/create/client/master

Choose a client ID, i.e. “wordpress” and provide the Root URL, i.e. https://www.example.com.

Creating a initial access token

Point your browser to https://sso.example.com/auth/admin/master/console/#/realms/master/client-registration/client-initial-access/create and click on save.

You will get the token displayed. Be aware that this token shows only once, copy and paste it to a secure place.

Enable WordPress for OpenID and connect it to Red Hat SSO

Point your brower to https://www.example.com/wp-admin/plugin-install.php?s=OpenID+Connect+Generic&tab=search&type=term to search for the Plugin “OpenID Connect Generic” and click on “Install Now”.

OpenID Setup

Point your browser to https://www.example.com/wp-admin/options-general.php?page=openid-connect-generic-settings.

Fill in the form as shown in the below screenshot. The “Client ID” and “Client Secret Key” corresponds to the previously defined ID and “initial Access Token” defined in Red Hat SSO before.

SSO in WordPress

Click on “save”, log out, log in again and client on the “Login with OpenID Connect”. You will get redirected to the Red Hat SSO login form, or in case you have a Kerbros Ticket, your are automatically logged in to WordPress.

Be aware that every user in Red Hat IdM will be able to login to WordPress in the role “Subscriber”. You need to promote them to another role manually.

This Guide is only about authentication, not about authorization. This will be covered in a separate article somewhere in the future.

Feedback is always welcome. Have fun 🙂

Installing Red Hat Satellite 6 with Letsencrypt certificates

Red Hat Satellite 6 is a nice tool for system life cycle management. It can get complex and even installation is sometimes tricky. This article is about how to install Satellite, it does not explain the principals and concepts behind it.

Requirements

A valid subscription for the Satellite (and optional for the capsule).

The system requirements are listed here.

There is one important thing the install guide is missing: Satellite 6.4 will not work in IPv6 only environments. There must be an IPv4 address configured, even if it is just an RFC1918 private address. You need to add this IP in /etc/hosts. The reason is that several daemons are listening on IPv4 addresses only. One of them is important: Apache QPID which is used for i.e. errata application via katello-agent. But there are two [update] undocumented installer parameters [/update].

Proxy is mandatory in IPv6 only environments

If your Satellite is only able to connect to the internet via IPv6, you need a IPv4 capable proxy to talk to subscription.rhsm.redhat.com which is not reachable by IPv6. That is the host the subscription manager is talking to.

Install EPEL and certbot

It makes sense to use officially valid certs since they are available for free usage from Letsencrypt. Certbot is available from EPEL and a handy way to request certificates using the ACME protocol.

[root@sat6 ~]# yum -y install http://ftp.tu-chemnitz.de/pub/linux/epel/7Server/x86_64/Packages/e/epel-release-7-11.noarch.rpm

It is important to disable EPEL by default to not get conflicts with RPMs from other repositories. Just enable EPEL when needed and double check.

[root@sat6 ~]# yum-config-manager --disable epel

Install the certbot package

[root@sat6 ~]# yum -y install certbot --enablerepo=epel

Issue the cert

[root@sat6 ~]# certbot certonly -n --standalone --agree-tos --domains sat6.example.com -m user@example.com

Download the CA-Certs

Root-CA

[root@sat6 ~]# wget https://www.identrust.com/node/935 -O trustidrootx3_chain.p7b

Convert the p7b to PEM format.

[root@sat6 ~]# openssl pkcs7 -in trustidrootx3_chain.p7b -inform DER -print_certs -out trustidrootx3_chain.pem

Intermediate CA-Cert

[root@sat6 ~]# wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

Create the CA bundle file

[root@sat6 ~]# cp trustidrootx3_chain.pem bundle-ca-cert.pem
[root@sat6 ~]# cat lets-encrypt-x3-cross-signed.pem.txt >> bundle-ca-cert.pem

Check the certificate

You can check if you made all things as expected, run the check command.

[root@sat6 ~]# katello-certs-check -c "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" -k "/etc/letsencrypt/live/sat6.example.com/privkey.pem" -b "/root/bundle-ca-cert.pem"

Running the installer

Note: The parameters –foreman-proxy-content-qpid-router-hub-addr :: –foreman-proxy-content-qpid-router-agent-addr :: are not documented and only needed if you want to be able that the capsule and/or clients will be able to communicate over IPv6.

[root@sat6 ~]# satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/archive/sat6.example.com/fullchain2.pem" --certs-server-key "/etc/letsencrypt/archive/sat6.example.com/privkey2.pem" --certs-server-ca-cert "/root/bundle-ca-cert.pem" --certs-update-server --certs-update-server-ca --katello-proxy-url=http://proxy.example.com --katello-proxy-port=3128 --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

Installing a Capsule Server

If you want to use a capsule server within an environment with Letsencrypt certificates, its a bit more complex, but however, it works.

Install the software

Needless to say that your capsule needs to have the correct repos enabled. For details, please see here.

[root@capsule ~]# yum install satellite-capsule

Request the Certificate

On the Satellite, request a certificate for the capsule. Note: This only works with the DNS challenge, so you need access to your DNS server.

[root@sat6 ~]# certbot -d capsule.example.com --manual --preferred-challenges dns certonly

Prepare the certificates and key

Create a directory and copy the certificates

[root@sat6 ~]# mkdir /root/capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/privkey.pem capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/cert.pem capsule.example.com
[root@sat6 ~]# cp /root/bundle-ca-cert.pem capsule.example.com

Validate the certificate

[root@sat6 ~]# katello-certs-check -c /root/capsule.example.com/cert.pem -b /root/capsule.example.com/bundle-ca-cert.pem -k /root/capsule.example.com/privkey.pem

If all is fine, run the capsule generator:

[root@sat6 ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar /root/capsule.example.com-certs.tar --server-cert /root/capsule.example.com/cert.pem --server-key /root/capsule.example.com/privkey.pem --server-ca-cert /root/capsule.example.com/bundle-ca-cert.pem

Copy the resulting tarball to the capsule server:

[root@sat6 ~]# scp capsule.example.com-certs.tar capsule.example.com

Running the installer

[root@capsule ~]#  satellite-installer --scenario capsule\
--foreman-proxy-content-parent-fqdn      "sat6.example.com"\
--foreman-proxy-register-in-foreman      "true"\
--foreman-proxy-foreman-base-url         "https://sat6.example.com"\
--foreman-proxy-trusted-hosts            "sat6.example.com"\
--foreman-proxy-trusted-hosts            "capsule.example.com"\
--foreman-proxy-oauth-consumer-key       "the key"\
--foreman-proxy-oauth-consumer-secret    "the secret"\
--foreman-proxy-content-certs-tar        "/root/capsule.example.com-certs.tar"\
--puppet-server-foreman-url              "https://sat6.example.com" \
--foreman-proxy-content-qpid-router-hub-addr :: \
--foreman-proxy-content-qpid-router-agent-addr ::

That’s it 🙂

Do not ask me how certificate renewal works, I’ll let you know in three months 😉

Workaround for IPv6-only Networks

Unfortunately the satellite-installer configures Apache QPID not correctly, it will be set up to use IPv4 only by default. That means, IPv6-only hosts (i.e. including the capsule) are unable to communicate with the Satellite.

There is a workaround: Add two additional listeners on the Satellite and one on the Capsule. Be aware: The Satellite installer overwrites your changes every time when you run it, i.e. for upgrades or adding new features. create a backup of the config file.

There are two undocumented parameters for the satellite installer: –foreman-proxy-content-qpid-router-hub-addr :: and –foreman-proxy-content-qpid-router-agent-addr ::. You can add them during first time run as well as after an initial installation.

It is the same procedure in the Satellite as well as on the Capsule.

satellite-installer --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

This behavior is already fixes upstream as you can see it here: https://github.com/theforeman/puppet-foreman_proxy_content/commit/89b4ea988d18f100b806e7cddc2dca623b68f084″.

Using Data Deduplication and Compression with VDO on RHEL 7 and 8

Storage deduplication technology has been on the market for quite some time now. Unfortunately all of the implementations have been vendor specific proprietary software. With VDO, there is now an open source Linux native solution available.

Red hat has introduced VDO (Virtual Data Optimizer) in RHEL 7.5, a storage deduplication technology bough with Permabit in 2017. Of course it has been open sourced since then.

In contrast to ZFS which provides the same functionality on the file system level, VDO is an inline data reduction which works on block device level, it is file system agnostic.

Use cases

There are basically two major use cases: VM Storage and Object Storage Backends.

VM Storage

The main use case is storage for virtual machines where a lot of data is redundant, i.e. the base operating system of the VMs. This allows to deduplicate the data on disk on a large scale, think about 100 VMs where the operating system takes about 5Gbyte each will be reduced to approx. 5 Gbyte instead of 500 Gbyte.

Typically VM storage can be over committed by factor 10.

Object- and Block storage backends

As a backend for CEPH and Glusterfs, it is recommended to not over commit more than factor 3. The reason for the lower over commitment is that the storage administrator usually does not know what kind of data will be stored on it.

Availability

VDO is available since RHEL 7.5, it is included in the base subscription. At the moment it is not available for Fedora (yet).

The source code is available on github:

At the moment the Kernel code is not yet in the upstream Mainline Kernel, it is ongoing work to get it into the Mainstream Kernel.

Typical setup

Physical disk -> VDO -> Volumegroup -> Logical volume -> file system.

Block device can be a physical disk (or a partition on it), multi path device, LUKS disk, or a software RAID device (md or LVM RAID).

Restrictions

You can not use LVM cache, LVM snapshots and thin provisioned logical volumes on top of VDO. Theoretically you can use LUKS on top of VDO, but it makes no sense because there is nothing to deduplicate. Needless to say that VDO on top of a VDO device does not make any sense as well. Be aware that you can not make use of partitioning or (LVM) Raid on top of VDO devices, all that things should be done in the underlying layer of VDO.

When using SAN, check if your storage box already does deduplication. In this case VDO is useless for you.

Installation

Its straight forward:

[root@vdotest ~]# yum -y install vdo kmod-kvdo

Create the VDO volume

In this test case, I attached a 110Gbyte disk, created a 100 GByte partition and will over commit it by factor 10.

Warning! As of writing this article, never use a whole physical disk, use a partition instead and leave some spare space in the disk to avoid data loss! (see further below)

[root@vdotest ~]# vdo create --name=vdo1 --device=/dev/vdb1 --vdoLogicalSize=1T

Creating volume group, logical volume and file system on top of the VDO volume

[root@vdotest ~]# pvcreate /dev/mapper/vdo1
[root@vdotest ~]# vgcreate vg_vdo /dev/mapper/vdo1
[root@vdotest ~]# lvcreate -n lv_vdo vg_vdo -L 900G
[root@vdotest ~]# mkfs.xfs -K /dev/vg_vdo/lv_vdo
[root@vdotest ~]# echo "/dev/mapper/vg_vdo-lv_vdo       /mnt    xfs     defaults,x-systemd.requires=vdo.service 0 0" >> /etc/fstab

Display the whole stack

[root@vdotest ~]# lsblk /dev/vdb
NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vdb                 252:16   0  110G  0 disk 
└─vdb1              252:17   0  100G  0 part 
  └─vdo1            253:7    0    1T  0 vdo  
    └─vg_vdo-lv_vdo 253:8    0  900G  0 lvm  /mnt
[root@vdotest ~]#

Populate the disk with data

The ideal test for VDO is to put some real-life VM-Images to the file system on top of it. In this case I scp’ed three IPA server and some instances to that file system. This kind of systems are all quite similar, the disk space saved is tremendous. The total size of the vm images is 105G

Lets have a look:

[root@vdotest ~]# df -h /mnt
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg_vdo-lv_vdo  900G  105G  800G  12% /mnt
[root@vdotest ~]# 
[root@vdotest ~]# ll -h /mnt
total 101G
-rw-------. 1 root root 21G Dec 17 09:41 ipa1.lab.delouw.ch.qcow2
-rw-------. 1 root root 21G Dec 17 09:47 ipa1.ldelouw.ch
-rw-------. 1 root root 21G Dec 17 09:54 ipa2.ldelouw.ch
-rw-r--r--. 1 root root 21G Dec 17 09:58 ipaclient-rhel6.home.delouw.ch
-rw-------. 1 root root 21G Dec 17 10:03 ipatest.delouw.ch.qcow2
[root@vdotest ~]#

Lets use the vdostats utility to display the actually used storage on disk:

[root@vdotest ~]# vdostats --si
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo1        107.4G     15.2G     92.2G  14%           89%
[root@vdotest ~]#

Performance Tuning

There are a lot of parameters that can be changed. Unfortunately the documentation available at the moment is rudimentary, thus its more a guesswork than facts.

  • Number of worker threads of different kind
  • Enable or Disable compression

On machines with a lot of CPUs using more threads than the defaults can dramatically boost performance. man 8 vdo gives a glimpse of the different parameters related to threads.

Compression is a quite expensive operation. On top of that, depending on the kind of data you are storing, it does not make much sense to use compression (Well, deduplication is kind of compression as well).

Pitfalls

Be aware! With every storage deduplication solution there comes a big pitfall: The logical volume on top of VDO shows free disk space while the actual disk space on the physical disk can be (almost) exhausted. You need to carefully monitor the actual disk usage.

The fill grade can rapidly change if the data to be stored contains a lot of non-deduplicatable and/or compressible data. A good example is virtual machine images containing a LUKS encrypted disk, In such a case, use LUKS on the storage, not on the VM level.

Even if you update one virtual machine, the delta to other machine images will grow and less physical space is available.

VDO comes with a few Nagios plugins which are very useful for alerting administrators in the cause the available physical disk is filling up. They are located in /usr/share/doc/vdo/examples/nagios

According to df -h, on my test system there is still 800 Gbyte available. What happens if I store my 700 Gbyte Satellite 6 image? The data is mostly RPMs which are already compressed quite well. Lets see….

After a transfer of approx 155 Gbyte, the physical disk got full and the file system is inaccessible. I was hitting the worst case that can happen: Complete and unrecoverable data loss.

The df command shows some 241 Gbyte free.

[root@vdotest ~]# df -h |grep mnt
/dev/mapper/vg_vdo-lv_vdo                900G  241G  660G  27% /mnt
[root@vdotest ~]#

The vdostat command tells a different story, like expected.

[root@vdotest ~]# vdostats --si
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo1        107.4G    107.4G      0.0B 100%           59%
[root@vdotest ~]# 

When attempting to access the data, there will be an I/O error.

[root@vdotest ~]# ll -h /mnt
ls: cannot access /mnt: Input/output error
[root@vdotest ~]# 

Thats bad. I mean really bad. The device is not accessible anymore.

xfs_repair does not work. Do not attempt to make use of the -L option! Your file system will be gone.

Recovering from a full physical disk

Lets resize the partition instead. First unmount the file system

[root@vdotest ~]# umount /mnt

Delete and recreate the partition using fdisk

[root@vdotest ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): d
Selected partition 1
Partition 1 is deleted

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
First sector (2048-41943039, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): 
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@vdotest ~]# partprobe
[root@vdotest ~]#
[root@vdotest ~]# vdo growPhysical -n vdo1

Run a file system check.

Now you are able to mount the file system again and your data is available again.

Documentation

Red Hat maintains a nice documentation about storage administration, VDO is covered by an own chapter. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/storage_administration_guide/#vdo

Conclusion

The technology is very interesting and will kick some ass. Storage deduplication will be more and more important, with VDO there is now a Linux native solution for that.

At the moment it is quite dangerous to use VDO in production. Filling up a physical disk without spare space is an unrecoverable error, a complete data loss. That means: Always create the VDO device on top of a partition that is not using the whole disk or another device that can grow in size to prevent data loss.

If you plan to use VDO in production make sure you have a proper monitoring in place that alerts quite ahead of time to be able to take corrective action.

Nevertheless: Its cool stuff and I’m sure the current situation will be fixed soon.