Installing RHEL 8 on Hetzner root servers

Hetzner is a very popular provider for so called root servers and VPS (Virtual private Servers) located in Germany with datacenters in Germany and Finnland. They are quite affordable and have good services as well.

The default installation image, sorry Hetzner, is crap (i.e. no logical volumes). The rescue system is not only a nice tool to recover from botched system configurations, it also comes with an image installer which allows users to install a custom system. The default selection is Arch, CentOS 7.6, some Debian and Ubuntu versions but no CentOS8 or even RHEL of any version.

Game over? No, you can create an own image, just put it on a web server and configure the installation accordingly.

Create your image

I just installed a VM from the ISO image that can be downloaded here.

Select the “Minimal” installation, register the machine at Red Hat with subscription-manager register, attach the RHEL OS with subscription-manager attach –auto and install tar and mdadm.

Caution! You need to put the image on a public available web space! Unregister your system and delete the SSH host keys!

subscription-manager unregister
subscription-manager clean
rm -rf /etc/ssh/ssh_host_*

An other important fact is that the Hetzner installer only knows about CentOS and only about version 7. On CentOS8 and RHEL8, the dracut binary is located elsewhere than on RHEL7, you need to create a Symlink.

[root@localhost /]# ln -s /usr/bin/dracut /sbin/dracut

For the same reason you need to name the tarball CentOS-<something>

The next step is to create the tarball. Important to know is that the tarball must not contain /dev, /proc and /sys folders.

[root@localhost /]# tar cJvf CentOS-80-el-x86_64-minimal.tar.xz --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/CentOS-80-el-x86_64-minimal.tar.xz /

Copy the resulting tarball to a public web space:

[root@localhost /]# scp CentOS-80-el-x86_64-minimal.tar.xz blog.delouw.ch:/var/www/html/blog.delouw.ch/pub

Yes, you can use my image if you like: http://blog.delouw.ch/pub/CentOS-80-el-x86_64-minimal.tar.xz

Reboot your server into the rescue mode

Point your browser to https://robot.your-server.de/server, select your server, click on “Rescue” on the server menue, select “64 bit” and click on “Activate Rescue System”. The root password will be shown which will be needed to log in to the system.

Configure the installer

Configuring the installer is straight forward. Create a file called config.txt with the following content:

DRIVE1 /dev/sda
DRIVE2 /dev/sdb
SWRAID 1
SWRAIDLEVEL 1
BOOTLOADER grub
HOSTNAME localhost.localdomain
PART /boot ext2     1024M
PART lvm   rhel       all

LV rhel   root   /       xfs     20G
LV rhel   swap   swap    swap     16G
LV rhel   tmp    /tmp    xfs      1G
LV rhel   home   /home   xfs      4G
LV rhel   var   /var   xfs      4G
LV rhel   var_log   /var/log   xfs      4G
LV rhel   var_log_audit   /var/log/audit   xfs      4G

IMAGE http://blog.delouw.ch/pub/CentOS-80-el-x86_64-minimal.tar.xz

The partitioning is according to PCI-DSS. Note that the installer can not set the mount options needed by PCI-DSS (and a lot of other regularities).

Side note: The Hetzner installer refuses to install the image if /boot is of type xfs. You need to choose ext2,3 or ext4. I’m using ext2 as this is the most efficient file system, fsck on unclean shutdown for 1GByte takes just a few seconds, it does not matter.

Run the installer

Just run it and expect an error….

installimage -a -c config.txt

Hetzner Installer

The installer tries to install some packages which will, of course, fail as the system is not registered to RHSM yet.

[14:26:11] # Running some centos specific functions
[14:26:11] # chroot: chkconfig iptables off
[14:26:11] :   error reading information on service iptables: No such file or directory
[14:26:11] # chroot: chkconfig ip6tables off
[14:26:11] :   error reading information on service ip6tables: No such file or directory
[14:26:11] # chroot: chkconfig postfix off
[14:26:11] :   error reading information on service postfix: No such file or directory
[14:26:11] # Testing and setup of cpanel image
[14:26:11] # chroot: yum check-update
[14:26:12] :   Updating Subscription Management repositories.
[14:26:12] :   Unable to read consumer identity
[14:26:12] :   This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
[14:26:12] :   Error: There are no enabled repos.
[14:26:12] # chroot: yum -y install polkit
[14:26:12] :   Updating Subscription Management repositories.
[14:26:12] :   Unable to read consumer identity
[14:26:12] :   This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
[14:26:13] :   Error: There are no enabled repos.
[14:26:13] => FAILED
[14:26:13] :   report install.conf to rz-admin: 1281870
[14:26:13] :   report debug.txt to rz-admin: ok
[14:26:13] cleaning up

You can just ignore that error messages, it does not matter.

Finalizing

Reboot the system and log in with the root password provided for the rescue system when activating the rescue system at https://robot.your-server.de/server.

Change the password (needless to say?)

Continue to configure the system according your needs.

Have fun 🙂

Renew Letsencrypt certificates for Red Hat Satellite 6 and Capsule

Letsencrypt certificates are only valid for just three months. The procedure to renew x509 certificates in Red Hat Satellite 6 is not so straight forward and its even more complex for Capsule servers.

In an earlier post I was writing about how to set up a Satellite 6 and a capsule using Letsencrypt certificates. This post is a follow up on that.

Be aware: You must follow this procedure before the certificate expires or the Satellite simply stops working, all clients will refuse to communicate with the Red Hat Satellite Server. This is a security feature, not a bug.

Obtain the Satellite certificate

This is straight forward, stop the httpd and just use certbot, you must use the -d parameter because for the capsule it will fail.

systemctl stop httpd && certbot renew -d sat6.example.com

Install the renewed certificate into Satellite

You need to run the Satellite installer to make your certificate active:

satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" --certs-server-key "/etc/letsencrypt/live/sat6.example.com/privkey.pem" --certs-server-ca-cert "/root/ca-cert.pem" --certs-update-server --certs-update-server-ca

Keep the output as you need the oauth key and secret for the capsule.

Obtain the new certifcate for the capsule

This step must be done on the Satellite, not on the Caspule.

The only way to obtain a cert for a server different than than the target is to make use of the DNS challenge.

certbot -d capsule.example.com --manual --preferred-challenges dns certonly

It will ask you to create a DNS TXT entry as a challenge to ensure you are in control of the domain. When your DNS entry is ready, hit enter.

Create the tarball with the certifcates for the Capsule

capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar  "~/$CAPSULE-certs.tar" --server-cert "/etc/letsencrypt/live/capsule.example.com/cert.pem" --server-key "/etc/letsencrypt/live/capsule.example.com/privkey.pem" --server-ca-cert "/root/capsule.example.com/bundle-ca-cert.pem" --certs-update-server

The next step is to copy the tarball to your capsule:

scp /root/capsule.example.com-certs.tar capsule.example.com:

Install the new certificate on the Capsule

This step must be done on the Capsule server

satellite-installer --scenario capsule\
                      --foreman-proxy-content-parent-fqdn           "sat6.example.com"\
                      --foreman-proxy-register-in-foreman           "true"\
                      --foreman-proxy-foreman-base-url              "https://sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "sat6.example.com"\
                      --foreman-proxy-trusted-hosts                 "capsule.exmple.com"\
                      --foreman-proxy-oauth-consumer-key            "The Key"\
                      --foreman-proxy-oauth-consumer-secret         "The Secret"\
                      --foreman-proxy-content-certs-tar             "/root/capsule.example.com-certs.tar"\
                      --puppet-server-foreman-url                   "https://sat6.example.com"

Feedback welcome…

Have fun 🙂

OpenID and SAML authentication with Keycloak and FreeIPA

Not every web application can handle Kerberos SSO, but some provide OpenID and/or SAML. There is how Keycloak comes into the game. You can use Keycloak to federate users from different sources. This guide shows how to integrate Keyclock and FreeIPA to authenticate users in WordPress. On clients that are enrolled in IPA, this even works without a password, a Kerberos ticket is good enough to log in.

What is Keycloak

Keycloak is the upstream project for Red Hat SSO. It is a JBoss application that can federate users from various LDAP servers such as 389-Server, OpenLDAP and also MS Active Directory. It provides Single Sign On (SSO) for web application capabilities with OpenID and SAML2.

A very nice feature is the capability of using Kerberos tickets from clients that makes password based authentication obsolete.

Requirements

I’ll describe how to set up the commercially supported products provided by Red Hat, namely RHEL8 and Red Hat SSO. It is expected to work as well with the upstream projects, but please be aware that upstream products never provide formal commercial support.

  • A base installation of RHEL8
  • A subscription for RHEL8 and JBoss EAP
  • A configured and working FreeIPA/Red Hat IdM environment (optional)
  • An instance of WordPress or any other OpenID enabled Webapplication (optional)

The system requirements for a very basic setup are rather small. 2 Gbyte of RAM and 50 Gbyte of disk is more than enough.

Be aware that Red Hat SSO comes with a basic Database called H2. That is not suited for larger production environment. For production environments, user PostgreSQL instead. For better scalability and availability you also should consider to create a cluster of SSO instances using the same shared Database. External Database and Clustering is out of scope in this document, it may be covered in a later article.

This setup is also using a Letsencrypt x509 certificate and makes use of an Apache HTTP based reverse Proxy for better handling of certificates and access control.

Installation

Ensure you have the following yum repositories available:

  • JBoss Enterprise Application Platform 7.2 RHEL 8 RPMs x86_64
  • Red Hat CodeReady Linux Builder for RHEL 8 x86_64 RPMs x86_64 8
  • Single Sign-On 7.3 for RHEL 8 x86_64 RPMs x86_64
subscription-manager repos --enable=jb-eap-7.2-for-rhel-8-x86_64-rpms --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms --enable=codeready-builder-for-rhel-8-x86_64-rpms

The next step is to install the yum packages needed

yum install rh-sso* httpd mod_ssl socat

Install the acme shell script for Letsencrypt certificate handling:

curl https://get.acme.sh | sh

Enable firewall

It is recommended to make use of an host based firewall, its simple:

# HTTP is used for letsencrypt only
firewall-cmd --add-service=http --permanent

# Needed for the reverse proxy
firewall-cmd --add-service=https --permanent
firewall-cmd --reload

Reverse Proxy configuration

Apply the following patch to make Red Hat SSO aware of the proxy usage:

--- /etc/opt/rh/rh-sso7/keycloak/standalone/standalone.xml.orig 2019-04-02 03:31:07.480115492 +0000
+++ /etc/opt/rh/rh-sso7/keycloak/standalone/standalone.xml      2019-04-02 03:32:45.946964803 +0000
@@ -464,7 +464,8 @@
         <subsystem xmlns="urn:jboss:domain:undertow:7.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other">
             <buffer-cache name="default"/>
             <server name="default-server">
-                <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
+                <!-- <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/> -->
+               <http-listener name="default" socket-binding="http" proxy-address-forwarding="true" redirect-socket="proxy-https" />
                 <https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
                 <host name="default-host" alias="localhost">
                     <location name="/" handler="welcome-content"/>
@@ -575,6 +576,8 @@
         <socket-binding name="https" port="${jboss.https.port:8443}"/>
         <socket-binding name="txn-recovery-environment" port="4712"/>
         <socket-binding name="txn-status-manager" port="4713"/>
+       <!-- added for reverse proxy -->
+       <socket-binding name="proxy-https" port="443"/>
         <outbound-socket-binding name="mail-smtp">
             <remote-destination host="localhost" port="25"/>
         </outbound-socket-binding>

Enable and start the Apache HTTPd

systemctl enable httpd
systemctl start httpd

Obtain a certificate

acme.sh --issue -d sso.example.com -w /var/www/html

Install the certificate

/root/.acme.sh/acme.sh --install-cert -d sso.example.com \
--cert-file      /etc/pki/tls/certs/sso.example.cert  \
--key-file       /etc/pki/tls/private/sso.example.com.key  \
--fullchain-file /etc/pki/tls/certs/fullchain.pem

Configure Apache

Edit /etc/httpd/conf.d/ssl.conf and change the certifcate configuration to point to the Letsencrypt certificates:

SSLCertificateFile /etc/pki/tls/certs/fullchain.pem
SSLCertificateKeyFile /etc/pki/tls/private/sso.example.com.key

Reverse Proxy config

ProxyPreserveHost On
SSLProxyEngine On
SSLProxyCheckPeerCN on
SSLProxyCheckPeerExpire on
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/

Ensure Apache is allowed to do network connections:

setsebool httpd_can_network_connect on -P

Restart Apache HTTPd

systemctl restart httpd

Final steps for Red Hat SSO

Enable and start Red Hat SSO

systemctl enable rh-sso7.service
systemctl start rh-sso7.service

To be able to login in into SSO, you need to create a local user.

/opt/rh/rh-sso7/root/usr/share/keycloak/bin/add-user-keycloak.sh -u admin

You are now able to log in to Red Hat SSO with your favorite browser.

Integration with Red Hat IdM

Ensure your SSO server is enrolled in the IPA domain. There is some preparation work to do such as creating a Kerbros Service Principal for the HTTP server and fetch the Kerberos Keytab.

Create the Kerbros Service Pricipal

ipa service-add HTTP/sso.example.com

Fetch the Keytab

ipa-getkeytab -p HTTP/sso.example.com -s ipa1.example.com -k /etc/krb5-keycloak.keytab

Set correct permissions for the Keytab

chown root /etc/krb5-keycloak.keytab
chgrp jboss /etc/krb5-keycloak.keytab
chmod 640 /etc/krb5-keycloak.keytab

User federation

User federation with IPA is the second important step. It is slightly different to the nomal LDAP federation.

Point your bowser to https://sso.example.com/auth/admin/master/console/#/realms/master/user-federation and click on “Add provider” and select LDAP. Fill out the form as follow:

“Edit Mode” READ_ONLY
“Vendor” Red Hat Directory Server
“Username LDAP Attribute” uid
“RDN LDAP attribute” uid
“UUID LDAP attribute” ipaUniqueID
“User Object Class” inetOrgPerson, organizationalPerson
“Connection URL” ldaps://ipa1.example.com
“Users DN” cn=users,cn=accounts,dc=example,dc=com
“Authentication Type” simple
“Bind DN” uid=binduser,cn=sysaccounts,cn=etc,dc=example,dc=com
“Bind Credential” your super secret password

“Allow Kerberos authentication” to On
“Kerberos Realm” EXAMPLE.COMA
“Server Principal” HTTP/sso.example.com
“Keytab” /etc/krb5-keycloak.keytab
“Use Kerberos For Password Authentication” On

Or have a look at the screenshot

SSO-IdM Federation

The next step is more or less cosmetic, the mapping of attributes. Go to the newly created federation provider and click in th “Mappers” tab, click on “First Name” and change “LDAP Attibute” to “givenName”.

Thats it.

Registering a client

Point your browser to https://sso.example.com/auth/admin/master/console/#/create/client/master

Choose a client ID, i.e. “wordpress” and provide the Root URL, i.e. https://www.example.com.

Creating a initial access token

Point your browser to https://sso.example.com/auth/admin/master/console/#/realms/master/client-registration/client-initial-access/create and click on save.

You will get the token displayed. Be aware that this token shows only once, copy and paste it to a secure place.

Enable WordPress for OpenID and connect it to Red Hat SSO

Point your brower to https://www.example.com/wp-admin/plugin-install.php?s=OpenID+Connect+Generic&tab=search&type=term to search for the Plugin “OpenID Connect Generic” and click on “Install Now”.

OpenID Setup

Point your browser to https://www.example.com/wp-admin/options-general.php?page=openid-connect-generic-settings.

Fill in the form as shown in the below screenshot. The “Client ID” and “Client Secret Key” corresponds to the previously defined ID and “initial Access Token” defined in Red Hat SSO before.

SSO in WordPress

Click on “save”, log out, log in again and client on the “Login with OpenID Connect”. You will get redirected to the Red Hat SSO login form, or in case you have a Kerbros Ticket, your are automatically logged in to WordPress.

Be aware that every user in Red Hat IdM will be able to login to WordPress in the role “Subscriber”. You need to promote them to another role manually.

This Guide is only about authentication, not about authorization. This will be covered in a separate article somewhere in the future.

Feedback is always welcome. Have fun 🙂

Installing Red Hat Satellite 6 with Letsencrypt certificates

Red Hat Satellite 6 is a nice tool for system life cycle management. It can get complex and even installation is sometimes tricky. This article is about how to install Satellite, it does not explain the principals and concepts behind it.

Requirements

A valid subscription for the Satellite (and optional for the capsule).

The system requirements are listed here.

There is one important thing the install guide is missing: Satellite 6.4 will not work in IPv6 only environments. There must be an IPv4 address configured, even if it is just an RFC1918 private address. You need to add this IP in /etc/hosts. The reason is that several daemons are listening on IPv4 addresses only. One of them is important: Apache QPID which is used for i.e. errata application via katello-agent. But there are two [update] undocumented installer parameters [/update].

Proxy is mandatory in IPv6 only environments

If your Satellite is only able to connect to the internet via IPv6, you need a IPv4 capable proxy to talk to subscription.rhsm.redhat.com which is not reachable by IPv6. That is the host the subscription manager is talking to.

Install EPEL and certbot

It makes sense to use officially valid certs since they are available for free usage from Letsencrypt. Certbot is available from EPEL and a handy way to request certificates using the ACME protocol.

[root@sat6 ~]# yum -y install http://ftp.tu-chemnitz.de/pub/linux/epel/7Server/x86_64/Packages/e/epel-release-7-11.noarch.rpm

It is important to disable EPEL by default to not get conflicts with RPMs from other repositories. Just enable EPEL when needed and double check.

[root@sat6 ~]# yum-config-manager --disable epel

Install the certbot package

[root@sat6 ~]# yum -y install certbot --enablerepo=epel

Issue the cert

[root@sat6 ~]# certbot certonly -n --standalone --agree-tos --domains sat6.example.com -m user@example.com

Download the CA-Certs

Root-CA

[root@sat6 ~]# wget https://www.identrust.com/node/935 -O trustidrootx3_chain.p7b

Convert the p7b to PEM format.

[root@sat6 ~]# openssl pkcs7 -in trustidrootx3_chain.p7b -inform DER -print_certs -out trustidrootx3_chain.pem

Intermediate CA-Cert

[root@sat6 ~]# wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

Create the CA bundle file

[root@sat6 ~]# cp trustidrootx3_chain.pem bundle-ca-cert.pem
[root@sat6 ~]# cat lets-encrypt-x3-cross-signed.pem.txt >> bundle-ca-cert.pem

Check the certificate

You can check if you made all things as expected, run the check command.

[root@sat6 ~]# katello-certs-check -c "/etc/letsencrypt/live/sat6.example.com/fullchain.pem" -k "/etc/letsencrypt/live/sat6.example.com/privkey.pem" -b "/root/bundle-ca-cert.pem"

Running the installer

Note: The parameters –foreman-proxy-content-qpid-router-hub-addr :: –foreman-proxy-content-qpid-router-agent-addr :: are not documented and only needed if you want to be able that the capsule and/or clients will be able to communicate over IPv6.

[root@sat6 ~]# satellite-installer --scenario satellite --certs-server-cert "/etc/letsencrypt/archive/sat6.example.com/fullchain2.pem" --certs-server-key "/etc/letsencrypt/archive/sat6.example.com/privkey2.pem" --certs-server-ca-cert "/root/bundle-ca-cert.pem" --certs-update-server --certs-update-server-ca --katello-proxy-url=http://proxy.example.com --katello-proxy-port=3128 --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

Installing a Capsule Server

If you want to use a capsule server within an environment with Letsencrypt certificates, its a bit more complex, but however, it works.

Install the software

Needless to say that your capsule needs to have the correct repos enabled. For details, please see here.

[root@capsule ~]# yum install satellite-capsule

Request the Certificate

On the Satellite, request a certificate for the capsule. Note: This only works with the DNS challenge, so you need access to your DNS server.

[root@sat6 ~]# certbot -d capsule.example.com --manual --preferred-challenges dns certonly

Prepare the certificates and key

Create a directory and copy the certificates

[root@sat6 ~]# mkdir /root/capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/privkey.pem capsule.example.com
[root@sat6 ~]# cp /etc/letsencrypt/live/capsule.example.com/cert.pem capsule.example.com
[root@sat6 ~]# cp /root/bundle-ca-cert.pem capsule.example.com

Validate the certificate

[root@sat6 ~]# katello-certs-check -c /root/capsule.example.com/cert.pem -b /root/capsule.example.com/bundle-ca-cert.pem -k /root/capsule.example.com/privkey.pem

If all is fine, run the capsule generator:

[root@sat6 ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.example.com --certs-tar /root/capsule.example.com-certs.tar --server-cert /root/capsule.example.com/cert.pem --server-key /root/capsule.example.com/privkey.pem --server-ca-cert /root/capsule.example.com/bundle-ca-cert.pem

Copy the resulting tarball to the capsule server:

[root@sat6 ~]# scp capsule.example.com-certs.tar capsule.example.com

Running the installer

[root@capsule ~]#  satellite-installer --scenario capsule\
--foreman-proxy-content-parent-fqdn      "sat6.example.com"\
--foreman-proxy-register-in-foreman      "true"\
--foreman-proxy-foreman-base-url         "https://sat6.example.com"\
--foreman-proxy-trusted-hosts            "sat6.example.com"\
--foreman-proxy-trusted-hosts            "capsule.example.com"\
--foreman-proxy-oauth-consumer-key       "the key"\
--foreman-proxy-oauth-consumer-secret    "the secret"\
--foreman-proxy-content-certs-tar        "/root/capsule.example.com-certs.tar"\
--puppet-server-foreman-url              "https://sat6.example.com" \
--foreman-proxy-content-qpid-router-hub-addr :: \
--foreman-proxy-content-qpid-router-agent-addr ::

That’s it 🙂

Do not ask me how certificate renewal works, I’ll let you know in three months 😉

Workaround for IPv6-only Networks

Unfortunately the satellite-installer configures Apache QPID not correctly, it will be set up to use IPv4 only by default. That means, IPv6-only hosts (i.e. including the capsule) are unable to communicate with the Satellite.

There is a workaround: Add two additional listeners on the Satellite and one on the Capsule. Be aware: The Satellite installer overwrites your changes every time when you run it, i.e. for upgrades or adding new features. create a backup of the config file.

There are two undocumented parameters for the satellite installer: –foreman-proxy-content-qpid-router-hub-addr :: and –foreman-proxy-content-qpid-router-agent-addr ::. You can add them during first time run as well as after an initial installation.

It is the same procedure in the Satellite as well as on the Capsule.

satellite-installer --foreman-proxy-content-qpid-router-hub-addr :: --foreman-proxy-content-qpid-router-agent-addr ::

This behavior is already fixes upstream as you can see it here: https://github.com/theforeman/puppet-foreman_proxy_content/commit/89b4ea988d18f100b806e7cddc2dca623b68f084″.