RHEV 3.1 – an overview about the new features

RHEV-M

RHEV-M

Recently Red Hat announced the public availability of RHEV 3.1.

Finally, no more Windows needed for the whole software stack 🙂

In 3.0, the new webadmin interface was already inncluded, as a tech preview and had its problems. Now with 3.1 its working great and looks neat. In contrary to 3.0, it is now listening on the standard ports 80 and 443. This will probably help users in organizations with strict proxy policies and setting.

So what else is new?

The supported number of virtual CPUs in a guest is now ridiculous 160, and RAM per guest is at ridiculous two Terabytes. But this are the least import updates.

Especially on the storage side, a lot of effort has been done and long missing features integrated.

From my point of view, the most important new feature is the possibility to have disks from more than one Storage Domain attached to a virtual machine. This would allow to install the Operating system to cheap SATA storage while data disks are super fast SSDs.

There is also support for live snapshots, but snapshots are (as on other platforms) kind of problematic because they are COW (Copy-On-Write). This can lead to I/O performance problems. Snapshots are a cool feature for i.e. taking a snapshot before updating software etc. Be sure you remove the snapshot afterwards if you want to keep a good I/O performance.

You now can use DirectLUN directly from the GUI without the usage of hooks. DirectLUN allows to attach FibreChannel and iSCSI LUNs directly to a Virtual Machine. This is great when you want to use shared filesystems such as GFS.

Another nice feature is Live Storage Migration which is a technical preview, means: Unsupported for the moment. It probably will be supported in a later version. Storage live migration is a nice feature when you need to free up some space on a storage domain and you can not shut down a VM. Be sure to power-cycle the VM in question as soon as your SLA allows it, to get rid of the Snapshot (COW here again).

If you want to script stuff or you are too lazy to open a brower, there is now a CLI available. Have a look to the documentation.

If you want to integrate RHEV deeper into your existing infrastructure, such as RHN Satellite, Cobbler, Your-super-duper-CMDB or IaaS/PaaS broker, there are two different APIs available. For the XML lovers, there is the previously known RestAPI which has some performance improvements. For the XML haters, there is now a native Python API which allows to to access RHEV entities directly as objects in your Python code. For both APIs, have a look to the Documentation.

I personally like the Python API, because a lot of other Red Hat infrastructure products come with Python APIs. So it is very easy to integrate those software pieces.

Under the hood, it is now powered by JBoss EAP6 instead of version 5. To be able to connect to standard ports 80 and 443, there is an Apache httpd with mod_proxy_ajp.

Have fun 🙂

Upgrading RHN Satellite 5.4.1 to 5.5

Red Hat has released RHN Satellite version 5.5. It is a release that is mainly a bug-fix release, but has some interesting new features as well. Here comes a brief guide how to update your RHN Satellite to the latest version. It is not a official guide, so if you trash your Satellite, it is not my fault…

Preparation
As always, before you upgrade the RHN Satellite, you need to order a new certificate. Open a Support case at Red Hat and tell them you need a new certificate for Version .5.5.

You also need to download the ISO file for the upgrade as the packages are only available in the software channel after the upgrade and activation. You can download the ISO at Red Hats download site. Of course you need to choose the architecture that matches your environment. Note that there is only one ISO available for each architecture, not two as it was before. The ISO comes with the embedded database. If you need to use an external database, use the --external-db parameter with install.pl

Ensure you have a working backup of your database before starting with the upgrade. Do this as follows:

su - oracle
db-control backup /your/back/up/directory
db-control verify /your/back/up/directory

A backup of your /etc/rhn directory is also a good idea, just for the case something is going wrong: cp -rp /etc/rhn /etc/rhn-$(date +"%F")

Ensure your database has enough free table space left. For the DATA_TBS and the UNDO_TBS it should be at least 1Gbyte, better are 2Gbyte. The following example shows an example:

[root@rhns ~]# su - oracle
-bash-4.1$ db-control report
Tablespace                  Size    Used   Avail   Use%
DATA_TBS                   16.1G   12.6G    3.5G    78%
SYSAUX                      500M  182.6M  317.3M    37%
SYSTEM                      400M  254.1M  145.8M    64%
TEMP_TBS                   1000M      0B   1000M     0%
UNDO_TBS                    3.9G  474.7M    3.4G    12%
USERS                       128M     64K  127.9M     0%
-bash-4.1$ 

You can grow the table spaces if needed by fire db-control extend UNDO_TBS.

It is also very important to have enoght free space in the /rhnsat filesystem, db-control gather-stats needs some extra space. At least 2 Gbyte to be on the safe side.

Having a look to the official upgrade guide is strongly recommended.

First you need to loop-back mount the ISO image and cd into the mountpoint:

[root@rhns ~]# mount satellite-5.5.0-20120911-rhel-6-x86_64.iso /mnt -o loop
[root@rhns ~]# cd /mnt
[root@rhns mnt]# 

Next step is to install the rhn-upgrade package.

[root@rhns mnt]# yum -y install rhn-upgrade
Loaded plugins: product-id, rhnplugin, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package rhn-upgrade.noarch 0:5.5.0.16-1.el6sat will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================
 Package         Arch       Version               Repository                                    Size
=====================================================================================================
Installing:
 rhn-upgrade     noarch     5.5.0.16-1.el6sat     redhat-rhn-satellite-5.4-server-x86_64-6      38 k

Transaction Summary
=====================================================================================================
Install       1 Package(s)

Total download size: 38 k
Installed size: 0  
Downloading Packages:
rhn-upgrade-5.5.0.16-1.el6sat.noarch.rpm                                      |  38 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : rhn-upgrade-5.5.0.16-1.el6sat.noarch                                              1/1 
Installed products updated.
  Verifying  : rhn-upgrade-5.5.0.16-1.el6sat.noarch                                              1/1 

Installed:
  rhn-upgrade.noarch 0:5.5.0.16-1.el6sat                                                             

Complete!
[root@rhns mnt]# 

The package contains documents and scripts to help you with the upgrade. They are located in the directory /etc/sysconfig/rhn/satellite-upgrade. Read those documents carefully before proceeding with the upgrade.

Upgrading
Lets do it… run the installer script with the --upgrade parameter, bold red letters are interactive input.

[root@rhns mnt]# ./install.pl --upgrade
* Starting the Red Hat Network Satellite installer.
* Performing pre-install checks.
* Pre-install checks complete.  Beginning installation.
* RHN Registration.
** Registration: System is already registered with RHN.  Not re-registering.
* Upgrade flag passed.  Stopping necessary services.
* Purging conflicting packages.
* Checking for uninstalled prerequisites.
** Checking if yum is available ...
There are some packages from Red Hat Enterprise Linux that are not part
of the @base group that Satellite will require to be installed on this
system. The installer will try resolve the dependencies automatically.
However, you may want to install these prerequisites manually.
Do you want the installer to resolve dependencies [y/N]? y
* Applying updates.
* Installing RHN packages.
Warning: yum did not install the following packages:
	geronimo-specs-compat
* Now running spacewalk-setup.
* Setting up Oracle environment.
* Setting up database.
** Database: Upgrading the database server to latest Oracle 10g:
** Database: This is a long process that is logged in:
** Database: /var/log/rhn/upgrade_db.log
*** Progress: ##############################################################
** Database: Setting up database connection for Oracle backend.
** Database: Testing database connection.
** Database: Populating database.
** Database: Skipping database population.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
* Performing initial configuration.
* Activating RHN Satellite.
** Certificate not activated.
** Upgrade process requires the certificate to be activated after the schema is upgraded.
* Enabling Monitoring.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? y
* Configuring tomcat.
** /etc/tomcat6/tomcat6.conf has been backed up to tomcat6.conf-swsave
** /etc/tomcat6/server.xml has been backed up to server.xml-swsave
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file web.xml.rej
* Configuring jabberd.
* Creating SSL certificates.
** Skipping SSL certificate generation.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
cobblerd does not appear to be running/accessible
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? y
cobblerd does not appear to be running/accessible
This portion of the RHN Satellite upgrade process has successfully completed.
Please refer to appropriate upgrade document in /etc/sysconfig/rhn/satellite-upgrade
for any remaining steps in the process.
[root@rhns mnt]# 

Now some database actions are needed. Make sure your Satellite is stopped and only the database is running:

rhn-satellite stop
service oracle start

You need to create schema statistics:

su - oracle
-bash-4.1$ db-control gather-stats
Gathering statistics...
WARNING: this may be a very slow process.
done.
-bash-4.1$ 

Now it is time to upgrade the database schema

[root@rhns mnt]# spacewalk-schema-upgrade
Schema upgrade: [satellite-schema-5.4.0.19-1.el6sat] -> [satellite-schema-5.5.0.13-1.el6sat]
Searching for upgrade path: [satellite-schema-5.4.0.19-1] -> [satellite-schema-5.5.0.13-1]
Searching for upgrade path: [satellite-schema-5.4.0.19] -> [satellite-schema-5.5.0.13]
Searching for upgrade path: [satellite-schema-5.4.0] -> [satellite-schema-5.5.0]
Searching for upgrade path: [satellite-schema-5.4] -> [satellite-schema-5.5]
The path: [satellite-schema-5.4] -> [satellite-schema-5.5]
Planning to run spacewalk-sql with [/var/log/spacewalk/schema-upgrade/20120922-132500-script.sql]
Hit Enter to continue or Ctrl+C to interrupt: Enter
Executing spacewalk-sql, the log is in [/var/log/spacewalk/schema-upgrade/20120922-132500-to-satellite-schema-5.5.log].
The database schema was upgraded to version [satellite-schema-5.5.0.13-1.el6sat].
[root@rhns mnt]# 

Now it is time to activate your RHN Satellite to be able to receive updates for the Satellite and running satellite-sync

[root@rhns ~]# rhn-satellite-activate --ignore-version-mismatch --rhn-cert=/root/rhns-cert55.cert 
RHN_PARENT: satellite.rhn.redhat.com
[root@rhns ~]# 

To rebuild the search index please run service rhn-search cleanindex

[root@rhns ~]# service rhn-search cleanindex
Stopping rhn-search...
rhn-search was not running.
Starting rhn-search...
[root@rhns ~]# 

Before restarting the RHN Satellite, check if any updates are available for it.
yum -y update

Afterward, please check if there is another database schema update available. If the output looks as following, you are safe.

[root@rhns ~]# spacewalk-schema-upgrade
Schema upgrade: [satellite-schema-5.5.0.13-1.el6sat] -> [satellite-schema-5.5.0.13-1.el6sat]
Your database schema already matches the schema package version [satellite-schema-5.5.0.13-1.el6sat].
[root@rhns ~]# 

As a verification that the upgrade is fine, run a satellite-sync to sync some new content and update a registered server. If you have more than one Satellite, run a ISS (Inter Satellite Sync) for proofing its functionality.

Troubleshooting
If something goes wrong with the database update, before reverting to a backup, first check the Oracle alert file /rhnsat/admin/rhnsat/bdump/alert_rhnsat.log to figure out what went wrong. Another good place to have a look at are the trace files located in /rhnsat/admin/rhnsat/udump

PAM and IPA authentication for RHN Satellite

If you have a larger installation on your site, you may wish to have a single source of credentials not only for common system services, but for your RHN Satellite too.

This will show you how to configure your RHN Satellite Server to use PAM with SSSD. SSSD, the System Security Services Daemon is a common framework to provide authentication services. Needless to say that IPA is supported as well.

Assumptions:

  • You have a RHN Satellite running on RHEL6
  • You have an IPA infrastructure running (at least on RHEL 6.2)

Preparations
First you need to install the ipa-client on your satellite:

yum -y install ipa-client

And then join the server to your IPA environment:

ipa-client-install -p admin

Configuring PAM as follows:

cat << EOF > /etc/pam.d/rhn-satellite
auth        required      pam_env.so
auth        sufficient    pam_sss.so 
auth        required      pam_deny.so
account     sufficient    pam_sss.so
account     required      pam_deny.so
EOF

Configure the RHN Satellite
Your Satellite now needs to be aware that there is the possibility to authenticate users with PAM against IPA.

echo "pam_auth_service = rhn-satellite" >> /etc/rhn/rhn.conf

If you have users in your IPA domain with usernames shorter than five characters, you will need to add one more line to be able to create the users in RHN Satellite:

echo "web.min_user_len = 3" >>   /etc/rhn/rhn.conf

After this change, restart your RHN Satellite

rhn-satellite restart

Configuring users
Now you can log in to your RHN Satellite with your already configured admin user and select the checkbox “Pluggable Authentication Modules (PAM)” on existing users and/or new users.

Things to be considered
It is strongly recomended to have at leat one user per organization (ususally a “Organization Administrator”) plus the “RHN Satellite Administrator” not having PAM authentication enabled. Despite of the easy implementation of redundancy with IPA, this is important for a fallback scenario when your IPA environment has some service interruptions due to mainenance or failure.

SSSD caches users credentials on the RHN Satellite system, but this is only true for users logged in at least once. The default value for offline_credentials_expiration is 0, which means no cache time limit. However, depending on your organizations scurity policy this value can vary. Please check the PAM section in /etc/sssd/sssd.conf

Further documents to read

Identity Management with IPA Part II – Kerberized NFS service

In part one I was writing how to set up an IPA server for basic user authentication.

One reason NFSv4 is not that widespreaded yet, is it needs Kerberos for proper operation. Of course this is now much easier thanks to IPA.

Goal for the part of the guide

  • Configure IPA to serve the NFS principle
  • Configure NFS to use IPA
  • Configure some IPA clients to use Kerberos for the NFS service

Requirements

  • A runing IPA service like discussed in Part I of this guide.
  • A NFS server based on RHEL6.2
  • One or more IPA-Client

Lets doit
First you need to add the NFS server and its service principal to the IPA server. On ipa1.example.com run:

[root@ipa1 ~]# ipa host-add nfs.example.com
[root@ipa1 ~]# ipa service-add nfs/nfs.example.com

Next, log on to you NFS server, lets call it nfs.example.com and install the needed additional software packages:

[root@nfs ~]# yum -y install ipa-client nfs-utils

You need to enroll you NFS-server on the IPA domain. Run the following on nfs.example.com:

[root@nfs ~]# ipa-client-install -p admin

The next step is to get a Kerberos ticket and fetch the entries needed to be added in the krb5.keytab

[root@nfs ~]# kinit admin
[root@nfs ~]# ipa-getkeytab -s ipa1.example.com -p nfs/nfs.example.com -k /etc/krb5.keytab

Before you proceed to your clients, you need to enable secure NFS, create an export and restart NFS:

[root@nfs ~]# perl -npe 's/#SECURE_NFS="yes"/SECURE_NFS="yes"/g' -i /etc/sysconfig/nfs
[root@nfs ~]# echo "/home  *(rw,sec=sys:krb5:krb5i:krb5p)" >> /etc/exports
[root@nfs ~]# mkdir /home/tester1 && cp /etc/skel/.bash* /home/tester && chmod 700 /home/tester1 && chown -R tester1:ipausers /home/tester1
[root@nfs ~]# service nfs restart

Assuming you already have set up one or more IPA-client(s), it is stright forward to enable kerberized NFS on your systems. Log in to a client and run the following:

[root@ipaclient1 ~]# yum -y install nfs-utils
[root@ipaclient1 ~]# perl -npe 's/#SECURE_NFS="yes"/SECURE_NFS="yes"/g' -i /etc/sysconfig/nfs
[root@ipaclient1 ~]# 

Lets have a look if you have been successful. First look up the users UID.

[root@ipaclient1 ~]# getent passwd tester1
tester1:*:1037700500:1037700500:Hans Tester:/home/tester1:/bin/bash
[root@ipaclient1 ~]# 

Lets mount that users home directory manually on a client:

mount -t nfs4 nfs.exmaple.com:/home/tester1 /home/tester1

To check if is working as expected, issue

[root@ipaclient1 ~]# su - tester1

Fire ls -lan and see if the UID matches the UID you got from getent. If you see UID 4294967294, then something went wrong, this is the UID for the user “nobody” when using NFSv4 on 64 bit machines.

Whats next?
You will figure out when I post part III of this guide 🙂

Have fun!