Install Ksplice kernel updates on Exadata

Since Exadata release 12.1.1.1.2, it is possible to install Oracle Linux Kernel updates via Ksplice. This allows to keep your servers up to date, without having to upgrade the whole Exadata or any downtime.

Quick overview

Ksplice allows you to apply Kernel patches (security updates and major bug fixes) whitout any downtime.
By installing a Ksplice updates, the patches are applied directly in memory which makes the updates transparent to running process. It does not affect the boot kernel or system library. Oracle instances, for example, do not need to be restarted to take the change into account. Oracle binaries do not need to be recompiled either.

Since the boot kernel is not modified, this means that the patches are not permanent, but they are applied automatically at each reboot of the server, early in the boot process (before the network configuration). Therefore we differentiate the “boot kernel” and the “effective kernel“.

Note that some patches are not available via Ksplice because of their complexity so Exadata upgrade is still necessary to fix some vulnerabilities or bugs. In addition, patching via KSplice is only allowed on Exadata VMs or dbnodes but not on cellnodes.

Also note: you will probably come across the term “Ksplice offline updates”. This does not mean that the installation requires a server downtime , but it refers to packages in RPM format that can be downloaded from the Oracle Linux website, uploaded to the server and installed without connection to internet or to a repository.

Prerequisites

In order to be able to download the necessary packages, you need to have an account on ULN (linux.oracle.com) to have access to the repository. This requires an “Oracle Linux Premier Support” contract.

Preparation

First, you have to identify your boot kernel version. You just have to connect to the desired node and use the command “uname -r”:

[root@dbnode1 ~]# uname -r
4.14.35-1902.303.5.3.el7uek.x86_64

Then, you need to retrieve the RPM containing the offline Ksplice updates. It is either possible to clone the ULN channel for your Linux version on a local YUM repository, or to get the RPM directly from the linux.oracle.com website. This is the method I will use in this post.

Sign In using your ULN account and click on the “Channels” tab. You can then choose your Linux version and architecture in the drop-down list:

Then you have to find the line “Ksplice for Oracle Linux 7 (x86_64)” in the displayed list and click on it (to be adapted to your version). In my case, it is at the very bottom of page 2.

Then you just have to click on the “Channel Packages” tab on the page that appears, to fill in the version number returned by the “uname -r” command previously in the search box and to click on “Go“. You should have only one result:

All you have to do is click on the download link to get the RPM:

The “uptrack-updates” RPM are cumulative. This means that it contains all the Ksplice patches available since the release of your kernel version, until the date of the RPM realease. So only the most recent RPM (i.e. the one with the latest patches) is available.
All patches will be automatically applied when the RPM is installed, but it is possible to disable a patch individually if necessary. It is also possible to disable the automatic reapplication of patches at each reboot.

Installation

Before starting, you must upload the previously downloaded RPM on the target server (in “/tmp” in my example).

On Exadata, Oracle automatically installs an “exadata-sun.*computenode-exact” package (the name varies depending on the type of environment: virtualize or bare-metal).
To simplify, this package allows to indicate that every Oracle supplied packages are in a specific release. It is installed automatically each time the Exadata is updated.
Since the installation of Ksplice patches will modify some of these packages, it is necessary to remove the RPM beforehand (this will not remove any component, it is just a “tracking” RPM).
If you have already updated some packages on the server, the “exadata-sun.*computenode-exact” package may already be removed.

(Don’t worry, this operation is approved by Oracle (Installing, Updating, and Managing Non-Oracle Software))

On the target server, identify the exact name of the package (“exadata-sun-kvm-computenode-exact.noarch” in my case, because I am on a VM on an Exadata X8M that uses KVM):

[root@dbnode1 ~]# yum list installed | grep 'exadata-sun.*computenode-exact'
exadata-sun-kvm-computenode-exact.noarch

And erase it:

[root@dbnode1 ~]# yum erase exadata-sun-kvm-computenode-exact.noarch
Resolving Dependencies
--> Running transaction check
---> Package exadata-sun-kvm-computenode-exact.noarch 0:20.1.0.0.0.200616-1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================================================================================================================
 Package                                                                      Arch                                              Version                                                         Repository                                            Size
===========================================================================================================================================================================================================================================================
Removing:
 exadata-sun-kvm-computenode-exact                                            noarch                                            20.1.0.0.0.200616-1                                             installed                                            0.0

Transaction Summary
===========================================================================================================================================================================================================================================================
Remove  1 Package

Installed size: 0
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : exadata-sun-kvm-computenode-exact-20.1.0.0.0.200616-1.noarch                                                                                                                                                                            1/1
  Verifying  : exadata-sun-kvm-computenode-exact-20.1.0.0.0.200616-1.noarch                                                                                                                                                                            1/1

Removed:
  exadata-sun-kvm-computenode-exact.noarch 0:20.1.0.0.0.200616-1

Complete!

You can now proceed to install the “uptrack-updates” package:

[root@dbnode1 ~]# yum install /tmp/uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch.rpm
Examining /tmp/uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch.rpm: uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch
Marking /tmp/uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64.noarch 0:20210330-0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================================================================================================================
 Package                                                                      Arch                             Version                               Repository                                                                                       Size
===========================================================================================================================================================================================================================================================
Installing:
 uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64                           noarch                           20210330-0                            /uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch                            48 M

Transaction Summary
===========================================================================================================================================================================================================================================================
Install  1 Package

Total size: 48 M
Installed size: 48 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch                                                                                                                                                                    1/1
The following steps will be taken:
Install [edt48ylb] Add ftrace safety guard for existing Ksplice updates.
Install [bipgvpw6] Known exploit detection.
Install [lylzl1sj] Known exploit detection for CVE-2017-7308.

[OUTPUT TRUNCATED]

Your kernel is fully up to date.
Effective kernel version is 4.14.35-2047.501.2.el7uek
  Verifying  : uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch                                                                                                                                                                    1/1

Installed:
  uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64.noarch 0:20210330-0

That’s it, the Ksplice patches are now applied!

Verification

If you use the command “uname -r”, you will not see any change on the kernel version. This is because the boot kernel is not modified:

[root@dbnode1 ~]# uname -r
4.14.35-1902.303.5.3.el7uek.x86_64

Instead, use the command “uptrack-uname -r” to see the effective kernel version

[root@dbnode1 ~]# uptrack-uname -r
4.14.35-2047.501.2.el7uek.x86_64

You can also list the updates that Ksplice has made to the effective kernel using the following command:

[root@dbnode1 ~]# uptrack-show
Installed updates:
[bipgvpw6] Known exploit detection.
[lylzl1sj] Known exploit detection for CVE-2017-7308.
[snvyltlq] Known exploit detection for CVE-2018-14634.

[OUTPUT TRUNCATED]

Effective kernel version is 4.14.35-2047.501.2.el7uek

I hope you find this post useful. Stay tuned for more DBA stuff!

Leave a Reply

Your email address will not be published. Required fields are marked *