Since Exadata release 220.127.116.11.2, it is possible to install Oracle Linux Kernel updates via Ksplice. This allows to keep your servers up to date, without having to upgrade the whole Exadata or any downtime.
Ksplice allows you to apply Kernel patches (security updates and major bug fixes) whitout any downtime.
By installing a Ksplice updates, the patches are applied directly in memory which makes the updates transparent to running process. It does not affect the boot kernel or system library. Oracle instances, for example, do not need to be restarted to take the change into account. Oracle binaries do not need to be recompiled either.
Since the boot kernel is not modified, this means that the patches are not permanent, but they are applied automatically at each reboot of the server, early in the boot process (before the network configuration). Therefore we differentiate the “boot kernel” and the “effective kernel“.
Note that some patches are not available via Ksplice because of their complexity so Exadata upgrade is still necessary to fix some vulnerabilities or bugs. In addition, patching via KSplice is only allowed on Exadata VMs or dbnodes but not on cellnodes.
Also note: you will probably come across the term “Ksplice offline updates”. This does not mean that the installation requires a server downtime , but it refers to packages in RPM format that can be downloaded from the Oracle Linux website, uploaded to the server and installed without connection to internet or to a repository.
In order to be able to download the necessary packages, you need to have an account on ULN (linux.oracle.com) to have access to the repository. This requires an “Oracle Linux Premier Support” contract.
First, you have to identify your boot kernel version. You just have to connect to the desired node and use the command “uname -r”:
[root@dbnode1 ~]# uname -r 4.14.35-1902.303.5.3.el7uek.x86_64
Then, you need to retrieve the RPM containing the offline Ksplice updates. It is either possible to clone the ULN channel for your Linux version on a local YUM repository, or to get the RPM directly from the linux.oracle.com website. This is the method I will use in this post.
Sign In using your ULN account and click on the “Channels” tab. You can then choose your Linux version and architecture in the drop-down list:
Then you have to find the line “Ksplice for Oracle Linux 7 (x86_64)” in the displayed list and click on it (to be adapted to your version). In my case, it is at the very bottom of page 2.
Then you just have to click on the “Channel Packages” tab on the page that appears, to fill in the version number returned by the “uname -r” command previously in the search box and to click on “Go“. You should have only one result:
All you have to do is click on the download link to get the RPM:
The “uptrack-updates” RPM are cumulative. This means that it contains all the Ksplice patches available since the release of your kernel version, until the date of the RPM realease. So only the most recent RPM (i.e. the one with the latest patches) is available.
All patches will be automatically applied when the RPM is installed, but it is possible to disable a patch individually if necessary. It is also possible to disable the automatic reapplication of patches at each reboot.
Before starting, you must upload the previously downloaded RPM on the target server (in “/tmp” in my example).
On Exadata, Oracle automatically installs an “exadata-sun.*computenode-exact” package (the name varies depending on the type of environment: virtualize or bare-metal).
To simplify, this package allows to indicate that every Oracle supplied packages are in a specific release. It is installed automatically each time the Exadata is updated.
Since the installation of Ksplice patches will modify some of these packages, it is necessary to remove the RPM beforehand (this will not remove any component, it is just a “tracking” RPM).
If you have already updated some packages on the server, the “exadata-sun.*computenode-exact” package may already be removed.
(Don’t worry, this operation is approved by Oracle (Installing, Updating, and Managing Non-Oracle Software))
On the target server, identify the exact name of the package (“exadata-sun-kvm-computenode-exact.noarch” in my case, because I am on a VM on an Exadata X8M that uses KVM):
[root@dbnode1 ~]# yum list installed | grep 'exadata-sun.*computenode-exact' exadata-sun-kvm-computenode-exact.noarch
And erase it:
[root@dbnode1 ~]# yum erase exadata-sun-kvm-computenode-exact.noarch Resolving Dependencies --> Running transaction check ---> Package exadata-sun-kvm-computenode-exact.noarch 0:18.104.22.168.0.200616-1 will be erased --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================================================================================================================================== Removing: exadata-sun-kvm-computenode-exact noarch 22.214.171.124.0.200616-1 installed 0.0 Transaction Summary =========================================================================================================================================================================================================================================================== Remove 1 Package Installed size: 0 Is this ok [y/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Erasing : exadata-sun-kvm-computenode-exact-126.96.36.199.0.200616-1.noarch 1/1 Verifying : exadata-sun-kvm-computenode-exact-188.8.131.52.0.200616-1.noarch 1/1 Removed: exadata-sun-kvm-computenode-exact.noarch 0:184.108.40.206.0.200616-1 Complete!
You can now proceed to install the “uptrack-updates” package:
[root@dbnode1 ~]# yum install /tmp/uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch.rpm Examining /tmp/uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch.rpm: uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch Marking /tmp/uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64.noarch 0:20210330-0 will be installed --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================================================================================================================================== Installing: uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64 noarch 20210330-0 /uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch 48 M Transaction Summary =========================================================================================================================================================================================================================================================== Install 1 Package Total size: 48 M Installed size: 48 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch 1/1 The following steps will be taken: Install [edt48ylb] Add ftrace safety guard for existing Ksplice updates. Install [bipgvpw6] Known exploit detection. Install [lylzl1sj] Known exploit detection for CVE-2017-7308. [OUTPUT TRUNCATED] Your kernel is fully up to date. Effective kernel version is 4.14.35-2047.501.2.el7uek Verifying : uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64-20210330-0.noarch 1/1 Installed: uptrack-updates-4.14.35-1902.303.5.3.el7uek.x86_64.noarch 0:20210330-0
That’s it, the Ksplice patches are now applied!
If you use the command “uname -r”, you will not see any change on the kernel version. This is because the boot kernel is not modified:
[root@dbnode1 ~]# uname -r 4.14.35-1902.303.5.3.el7uek.x86_64
Instead, use the command “uptrack-uname -r” to see the effective kernel version
[root@dbnode1 ~]# uptrack-uname -r 4.14.35-2047.501.2.el7uek.x86_64
You can also list the updates that Ksplice has made to the effective kernel using the following command:
[root@dbnode1 ~]# uptrack-show Installed updates: [bipgvpw6] Known exploit detection. [lylzl1sj] Known exploit detection for CVE-2017-7308. [snvyltlq] Known exploit detection for CVE-2018-14634. [OUTPUT TRUNCATED] Effective kernel version is 4.14.35-2047.501.2.el7uek
I hope you find this post useful. Stay tuned for more DBA stuff!