Learn How to Perform Live Kernel Updates in Virtuozzo 7 with ReadyKernel
December 20, 2016Written by Evgeniy Korepanov
Every server should be updated in order to receive new fixes for performance, stability, and most importantly security vulnerabilities. In Linux most updates can be installed without a system reboot, with the notable exception of kernel updates. Due to the reboot that is required server owners need to plan for downtime, maintenance windows, and user notification.
Last years’ statistics show that Red Hat released minor kernel updates approximately 10 times. In addition to that, virtualization software (like KVM) is also releasing critical security updates several times a year. Lastly, Virtuozzo also occasionally fixes CVEs that are delayed or rejected by RedHat (due to a different view on severity of security threats) which can compromise the security of public cloud servers.
Some of these updates contain publicly known kernel vulnerabilities and thus are considered as urgent installations.
Reboots are always a problem when dealing with mission critical services and sensitive data while at the same time trying to achieve a high level of uptime. To help eliminate a need for server reboots when applying a server update, Virtuozzo 7 now includes a service called ReadyKernel™.
Stepping back in the history of Virtuozzo, Virtuozzo 6 implemented several techniques aimed to reduce kernel update related downtime and make server management easier.
Version 6 had a feature called “Rebootless Kernel Updates”, or RKU. The idea behind it was to install a new kernel, then suspend the user space processes of each container and VM by using the same mechanism utilized for live migration (but without transferring the data anywhere). Later, they would be resumed after the host restart. The suspended VMs and containers were left right in the computer RAM, so there was no excess disk IO during the resume. However, that technology was not without its disadvantages – in particular, it could still lead to a significant (minutes for servers with large amounts of RAM) service downtime (perceived from the outside as a network connectivity interruption). Plus, it did not work on certain server hardware that required full re-initialization during such a reboot.
To overcome these problems in Virtuozzo 7 we decided to integrate a kernel update service. Since such updates may have a critical impact on the server stability (you are in essence performing “code surgery” on a running kernel), we chose to build it based on the most stable and supported mainstream Linux kernel patching mechanism called kpatch. Here is a description of kpatch from Wikipedia: https://en.wikipedia.org/wiki/Kpatch
kpatch is a feature of the Linux kernel that implements live patching of a running kernel, which allows kernel patches to be applied while the kernel is still running. By avoiding the need for rebooting the system with a new kernel that contains the desired patches, kpatch aims to maximize the system uptime and availability. At the same time, kpatch allows kernel-related security updates to be applied without deferring them to scheduled downtimes. Internally, kpatch allows entire functions in a running kernel to be replaced with their patched versions, doing that safely by stopping all running processes while the live patching is performed.
The whole idea of kpatch is to replace the functions that are found to be vulnerable with a new implementation – by redirecting the function call to a new implementation. A graphical representation of kernel functions that are replaced in the updated kernel is shown below.
Kpatch works on a functional level and replaces a whole foo() function. After the patch is applied, the original kernel function foo() stays intact and is actually still called by processes. Once the function is called, ftrace catches this event, searches for a location of a fixed version of this function, and returns its location in memory to continue code execution from the start of the fixed version of foo(). This approach lets us remove a patch at any moment safely and simply by returning to the original function in memory.
Depending on the specifics of the update, one kernel update can deliver anywhere from one to multiple patches, but the overall idea remains the same.
Clearly, applying these patches to the version N kernel does not turn it into an equivalent of its next “full release” version N+1. However, it does solve the problem of eliminating security vulnerabilities without requiring a reboot.
Virtuozzo 7 includes the ReadyKernel service to deliver patches for every supported kernel in such “live” formats. The ReadyKernel subscription for Virtuozzo (as well as for Virtuozzo Linux) is included in the base subscription at no additional cost.
Virtuozzo has a strong kernel development team that is constantly improving the Virtuozzo kernel and porting functionality and security patches to the mainstream. We also have a dedicated security group in the kernel team who are constantly monitoring public CVEs and porting them into our kernel, or fixing the issues in-house. After an issue is fixed in a development branch, it becomes possible to release a live kernel patch addressing the issue in stable kernels. Once the patch is available, it is immediately published through the official ReadyKernel repository, and the production installations can be immediately patched.
There is an online portal that lists all the supported kernels, patches for them, and a description of each patch at: https://readykernel.com/. Every patch listed there contains information about the kernel it applies to, a description of the patch, and a list of the CVEs fixed in the patch. Patches for every supported kernel are cumulative (much like Windows service packs) – the next version will contain every patch from an older version, plus some of its own.
ReadyKernel Patch Installation
The goal of ReadyKernel is to make the Virtuozzo platform more secure, and make this security easy to maintain. No more waiting for or scheduling installations and no more maintenance windows.
Patch application will cause certain kernel activities to freeze, but in reality this freeze is so brief that it can safely be ignored. Here is a real life measurement how long it takes to patch a kernel with a ReadyKernel patch:
[root@rk-vm ~]# time readykernel load
loading patch module: /var/lib/kpatch/3.10.0-327.18.2.vz7.15.2/kpatch-cumulative-2-1-r1.ko
To view ReadyKernel patches that are applied to a system, use the “readykernel info” command.
[root@evg-vzt-vm ~]# readykernel info
Patch module: kpatch_cumulative_7_0_r1
Installed package: readykernel-patch-18.7-7.0-1.vl7
The following issues are fixed by the ReadyKernel patch:
v7.0 and newer:
A BUG() statement can be hit in net/ipv4/tcp_input.c.
Potential double free in netlink_dump().
Some operations with btrfs subvolumes may consume all kernel memory.
And to automate the process entirely, you can schedule the installation of the latest ReadyKernel patch by executing the following command:
# readykernel autoupdate enable
Or even specify an hour to check for updates each day:
# readykernel autoupdate enable 12
If for any reason a host with installed ReadyKernel patches needs to be rebooted, the previously installed patches will be applied automatically upon the system boot, and the system will again be secure against patched CVEs.
If you’re an existing Virtuozzo 7 or Virtuozzo Linux user, your ReadyKernel subscription is included and available for immediate use. Once you have activated your installation, all you need to do in order to apply the latest ReadyKernel updates and do initial configuration is to run the following command:
If you would like to try Virtuozzo, follow this link to get your own evaluation copy: https://virtuozzo.com/products/virtuozzo/
In the near future we plan to deliver ReadyKernel for other popular Linux distributions- stay tuned for more announcements!