LTS Archives - Syslinbit

Last updated: September 17, 2021

New Linux users are often confused by the number of different kernel versions available at the same time. In this article we are going to explain what is a merge window, a development kernel, or an LTS.

Development kernel and the merge window

In the Linux development, developers may submit their changes (called patches) at any moment. However, patches that got approved only land in the main version (the master branch) during a short time period called a merge window (the kernel documentation offers a detailed description of the process). The merge window typically lasts for two weeks. During that time, all types of changes are possible. They may change the internal kernel API, add new system calls and so on. The changes included during a merge window were usually ready, tested and reviewed well before. It might happen that a change is considered not ready enough, and in this case it must wait for the next merge window.

Release candidates (-rc)

After all changes are merged during the merge window, the kernel enters the stabilization phase. It then receives release candidates (called rc in short) releases every week, and developers are encouraged to test those versions to report issues. This phase lasts for six to ten weeks (typically seven or eight).

When the kernel is in the -rc stage, only bugfixes are accepted. All feature changes must wait until the next merge window.

We can summarize the rc kernel features this way:
Kernel type: rc
Advantages: latest features
Disadvantages: under test, might include bugs (especially in early rc)

Stable versions

The stable kernel version is another name for the last released kernel. It receives updates until the next release, and updates include only bug fixes. There is no backport of features (which will be available, as we said before, only in the next version).

Kernel type: stable version
Advantages: latest features, relatively stable, up-to-date bugfixes
Disadvantages: will be discontinued after the new version is released, will require an update

Long term versions

The long term version (or long time support, LTS) is supported longer than until the next release. Usually it is the last version from the calendar year that becomes the LTS, for example 5.10 released in December 2020, 5.4 released in November 2019 and so on. It includes backports of fixes from newer releases. Those backports cannot change the kernel ABI, so there should be no difference for applications, nor additional kernel modules.

A screenshot from kernel.org in September 2021 with the current LTS versions

The LTS is usually supported during 6 years. Those versions are often used for embedded products using out-of-tree and custom modules. Staying on an LTS release allows you to avoid modifications when maintaining the product.

Kernel type: LTS (5.10, 5.4, 4.19, 4.14, 4.9 and 4.4)
Advantages: very stable, up-to-date bugfixes
Disadvantages: does not include new features

Distribution kernels

Distribution kernels require a separate mention. They usually start from an LTS version, but they might include backports of functionalities from further releases. Because of this, after a few years of the release, the distribution kernel may differ in an important way from the standard (or vanilla) LTS version.

Kernel type: distribution
Advantages: stable, up-to-date bugfixes, distribution features
Disadvantages: might be missing some latest features, does not exactly match the same version LTS

Which Linux kernel version should I use?

The answer is: it depends. You should already have an idea after reading this article, but you can read more on this subject in a blog post from the stable kernel maintainer.

Summary

In this document we have described the main types of kernel versions:
on kernel.org one can find:

development kernels for testing,
stable ones for production use if you are ready to update to a never version;
long term version useful for embedded devices, for example.

Distributions ship modified versions of stable and LTS kernels. The choice of the kernel depends on your needs, especially kernel modules you run, and features needed.

Last updated: September 17, 2021.

When you look under the hood of an embedded device, you may find out that it is running an ancient Linux kernel. Usually this will be one of the so called “longterm” or “LTS” kernels – a version that gets important bugfixes for a number of years, typically six. The latest such version is 5.10 (as of April 2021) released on December 14, 2020. The support duration for this one shown on the official kernel site is only two years. In February 2021 there was a discussion about the length of this support, with the options being two or six years. It is finally six years, and you can see the difference from the kernel releases table in April and September 2021 below.

The Linux kernel stable versions from kernel.org (as in April 2021) — The Linux kernel stable versions in April 2021 showing 5.10 supported for two years only.

Linux kernel versions in September 2021 with 5.10 support of 6 years — … and the same table in September 2021 showing 5.10 support for 6 years

The ordinary kernel versions get support only until the release of the next kernel version (for example, 5.11 will be supported until the release of 5.12 and you can check the support duration history at a dedicated Wikipedia page). Users are expected to move to the next one. However, on embedded devices, developers rarely change the kernel version, they usually support a single one during the whole lifetime of the device. The reason for this is the number of out-of-tree kernel changes they have. Out-of-tree kernel changes consist of all modifications done over the mainline version. Examples of such change include drivers not included in that specific kernel version (and added later), or received directly from the vendor, board-specific fixes and more. Linux kernel changes internal APIs between versions, so supporting out-of-tree changes might be complicated, as out-of-tree changes must be updated and tested for each kernel version.

For embedded developers who do not have out-of-tree kernel code, keeping up with the latest version might be a solution, as it is for application developers. The reason is simple. Contrary to internal APIs, one of the strong kernel development rules is avoiding breaking userspace (external APIs). Application developers should then see no regressions when upgrading from an older kernel version to a newer one.

Supporting a kernel version for 6 years means important effort, and makes sense only if there are devices using it. In addition, there is a need for people who will be test those kernels before a release. This is why the stable kernel maintainer asked for companies and project to declare if they want to use 5.10, and plan to support the testing. There have been some answers posted publicly showing engagement. It seems likely now that the support will be extended, but likely we need to wait a little bit more to know for sure. The decision has not been taken yet.