Tag: kernel

Categories
conferences

“From an Idea to a Patch in the Linux Mainline” at Open Source Summit/Embedded Linux Conference 2021

Last updated: September 20, 2021

Do you know that more than one in ten kernel developers in the latest 5.14 Linux kernel is a new developer? It means that they have had their first (ever!) patch accepted into this version. In addition, this is not a single event. In fact, it is happening for every single version in the recent history.

This will be the beginning of one of the two talks from Marta Rybczynska, Syslinbit founder, at the Open Source Summit/Embedded Linux Conference 2021 (schedule available). The conference starts in a week. You can download slides from both talks in advance. Let us start from the first one  – “From an Idea to a Patch in the Linux Mainline”. This is the third edition of the popular tutorial, updated with the recent content. If you are afraid of submitting your first Linux kernel patch, this is a session to follow. Marta takes a practical, step-by-step approach.

Set-up

The first step is to set up everything needed for the work. This starts from finding the right problem. Marta’s suggestion is to start with a bug fix or an improvement. If you are working on a new feature, it is likely you are going to fix bugs found during the development. Or adjust some internal kernel interface. Those also could be a good sources for a first patch. The tutorial suggests to avoid complex new features at the beginning. At least before you feel confident with the process…

When we do have the right problem, the next step will be setting up the development machine. This includes installing dependencies, downloading the kernel source tree (or more kernel source trees), getting familiar with the kernel source code indexing.

Then the tutorial moves on finding the bug source, illustrating it with an example of a real kernel bug. Marta shows how after the initial trace, we analyze the source. With the analysis come possible ideas of the bug source. The audience is then invited to repeat the procedure with a (fixed) bug of their choice.

Source analysis is not always enough. We might need kernel debugging tools. The tutorial covers the techniques for showing debug messages. It mentions printk() (currently rarely used directly), the general-purpose pr_* family with functions like pr_err(), and dev_* functions, like dev_err(), frequently seen in device drivers. Then it covers other techniques like tracing, performance monitoring, in-kernel debugging, and kernel testing framework.

Fix preparation

When the actual fix is done, the part many people fear is preparing and submitting the patch itself. The tutorial gives a gentle overview of the Linux Coding Style summary on one slide. Detailed instructions on checkpatch.pltake more than one, however. When the solution is ready, the time comes for the commit message with all important parts explained. This includes what Signed-off-by means and why it is important. At this point developers need a list of people to send the message to: maintainers and appropriate mailing list… Then the only thing is formatting the email, with the step-by-step commands for setting up git send-email. There are also some things to avoid when sending to the Linux Kernel Mailing List (abbreviated LKML), like sending attachments.

This is common to have a patch accepted in the 2nd (or later!) version. The talk concludes this part with tips on handling the review, like reposting after one week without a reply and checking if the distribution list is appropriate.

Some members of the audience might be working on something different than a bug. There is an additional section for them. This section includes working with low maturity patches called RFC (Request For Comments) and splitting a bigger change into a patch set.

With this set of tips and tools the audience will be equipped enough to send their first patch. The same procedure applies to many other Open Source projects, with slight modification to the tools and procedures to use.

Resources

The slides of the talk are waiting at the conference schedule page https://sched.co/lAS5 and everyone can review them before the event.

If you are interested this talk, you might also have a look at the one about working with unfamiliar code from foss-north.

Categories
kernel

Linux kernel versions

Last updated: September 17, 2021

New Linux users are often confused by the number of different kernel versions available at the same time. In this article we are going to explain what is a merge window, a development kernel, or an LTS.

Development kernel and the merge window

In the Linux development, developers may submit their changes (called patches) at any moment. However, patches that got approved only land in the main version (the master branch) during a short time period called a merge window (the kernel documentation offers a detailed description of the process). The merge window typically lasts for two weeks. During that time, all types of changes are possible. They may change the internal kernel API, add new system calls and so on. The changes included during a merge window were usually ready, tested and reviewed well before. It might happen that a change is considered not ready enough, and in this case it must wait for the next merge window.

Release candidates (-rc)

After all changes are merged during the merge window, the kernel enters the stabilization phase. It then receives release candidates (called rc in short) releases every week, and developers are encouraged to test those versions to report issues. This phase lasts for six to ten weeks (typically seven or eight).

When the kernel is in the -rc stage, only bugfixes are accepted. All feature changes must wait until the next merge window.

We can summarize the rc kernel features this way:
Kernel type: rc
Advantages: latest features
Disadvantages: under test, might include bugs (especially in early rc)

Stable versions

The stable kernel version is another name for the last released kernel. It receives updates until the next release, and updates include only bug fixes. There is no backport of features (which will be available, as we said before, only in the next version).

Kernel type: stable version
Advantages: latest features, relatively stable, up-to-date bugfixes
Disadvantages: will be discontinued after the new version is released, will require an update

Long term versions

The long term version (or long time support, LTS) is supported longer than until the next release. Usually it is the last version from the calendar year that becomes the LTS, for example 5.10 released in December 2020, 5.4 released in November 2019 and so on. It includes backports of fixes from newer releases. Those backports cannot change the kernel ABI, so there should be no difference for applications, nor additional kernel modules.

A screenshot from kernel.org in September 2021 with the current LTS versions

The LTS is usually supported during 6 years. Those versions are often used for embedded products using out-of-tree and custom modules. Staying on an LTS release allows you to avoid modifications when maintaining the product.

Kernel type: LTS (5.10, 5.4, 4.19, 4.14, 4.9 and 4.4)
Advantages: very stable, up-to-date bugfixes
Disadvantages: does not include new features

Distribution kernels

Distribution kernels require a separate mention. They usually start from an LTS version, but they might include backports of functionalities from further releases. Because of this, after a few years of the release, the distribution kernel may differ in an important way from the standard (or vanilla) LTS version.

Kernel type: distribution
Advantages: stable, up-to-date bugfixes, distribution features
Disadvantages: might be missing some latest features, does not exactly match the same version LTS

Which Linux kernel version should I use?

The answer is: it depends. You should already have an idea after reading this article, but you can read more on this subject in a blog post from the stable kernel maintainer.

Summary

In this document we have described the main types of kernel versions:
on kernel.org one can find:

  • development kernels for testing,
  • stable ones for production use if you are ready to update to a never version;
  • long term version useful for embedded devices, for example.

Distributions ship modified versions of stable and LTS kernels. The choice of the kernel depends on your needs, especially kernel modules you run, and features needed.

Categories
embedded kernel

How long is a Linux kernel version supported?

Last updated: September 17, 2021.

When you look under the hood of an embedded device, you may find out that it is running an ancient Linux kernel. Usually this will be one of the so called “longterm” or “LTS” kernels – a version that gets important bugfixes for a number of years, typically six. The latest such version is 5.10 (as of April 2021) released on December 14, 2020. The support duration for this one shown on the official kernel site is only two years. In February 2021 there was a discussion about the length of this support, with the options being two or six years. It is finally six years, and you can see the difference from the kernel releases table in April and September 2021 below.

The Linux kernel stable versions from kernel.org (as in April 2021)
The Linux kernel stable versions in April 2021 showing 5.10 supported for two years only.
Linux kernel versions in September 2021 with 5.10 support of 6 years
… and the same table in September 2021 showing 5.10 support for 6 years

The ordinary kernel versions get support only until the release of the next kernel version (for example, 5.11 will be supported until the release of 5.12 and you can check the support duration history at a dedicated Wikipedia page). Users are expected to move to the next one. However, on embedded devices, developers rarely change the kernel version, they usually support a single one during the whole lifetime of the device. The reason for this is the number of out-of-tree kernel changes they have. Out-of-tree kernel changes consist of all modifications done over the mainline version. Examples of such change include drivers not included in that specific kernel version (and added later), or received directly from the vendor, board-specific fixes and more. Linux kernel changes internal APIs between versions, so supporting out-of-tree changes might be complicated, as out-of-tree changes must be updated and tested for each kernel version.

For embedded developers who do not have out-of-tree kernel code, keeping up with the latest version might be a solution, as it is for application developers. The reason is simple. Contrary to internal APIs, one of the strong kernel development rules is avoiding breaking userspace (external APIs). Application developers should then see no regressions when upgrading from an older kernel version to a newer one.

Supporting a kernel version for 6 years means important effort, and makes sense only if there are devices using it. In addition, there is a need for people who will be test those kernels before a release. This is why the stable kernel maintainer asked for companies and project to declare if they want to use 5.10, and plan to support the testing. There have been some answers posted publicly showing engagement. It seems likely now that the support will be extended, but likely we need to wait a little bit more to know for sure. The decision has not been taken yet.