Categories
conferences tools

Yocto Project Summit 2023.11

Last week, we participated in the Yocto Project Summit 2023.11, held online from November 28 to 30, with several interesting presentations regarding the Yocto Project itself and various use cases.

Marta Rybczynska led two sessions. The first one explained the procedure for submitting fixes for known security issues to the Yocto Projects, CVE (Common Vulnerability Enumeration) fixes. Developers can learn about them from multiple sources, like the runs of cve-check. In “stable” branches, the fix usually requires the backport of an upstream patch. Most of the submission process is the same as for any other bug, but there are small but significant differences to accommodate the cve-check tool. The developer should name the patch based on the CVE number and add a “CVE:” tag inside with that information. Marta has also shared submission hints like the usage of devtool if the original upstream patch does not apply and requires modifications. Refer to the slides below; the video link will be posted when available.

YPS-2023-11-Rybczynska-How-to-submit-a-CVE-fix-v2Download

The second session concentrated on the changes in the Yocto Project security area in 2023, with the important impact of the founding of the Sovereign Tech Fund.

Do you know that it has a security team now? The project also recommends that all layers have a SECURITY.md file with information for security researchers to contact the right people in case of a security issue.

Marta also explained the work in progress. CVE synchronization work aims to avoid duplication of work when fixing CVE issues. A team is looking into the usage of SRTool for triage. A proof-of-concept implementation of the upcoming SPDX3 standard is in the works, too.

Everyone can participate in all those security initiatives; you can check slides for pointers to wiki pages and discussions.

Slides from this session are available, too.

YPS2023-11-Rybczynska-YP-Security-Now-and-Future-v2Download
Categories
conferences

“From an Idea to a Patch in the Linux Mainline” at Open Source Summit/Embedded Linux Conference 2021

Last updated: September 20, 2021

Do you know that more than one in ten kernel developers in the latest 5.14 Linux kernel is a new developer? It means that they have had their first (ever!) patch accepted into this version. In addition, this is not a single event. In fact, it is happening for every single version in the recent history.

This will be the beginning of one of the two talks from Marta Rybczynska, Syslinbit founder, at the Open Source Summit/Embedded Linux Conference 2021 (schedule available). The conference starts in a week. You can download slides from both talks in advance. Let us start from the first one  – “From an Idea to a Patch in the Linux Mainline”. This is the third edition of the popular tutorial, updated with the recent content. If you are afraid of submitting your first Linux kernel patch, this is a session to follow. Marta takes a practical, step-by-step approach.

Set-up

The first step is to set up everything needed for the work. This starts from finding the right problem. Marta’s suggestion is to start with a bug fix or an improvement. If you are working on a new feature, it is likely you are going to fix bugs found during the development. Or adjust some internal kernel interface. Those also could be a good sources for a first patch. The tutorial suggests to avoid complex new features at the beginning. At least before you feel confident with the process…

When we do have the right problem, the next step will be setting up the development machine. This includes installing dependencies, downloading the kernel source tree (or more kernel source trees), getting familiar with the kernel source code indexing.

Then the tutorial moves on finding the bug source, illustrating it with an example of a real kernel bug. Marta shows how after the initial trace, we analyze the source. With the analysis come possible ideas of the bug source. The audience is then invited to repeat the procedure with a (fixed) bug of their choice.

Source analysis is not always enough. We might need kernel debugging tools. The tutorial covers the techniques for showing debug messages. It mentions printk() (currently rarely used directly), the general-purpose pr_* family with functions like pr_err(), and dev_* functions, like dev_err(), frequently seen in device drivers. Then it covers other techniques like tracing, performance monitoring, in-kernel debugging, and kernel testing framework.

Fix preparation

When the actual fix is done, the part many people fear is preparing and submitting the patch itself. The tutorial gives a gentle overview of the Linux Coding Style summary on one slide. Detailed instructions on checkpatch.pltake more than one, however. When the solution is ready, the time comes for the commit message with all important parts explained. This includes what Signed-off-by means and why it is important. At this point developers need a list of people to send the message to: maintainers and appropriate mailing list… Then the only thing is formatting the email, with the step-by-step commands for setting up git send-email. There are also some things to avoid when sending to the Linux Kernel Mailing List (abbreviated LKML), like sending attachments.

This is common to have a patch accepted in the 2nd (or later!) version. The talk concludes this part with tips on handling the review, like reposting after one week without a reply and checking if the distribution list is appropriate.

Some members of the audience might be working on something different than a bug. There is an additional section for them. This section includes working with low maturity patches called RFC (Request For Comments) and splitting a bigger change into a patch set.

With this set of tips and tools the audience will be equipped enough to send their first patch. The same procedure applies to many other Open Source projects, with slight modification to the tools and procedures to use.

Resources

The slides of the talk are waiting at the conference schedule page https://sched.co/lAS5 and everyone can review them before the event.

If you are interested this talk, you might also have a look at the one about working with unfamiliar code from foss-north.

Categories
kernel

Linux kernel versions

Last updated: September 17, 2021

New Linux users are often confused by the number of different kernel versions available at the same time. In this article we are going to explain what is a merge window, a development kernel, or an LTS.

Development kernel and the merge window

In the Linux development, developers may submit their changes (called patches) at any moment. However, patches that got approved only land in the main version (the master branch) during a short time period called a merge window (the kernel documentation offers a detailed description of the process). The merge window typically lasts for two weeks. During that time, all types of changes are possible. They may change the internal kernel API, add new system calls and so on. The changes included during a merge window were usually ready, tested and reviewed well before. It might happen that a change is considered not ready enough, and in this case it must wait for the next merge window.

Release candidates (-rc)

After all changes are merged during the merge window, the kernel enters the stabilization phase. It then receives release candidates (called rc in short) releases every week, and developers are encouraged to test those versions to report issues. This phase lasts for six to ten weeks (typically seven or eight).

When the kernel is in the -rc stage, only bugfixes are accepted. All feature changes must wait until the next merge window.

We can summarize the rc kernel features this way:
Kernel type: rc
Advantages: latest features
Disadvantages: under test, might include bugs (especially in early rc)

Stable versions

The stable kernel version is another name for the last released kernel. It receives updates until the next release, and updates include only bug fixes. There is no backport of features (which will be available, as we said before, only in the next version).

Kernel type: stable version
Advantages: latest features, relatively stable, up-to-date bugfixes
Disadvantages: will be discontinued after the new version is released, will require an update

Long term versions

The long term version (or long time support, LTS) is supported longer than until the next release. Usually it is the last version from the calendar year that becomes the LTS, for example 5.10 released in December 2020, 5.4 released in November 2019 and so on. It includes backports of fixes from newer releases. Those backports cannot change the kernel ABI, so there should be no difference for applications, nor additional kernel modules.

A screenshot from kernel.org in September 2021 with the current LTS versions

The LTS is usually supported during 6 years. Those versions are often used for embedded products using out-of-tree and custom modules. Staying on an LTS release allows you to avoid modifications when maintaining the product.

Kernel type: LTS (5.10, 5.4, 4.19, 4.14, 4.9 and 4.4)
Advantages: very stable, up-to-date bugfixes
Disadvantages: does not include new features

Distribution kernels

Distribution kernels require a separate mention. They usually start from an LTS version, but they might include backports of functionalities from further releases. Because of this, after a few years of the release, the distribution kernel may differ in an important way from the standard (or vanilla) LTS version.

Kernel type: distribution
Advantages: stable, up-to-date bugfixes, distribution features
Disadvantages: might be missing some latest features, does not exactly match the same version LTS

Which Linux kernel version should I use?

The answer is: it depends. You should already have an idea after reading this article, but you can read more on this subject in a blog post from the stable kernel maintainer.

Summary

In this document we have described the main types of kernel versions:
on kernel.org one can find:

  • development kernels for testing,
  • stable ones for production use if you are ready to update to a never version;
  • long term version useful for embedded devices, for example.

Distributions ship modified versions of stable and LTS kernels. The choice of the kernel depends on your needs, especially kernel modules you run, and features needed.

Categories
tools

Useful Linux network commands

Last updated: September 17, 2021.

Those using Linux for a long time remember that networking was configured using ifconfig and route. Those two commands are still available in some distributions, but they have disappeared in others. In addition, a number of other tools appeared. For example, NetworkManager became de facto standard of the network management service.

Let us cover a number of useful networking commands. You should replace the parts marked with <> with your own configuration. For example, <ifce> may be eth0 in your configuration. The $ sign marks your prompt.

Basic networking

Basic networking operations include showing available interfaces, their addresses and configuration options; it also includes routing.

Showing all interfaces with their addresses

You can show all interfaces with their status, and assigned addresses using one of the two commands (the second one is the abbreviated version):
$ ip addr show
$ ip a

It is an equivalent of (older) ifconfig or ifconfig -a.

Setting addresses

The commands for adding and deleting network addresses are quite similar. With ip they have the following form:
$ ip addr add <address> dev <ifce>
$ ip addr del <address>/<prefix> dev <ifce>

In the delete command, you can still (in 2021) specify the address without a prefix, but you will get a warning message.

They are equivalents of older:
$ ifconfig <ifce> add <address>
$ ifconfig <ifce> del <address>

Setting up MTU

In some cases we might want to change the MTU (Maximum Transmission Unit) value, giving the maximum size of a datagram packet in the network. With ip the command is:
$ ip link set dev <ifce> mtu <mtu_value>
It is an equivalent of:
$ ifconfig <ifce> mtu <mtu_value>

Setting interfaces up and down

If we want temporarily disable an interface and stop receiving packets, we put the interface “down”. It will start receiving packets again when we move it back “up”. You can change the state of the interface using commands like:

$ ip link set <ifce> down
$ ip link set <ifce> up

They are equivalents of older:
$ ifconfig <ifce> down
$ ifconfig <ifce> up

Showing routing

If you want to show the routing information of the host, use the following command:
$ ip route show
It is an equivalent of the older route command:
$ route -a
or

$ route

Adding a default gateway

The default gateway points to the gateway to be used if there are no other matches in the routing table. Usually it will be your external interface. To set up the default gateway use:

$ ip route add default via <gateway_ip>
It is an equivalent of:
$ route add default gw <gateway_ip>

Adding and removing routes

If you’d like to learn more about the routing concepts, you can read the introduction guide (using route-based commands).

The command to add and remove a route look like:

$ ip route add <network>/<mask> via <gateway_ip> dev <ifce>
$ ip route del <network>/<mask> via <gateway_ip> dev <ifce>

We can also start typing the beginning of the command, for example ip route del <network>/<mask>

The commands using route were:

$ route add -net <network>/<mask> gw <gateway_ip> <ifce>

$ route del -net <network>/<mask> gw <gateway_ip> <ifce>

Getting packet statistics

In Linux there are a number of ways to get statistics about received and sent packets. We can obtain them using the following commands:
$ ip -s link
$ cat /proc/net/dev
You can also look into individual files with the interface statistics, for example the number of received bytes is available in /sys/class/net/<ifce>/statistics/rx_bytes. To show the value you can run:
$ cat /sys/class/net/<ifce>/statistics/rx_bytes

All of them can replace packet statistics of ifconfig or netstat -i

NetworkManager

NetworkManager is a tool configuring automatically all types of network, included in most distributions. It has a number of graphical frontends, but also a handy command-line tool, nmcli.

Making an interface unmanaged

In a typical configuration NetworkManager will be taking care of all network interfaces of your system. Sometimes we need more control, for example to set up an IP address manually for testing. To do this, we need to put the interface in the unmanaged mode for NetworkManager. This commands allows us to manually handle addresses and routing of an interface. If an interface stays in managed mode, NetworkManager will reapply its configuration and manual changes may disappear.

$ nmcli device set <ifce> managed no

This command will take effect only until a reboot or restarting NetworkManager. It is also possible to permanently move an interface into unmanaged mode. In this case, change /etc/NetworkManager/conf.d/99-unmanaged-devices.conf or /etc/NetworkManager/NetworkManager.conf and add the following:

[keyfile]
unmanaged-devices=interface-name:<ifce>

You can learn more about configuration of NetworkManager from the documentation.

Verifying status of interfaces in NetworkManager

If you need to check the high level status of devices (names, connected or not), the NetworkManager command is:

$ nmcli device status

Checking detailed status in NetworkManager

NetworkManager allows also to check all detailed information about system network interfaces, including the IP addresses, DNS, routing, MTU. The command is:

$ nmcli device show

Categories
conferences kernel

Working with unknown code: a talk at foss-north 2021

During the foss-north conference, Syslinbit founder Marta Rybczynska will be delivering a keynote. The talk entitled “Into the Jungle – How to write a Linux kernel driver for any subsystem” will lead the audience through writing a new module for an unknown code base. It will show tips and techniques useful not only when working with Linux, but any other big software project.

foss-north 2021: Marta Rybczynska, "Into the Jungle - How to write a Linux kernel driver for any subsystem"

During 20 years in Open Source projects, Marta saw many times developers approaching unknown code. Their reactions differ. Some read all the documentation online for days, others develop from scratch without planning. Finally, we can also see people who decide to re-write everything from scratch, or those who stop paralyzed in panic.

In the talk, Marta is going to show a step-by-step way to approach some unknown code. The audience will see how to learn the new subsystem, and how to test if the taken approach is the right one.

She will use an example of writing a new network driver for Linux, from scratch. It will simulate adding support for a new hardware device. During the process, she will guide the audience to look into the extensive kernel documentation, source code of similar drivers, show what the driver should be doing to work correctly… and more.

Even if the example is will be specific to Linux, the method is not. The audience will be able to use it, with minor changes, for any other unfamiliar code base.

The conference takes place on May 31st and June 1st, and will be a virtual event, with a community day on May 30. It is then accessible to everyone. The list of keynotes is already available on the website, and a blog posts gives some more details.

Categories
embedded kernel

How long is a Linux kernel version supported?

Last updated: September 17, 2021.

When you look under the hood of an embedded device, you may find out that it is running an ancient Linux kernel. Usually this will be one of the so called “longterm” or “LTS” kernels – a version that gets important bugfixes for a number of years, typically six. The latest such version is 5.10 (as of April 2021) released on December 14, 2020. The support duration for this one shown on the official kernel site is only two years. In February 2021 there was a discussion about the length of this support, with the options being two or six years. It is finally six years, and you can see the difference from the kernel releases table in April and September 2021 below.

The Linux kernel stable versions from kernel.org (as in April 2021)
The Linux kernel stable versions in April 2021 showing 5.10 supported for two years only.
Linux kernel versions in September 2021 with 5.10 support of 6 years
… and the same table in September 2021 showing 5.10 support for 6 years

The ordinary kernel versions get support only until the release of the next kernel version (for example, 5.11 will be supported until the release of 5.12 and you can check the support duration history at a dedicated Wikipedia page). Users are expected to move to the next one. However, on embedded devices, developers rarely change the kernel version, they usually support a single one during the whole lifetime of the device. The reason for this is the number of out-of-tree kernel changes they have. Out-of-tree kernel changes consist of all modifications done over the mainline version. Examples of such change include drivers not included in that specific kernel version (and added later), or received directly from the vendor, board-specific fixes and more. Linux kernel changes internal APIs between versions, so supporting out-of-tree changes might be complicated, as out-of-tree changes must be updated and tested for each kernel version.

For embedded developers who do not have out-of-tree kernel code, keeping up with the latest version might be a solution, as it is for application developers. The reason is simple. Contrary to internal APIs, one of the strong kernel development rules is avoiding breaking userspace (external APIs). Application developers should then see no regressions when upgrading from an older kernel version to a newer one.

Supporting a kernel version for 6 years means important effort, and makes sense only if there are devices using it. In addition, there is a need for people who will be test those kernels before a release. This is why the stable kernel maintainer asked for companies and project to declare if they want to use 5.10, and plan to support the testing. There have been some answers posted publicly showing engagement. It seems likely now that the support will be extended, but likely we need to wait a little bit more to know for sure. The decision has not been taken yet.

Categories
tools

Time zones: How time changes on Linux

During the time period when we change time from the winter one to the summer one (or the other way around), millions of Linux devices do the switch automatically. How is it possible, when you know that the change happens in different places in the world at different dates? Learn how the time zone database works on Linux and how to check how the time changes anywhere in the world.

The source of the solution is the timezone database (with tools), which you can download from ftp://ftp.iana.org/tz/releases/ The database describes how much time at each location is offset from the GMT time, and when it changes. Then the Linux system uses this data to adjust the displayed time.

You can check your current time zone by looking into /etc/timezone, which contains the user-readable name. For example:
$ cat /etc/timezone
Europe/Paris

The machine-readable data is somewhere else. On Debian, you can find it in /etc/localtime, which is a symbolic link to the correct file. For example:
$ ls -l /etc/localtime
lrwxrwxrwx 1 root root 32 Feb 12 09:29 /etc/localtime ->
/usr/share/zoneinfo/Europe/Paris

You can dump the dates of the time change using the zdump tool (filter the year, otherwise the output will be long):
$ zdump -v /etc/localtime|grep 2021
/etc/localtime Sun Mar 28 00:59:59 2021 UT = Sun Mar 28 01:59:59 2021
CET isdst=0 gmtoff=3600
/etc/localtime Sun Mar 28 01:00:00 2021 UT = Sun Mar 28 03:00:00 2021
CEST isdst=1 gmtoff=7200
/etc/localtime Sun Oct 31 00:59:59 2021 UT = Sun Oct 31 02:59:59 2021
CEST isdst=1 gmtoff=7200
/etc/localtime Sun Oct 31 01:00:00 2021 UT = Sun Oct 31 02:00:00 2021
CET isdst=0 gmtoff=3600

The output contains the moment the time changes, the binary flag if this time
is DST (daylight savings time), and the offset to GMT (in seconds).

With the same tool you can also check the time changes for any other place,
for example:

$ zdump -v /usr/share/zoneinfo/Antarctica/South_Pole |grep 2021
/usr/share/zoneinfo/Antarctica/South_Pole Sat Apr 3 13:59:59 2021 UT
= Sun Apr 4 02:59:59 2021 NZDT isdst=1 gmtoff=46800
/usr/share/zoneinfo/Antarctica/South_Pole Sat Apr 3 14:00:00 2021 UT
= Sun Apr 4 02:00:00 2021 NZST isdst=0 gmtoff=43200
/usr/share/zoneinfo/Antarctica/South_Pole Sat Sep 25 13:59:59 2021 UT
= Sun Sep 26 01:59:59 2021 NZST isdst=0 gmtoff=43200
/usr/share/zoneinfo/Antarctica/South_Pole Sat Sep 25 14:00:00 2021 UT
= Sun Sep 26 03:00:00 2021 NZDT isdst=1 gmtoff=46800

If you want to display time in your program, the system libraries do the conversion automatically. For example, in C the function to use is localtime(). You can learn more from the function’s manual page or from https://man7.org/linux/man-pages/man3/localtime.3.html