Live Embedded Event: a new online conference

Live Embedded EventIn these times of COVID19, pretty much all of the existing conferences have moved to an online format. For example, the Embedded Linux Conference Europe is going to take place next week, online, and Bootlin will significantly contribute to the event with no less than 7 talks on a wide range of topics.

But this trend for online conferences has also spurred the creation of new events. And specifically, we’re happy to announce the creation of a new conference oriented towards our favorite topic of embedded systems: Live Embedded Event. It will take place online on December 3 and will have a broader range of topics covered than ELC typically has, as Live Embedded Event is open to non-Linux embedded topics, hardware platform and interfaces discussions, and more.

The event is mainly organized by four French companies specialized in embedded systems, and we expect the contents to be a mix of talks in English and in French. Speaking of contents, the Call For Papers is open until November 10, so you can have the chance to share your knowledge about embedded topics, show your expertise, and meet new users and customers. And of course, you can also simply register as an attendee: the event is free !

Important links:

Feedback from the Netdev 0x13 conference

The Netdev 0x13 conference took place last week in Prague, Czech Republic. As we work on a variety of networking topics as part of our Linux kernel contributions, Bootlin engineers Maxime Chevallier and Antoine Ténart went to meet with the Linux networking community and to see a lot of interesting sessions. It’s the third time we enjoy attending the Netdev conference (after Netdev 2.1 and Netdev 2.2) and as always, it was a blast!

The 3-day conference started with a first day of workshops and tutorials. We enjoyed learning how to be the cool kids thanks to the XDP hands-on tutorial where Jesper Brouer and Toke Høiland-Jørgensen cooked us a number of lessons to progressively get to learn how to write and load XDP programs. This was the first trial-run of the tutorial which is meant to be extended and used as a material to go through the XDP basics. The instructions are all available on Github.

We then had the chance to attend the TC workshop where face to face discussions and presentations of the traffic control hot topics being worked on happened. The session caught our attention as the topic is related to current subjects being worked on at Bootlin.

Being used to work on embedded systems, seeing the problems the Network developers face can sometimes come as a surprise. During the TC workshop, Vlad Buslov presented his recent work on removing TC flower’s the dependency to the global rtnl lock, which is an issue when you have a million classification rules to update quickly.

We also went to the hardware offload workshop. The future of the network offload APIs and support in the Linux kernel was discussed, with various topics ranging from ASIC support to switchev advanced use-cases or offloading XDP. This was very interesting to us as we do work on various networking engines providing many offloading facilities to the kernel.

The next two days were a collection of talks presenting the recent advances in the networking subsystem of the Linux kernel, as well as current issues and real-world examples of recent functionalities being leveraged.

As always XDP was brought-up with a presentation of XDP offloading using virtio-net, recent advances in combining XDP and hardware offloading techniques and a feedback from Cloudflare using XDP in their DDOS mitigation in-house solution.

But we also got to see other topics, such as SO_TIMESTAMPING being used for performance analytics. In this talk Soheil Hassas Yeganeh presented how the kernel timestamping facilities can be used to track individual packets withing the networking stack for performance analysis and debugging. This was nice to see as we worked on enabling hardware timestamping in networking engines and PHYs for our clients.

Another hot topic this year was the QUIC protocol, which was presented in details in the very good QUIC tutorial by Jana Iyengar. Since this protocol is fairly new, it was brought-up in several sessions from a lot of interesting angles.

Although QUIC was not the main subject of Alissa Cooper’s keynote on Open Source, the IETF, and You, she explained how QUIC was an example of a protocol that is designed alongside its implementations, having a tight feedback loop between the protocol specifications and its usage in real-life. Alissa shared Jana’s point on how middle-boxes are a problem when designing and deploying new protocols, and explained that an approach to overcome this “ossification” is to encrypt the protocol header themselves and document the invariant parts of the non-encrypted parts.

A consequence of having a flexible protocol is that it is not meant to be implemented in the kernel. However, Maciej Machnikowski and Joshua Hay explained that it is still possible to offload some of the processing to hardware, which sparked interesting discussions with the audience on how to do so.

Conclusion

The Netdev 0x13 conference was well organized and very pleasant to attend. The content was deeply technical and allowed us to stay up-to-date with the latest developments. We also had interesting discussions and came back with lots of ideas to explore.

Thanks for organizing Netdev, we had an amazing time!

Bootlin at the Buildroot Developers Meeting, February 2019

Buildroot logoIt’s now a tradition: the Buildroot project organizes one of its Buildroot Developers Meeting right after the FOSDEM conference. 2019 was no exception, and the meeting took place from February 4 to February 6, a three days duration, instead of the traditional two days duration from the previous years. Once again, the meeting was sponsored by Google, who provided the meeting location and lunch for all participants. Bootlin participated to the event, by allowing its engineer Thomas Petazzoni to join the meeting.

The meeting was a mix of discussions on various topics and actual hacking, with a focus on reducing the backlog of pending patches. The report synthetizes the most important discussion items:

  • A short general assembly of the Buildroot Association took place
  • Some discussions around the download infrastructure took place, related to the re-introduction of the make source-check feature and the issue of tarball reproducibility with version control system download backends
  • Discussion about introducing Config.in options for all host packages, an idea that we decided to not pursue for the moment.
  • Discussion about the instrumentation hooks that are used to collect the list of files installed by packages, and how we can achieve this goal in a way that is both efficient and reliable
  • Discussion on which Qt5 versions to support
  • Discussion on participating to the Google Summer of Code. We wrote a few topic ideas and applied as an organization for GSoC 2019.
  • Discussion on how to integrate support for systemd sysusers mechanism

Reading the work on the pending patches, we managed to reduce the backlog from about 300 patches to around 170 patches, which is a very significant achievement.

From left to right: Mark Corbin, Adam Duskett, Angelo Compagnucci (front), Peter Korsgaard (back), Thomas Petazzoni (front), Arnout Vandecappelle (back), Thomas De Schampheleire, Adrian Perez de Castro and Titouan Christophe. Behind the camera: Yann E. Morin.

More specifically, Thomas Petazzoni took advantage of this meeting to:

  • Finalize his work on the pkg-stats script, to include details about the latest available upstream version of each Buildroot package. To do so, it relies on information provided by the release-monitoring.org website. The information is not yet accurate for all packages, but the accuracy can be improved by contributing to release-monitoring.org. The updated package statistics page now provides those details, which will help ensure Buildroot packages are kept up-to-date.
  • Review in detail the patch series from Adam Duskett introducing support for GObject Introspection. It is far from a trivial package due to the need to run during the build some small binaries using Qemu. While the series is not merged yet, we have a much better understanding of it, which will help complete the review process.
  • Do a final review and apply the lengthy patch series reworking the fftw package.
  • Participate, as a Buildroot co-maintainer, to the pending patches backlog cleanup, by reviewing and/or merging a significant number of patches.

It was once again a very nice and productive meeting. The next meeting will take place as usual around the Embedded Linux Conference Europe, in October, in Lyon (France).

Feedback from Bootlin at the Linux Plumbers Conference 2018

The Linux Plumbers Conference (LPC) was held a few weeks ago in Vancouver, BC. As always there were several tracks where contributors gave a presentation of on-going or future work, and discussed it with the audience, on specific topics such as thermal, containers, real time, device tree and many more. For the first time at LPC a 2-day networking track took place. As we work on a diversity of networking projects at Bootlin we decided to attend.

Networking track at LPC. Photo @linuxplumbers.

The hot topic of the last couple of years in conferences in the network subsystem is XDP, so the conference was not exception. We saw a handful of talks and discussions about the on-going work and support of XDP within the kernel. XDP provides a programmable network data path (using eBPF) in the Linux kernel to process bare metal packets at the lowest point in the network stack. Packets are processed directly in the drivers’ Rx queues, before any allocation happen (such as socket buffers). Facebook is one well known heavy user of this technology (every packet toward Facebook is processed by XDP) and its engineers gave feedback about how they use XDP and the issues they faced. Other projects and companies are currently evaluating and starting to use XDP as well: we also saw presentations about XDP/eBPF in Open vSwitch, DPDK or kTLS.

While XDP/eBPF was featured in most of the discussions, other interesting topics where brought up. Andrew Lunn gave a presentation about the current need to go beyond 1G copper PHYs for many Linux enabled embedded devices. This was very interesting for us as we used and worked on the technologies used within the Linux kernel to address this, such as Phylink and the SFP bus (we used those when enabling 10G interfaces in the Marvell MacchiatoBin board).

Another presentation caught our attention as the topic was related to what we do at Bootlin. Jesse Brandeburg from Intel talked about the networking hardware offloads and their APIs. He exposed a brief history of the offloads supported by NICs and then showed some issues with the current APIs, where some use cases or behaviors are not clearly defined and sometimes overlap. This is a feeling we share as we experienced it while implementing some of those hardware networking offloads. Jesse’s idea was to open a discussion to come up with better solutions within the next years, as NICs offloading continue to grow.

The Linux Plumbers Conference was very pleasant and well organized. We had the chance to attend the networking track, seeing lots of great cutting-edge topics being discussed; as well as other interesting tracks.

We’d like to thank the conference and track organizers, we had a great time! Videos, slides and papers are now available on the official website or on Youtube.

Bootlin at the Embedded Linux Conference 2018

Like every year for more than 10 years, Bootlin engineers will participate to the next Embedded Linux Conference, which takes place in Portland on March 12-14. Of course, it will be our first ELC with our new company name! In total, eight engineers from Bootlin will participate to the event. Maxime Chevallier, who joined Bootlin last Monday, will be attending the conference, his first one with a Bootlin hat (but Maxime has already been a speaker at the last Embedded Linux Conference Europe).

Embedded Linux Conference 2018

We will also be giving a number of talks, tutorials or moderating Bird of a Feather sessions:

We’re really happy to again meet the embedded Linux open-source community at this event! It is worth mentioning that following this event, Bootlin CTO Thomas Petazzoni will be in the Silicon Valley on March 15-16, available for business meetings: do not hesitate to contact us if you’re interested.

Feedback from the Netdev 2.2 conference

The Netdev 2.2 conference took place in Seoul, South Korea. As we work on a diversity of networking topics at Bootlin as part of our Linux kernel contributions, Bootlin engineers Alexandre Belloni and Antoine Ténart went to Seoul to attend lots of interesting sessions and to meet with the Linux networking community. Below, they report on what they learned from this conference, by highlighting two talks they particularly liked.

Linux Networking Dietary Restrictions — slides — video

David S. Miller gave a keynote about reducing the size of core structures in the Linux kernel networking core. The idea behind his work is to use smaller structures which has many benefits in terms of performance as less cache misses will occur and less memory resources are needed. This is especially true in the networking core as small changes may have enormous impacts and improve performance a lot. Another argument from his maintainer hat perspective is the maintainability, where smaller structures usually means less complexity.

He presented five techniques he used to shrink the networking core data structures. The first one was to identify members of common base structures that are only used in sub-classes, as these members can easily be moved out and not impact all the data paths.

The second one makes use of what David calls “state compression”, aka. understanding the real width of the information stored in data structures and to pack flags together to save space. In his mind a boolean should take a single bit whereas in the kernel it requires way more space than that. While this is fine for many uses it makes sense to compress all these data in critical structures.

Then David S. Miller spoke about unused bits in pointers where in the kernel all pointers have 3 bits never used. He argued these bits are 3 boolean values that should be used to reduce core data structure sizes. This technique and the state compression one can be used by introducing helpers to safely access the data.

Another technique he used was to unionize members that aren’t used at the same time. This helps reducing even more the structure size by not having areas of memory never used during identified steps in the networking stack.

Finally he showed us the last technique he used, which was using lookup keys instead of pointers when the objects can be found cheaply based on their index. While this cannot be used for every object, it helped reducing some data structures.

While going through all these techniques he gave many examples to help understanding what can be saved and how it was effective. This was overall a great talk showing a critical aspect we do not always think of when writing drivers, which can lead to big performance improvements.

David S. Miller at Nedev 2.2

WireGuard: Next-generation Secure Kernel Network Tunnel — slides — video

Jason A. Donenfeld presented his new and shiny L3 network tunneling mechanism, in Linux. After two years of development this in-kernel formally proven cryptographic protocol is ready to be submitted upstream to get the first rounds of review.

The idea behind Wireguard is to provide, with a small code base, a simple interface to establish and maintain encrypted tunnels. Jason made a demo which was impressive by its simplicity when securely connecting two machines, while it can be a real pain when working with OpenVPN or IPsec. Under the hood this mechanism uses UDP packets on top of either IPv4 and IPv6 to transport encrypted packets using modern cryptographic principles. The authentication is similar to what SSH is using: static private/public key pairs. One particularly nice design choice is the fact that Wireguard is exposed as a stateless interface to the administrator whereas the protocol is stateful and timer based, which allow to put devices into sleep mode and not to care about it.

One of the difficulty to get Wireguard accepted upstream is its cryptographic needs, which do not match what can provide the kernel cryptographic framework. Jason knows this and plan to first send patches to rework the cryptographic framework so that his module nicely integrates with in-kernel APIs. First RFC patches for Wireguard should be sent at the end of 2017, or at the beginning of 2018.

We look forward to seeing Wireguard hit the mainline kernel, to allow everybody to establish secure tunnels in an easy way!

Jason A. Donenfeld at Netdev 2.2

Conclusion

Netdev 2.2 was again an excellent experience for us. It was an (almost) single track format, running alongside the workshops, allowing to not miss any session. The technical content let us dive deeply in the inner working of the network stack and stay up-to-date with the current developments.

Thanks for organizing this and for the impressive job, we had an amazing time!

Back from ELCE: selection of talks from the Bootlin team

As discussed in our previous blog post, Bootlin had a strong presence at the Embedded Linux Conference Europe, with 7 attendees, 4 talks, one BoF and one poster during the technical show case.

In this blog post, we would like to highlight a number of talks from the conference that we found interesting. Each Bootlin engineer who attended the conference has selected one talk, and gives his/her feedback about this talk.

uClibc Today: Still Makes Sense – Alexey Brodkin

Talk selected by Michael Opdenacker

uClibc Today: Still Makes Sense – Alexey BrodkinAlexey Brodkin, an active contributor to the uClibc library, shared recent updates about this C library, trying to show people that the project was still active and making progress, after a few years during which it appeared to be stalled. Alexey works for Synopsys, the makers of the ARC architecture, which uClibc supports.

If you look at the repository for uClibc releases, you will see that since version 1.0.0 released in 2015, the project has made 26 releases according to a predictable schedule. The project also runs runtime regression tests on all its releases, which weren’t done before. The developers have also added support for 4 new architectures (arm64 in particular), and uClibc remains the default C library that Buildroot proposes.

Alexey highlighted that in spite of the competition from the musl library, which causes several projects to switch from uClibc to musl, uClibc still makes a lot of sense today. As a matter of fact, it supports more hardware architectures than glibc and musl do, as it’s the only one to support platforms without an MMU (such as noMMU ARM, Blackfin, m68k, Xtensa) and as the library size is still smaller that what you get with musl (though a static hello_world program is much smaller with musl if you have a close look at the library comparison tests he mentioned).

Alexey noted that the uClibc++ project is still alive too, and used in OpenWRT/Lede by default.

Read the slides and watch the video of his talk.

Identifying and Supporting “X-Compatible” hardware blocks – Chen-Yu Tsai

Talk selected by Quentin Schulz

Identifying and Supporting “X-Compatible” hardware blocks – Chen-Yu TsaiAn SoC is made of multiple IP blocks from different vendors. In some cases the source or model of the hardware blocks are neither documented nor marketed by the SoC vendor. However, since there are only very few vendors of a given IP block, stakes are high that your SoC vendor’s undocumented IP block is compatible with a known one.

With his experience in developing drivers for multiple IP blocks present in Allwinner SoCs and as a maintainer of those same SoCs, Chen-Yu first explained that SoC vendors often either embed some vendors’ licensed IP blocks in their SoCs and add the glue around it for platform- or SoC-specific hardware (clocks, resets and control signals), or they clone IP blocks with the same logic but some twists (missing, obfuscated or rearranged registers).

To identify the IP block, we can dig into the datasheet or the vendor BSP and compare those with well documented datasheets such as the one for NXP i.MX6, TI KeyStone II or the Zynq UltraScale+ MPSoC, or with mainline drivers. Asking the community is also a good idea as someone might have encountered an IP with the same behaviour before and can help us identify it quicker.

Some good identifiers for IPs could be register layouts or names along with DMA logic and descriptor format. For the unlucky ones that have been provided only a blob, they can look for the symbols in it and that may slightly help in the process.

He also mentioned that identifying an IP block is often the result of the developer’s experience in identifying IPs and other time just pure luck. Unfortunately, there are times when someone couldn’t identify the IP and wrote a complete driver just to be told by someone else in the community that this IP reminds him of that IP in which case the work done can be just thrown away. That’s where the community plays a strong role, to help us in our quest to identify an IP.

Chen-Yu then went on with the presentation of the different ways to handle the multiple variants of an IP blocks in drivers. He said that the core logic of all IP drivers is usually presented as a library and that the different variants have a driver with their own resources, extra setup and use this library. Also, a good practice is to use booleans to select features of IP blocks instead of using the ID of each variant.
For IPs whose registers have been altered, the way to go is usually to write a table for the register offsets, or use regmaps when bitfields are also modified. When the IP block differs a bit too much, custom callbacks should be used.

He ended his talk with his return from experience on multiple IP blocks (UART, USB OTG, GMAC, EMAC and HDMI) present in Allwinner SoCs and the differences developers had to handle when adding support for them.

Read the slides and watch the video of his talk.

printk(): The Most Useful Tool is Now Showing its Age – Steven Rostedt & Sergey Senozhatsky

Talks selected by Boris Brezillon. Boris also covered the related talk “printk: It’s Old, What Can We Do to Make It Young Again?” from the same speakers.

printk(): The Most Useful Tool is Now Showing its Age – Steven Rostedt & Sergey SenozhatskyMaybe I should be ashamed of saying that but printk() is one of the basic tool I’m using to debug kernel code, and I must say it did the job so far, so when I saw these presentations talking about how to improve printk() I was a bit curious. What could be wrong in printk() implementation?
Before attending the talks, I never digged into printk()’s code, because it just worked for me, but what I thought was a simple piece of code appeared to be a complex infrastructure with locking scheme that makes you realize how hard it is when several CPUs are involved.

At its core, printk() is supposed to store logs into a circular buffer and push new entries to one or several consoles. In his first talk Steven Rostedt walked through the history of printk() and explained why it became more complex when support for multi CPU appeared. He also detailed why printk() is not re-entrant and the problem it causes when called from an NMI handler. He finally went through some fixes that made the situation a bit better and advertised the 2nd half of the talk driven by Sergey Senozhatsky.

Note that between these two presentations, the printk() rework has been discussed at Kernel Summit, so Sergey already had some feedback on his proposals. While Steven presentation focused mainly on the main printk() function, Sergey gave a bit more details on why printk() can deadlock, and one of the reasons why preventing deadlocks is so complicated is that printk() delegates the ‘print to console’ aspect to console drivers which have their own locking scheme. To address that, it is proposed to move away from the callback approach and let console drivers poll for new entries in the console buffer instead, which would remove part of the locking issues. The problem with this approach is that it brings even more uncertainty on when the logs are printed on the consoles, and one of the nice things about printk() in its current form is that you are likely to have the log printed on the output before printk() returns (which helps a lot when you debug things).

He also mentioned other solutions to address other possible deadlocks, but I must admit I got lost at some point, so if you’re interested in this topic I recommend that you watch the video (printk(): The Most Useful Tool is Now Showing its Age, sadly no video is available for the second talk) and read the slides (printk(): The Most Useful Tool is Now Showing its Age and printk: It’s Old, What Can We Do to Make It Young Again?).

More robust I2C designs with a new fault-injection driver – Wolfram Sang

Talk selected by Miquèl Raynal

More robust I2C designs with a new fault-injection driver – Wolfram SangAlthough Wolfram had a lot of troubles starting its presentation lacking a proper HDMI adaptater, he gave an illuminating talk about how, as an I2C subsystem maintainer, he would like to strengthen the robustness of I2C drivers.

He first explained some basics of the I2C bus like START and STOP conditions and introduced us to a few errors he regularly spots in drivers. For instance, some badly written drivers used a START and STOP sequence while a “repeated START” was needed. This is very bad because another master on the bus could, in this little spare idle delay, decide to grab the medium and send its message. Then the message right after the repeated START would not have the expected effect. Of course plenty other errors can happen: stalled bus (SDA or SCL stuck low), lost arbitration, faulty bits… All these situations are usually due to incorrect sequences sent by the driver.

To avoid so much pain debugging obscure situations where this happens, he decided to use an extended I2C-gpio interface to access SDA and SCL from two external GPIOs and this way forces faulty situations by simply pinning high or low one line (or both) and see how the driver reacts. The I2C specification and framework provide everything to get out of a faulty situation, it is just a matter of using it (sending a STOP condition, clocking 9 times, operate a reset, etc).

Wolfram is aware of his quite conservative approach but he is really scared about breaking users by using random quirks so he tried with this talk to explain his point of view and the solutions he wants to promote.

Two questions that you might have a hard time hearing were also interesting. The first person asked if he ever considered using a “default faulty chip” designed to do by itself this kind of fault injection and see how the host reacts and behaves. Wolfram said buying hardware is too much effort for debugging, so he was more motivated to get something very easy and straightforward to use. Someone else asked if he thought about multiple clients situation, but from Wolfram’s point of view, all clients are in the same state whether the bus is busy or free and should not badly behave if we clock 9 times.

Watch the video and grab the slides.

HDMI 4k Video: Lessons Learned – Hans Verkuil

Talk selected by Maxime Ripard

HDMI 4k Video: Lessons Learned – Hans VerkuilHaving worked recently on a number of display related drivers, it was quite natural to go see what I was probably going to work on in a quite close future.

Hans started by talking about HDMI in general, and the various signals that could go through it. He then went on with what was basically a war story about all the mistakes, gotchas and misconceptions that he encountered while working on a video-conference box for Cisco. He covered the hardware itself, but also more low-level oriented aspects, such as the clocks frequencies needed to operate properly, the various signals you could look at for debugging or the issues that might come with the associated encoding and / or color spaces, especially when you want to support as many displays as you can. He also pointed out the flaws in the specifications that might lead to implementation inconsistencies. He concluded with the flaws of various HDMI adapters, the issues that might arise using them on various OSes, and how to work around them when doable.

Watch the video and the slides.

The Serial Device Bus – Johan Hovold

Talk selected by Thomas Petazzoni

The Serial Device Bus – Johan HovoldJohan started his talk at ELCE by exposing the problem with how serial ports (UARTs) are currently handled in the Linux kernel. Serial ports are handled by the TTY layer, which allows user-space applications to send and receive data with what is connected on the other side of the UART. However, the kernel doesn’t provide a good mechanism to model the device that is connected at the other side of the UART, such as a Bluetooth chip. Due to this, people have resorted to either writing user-space drivers for such devices (which falls short when those devices need additional resources such as regulators, GPIOs, etc.) or to developing specific TTY line-discipline in the kernel. The latter also doesn’t work very well because a line discipline needs to be explicitly attached to a UART to operate, which requires a user-space program such as hciattach used in Bluetooth applications.

In order to address this problem, Johan picked up the work initially started by Rob Herring (Linaro), which consisted in writing a serial device bus (serdev in short), which consists in turning UART into a proper bus, with bus controllers (UART controllers) and devices connected to this bus, very much like other busses in the Linux kernel (I2C, SPI, etc.). serdev was initially merged in Linux 4.11, with some improvements being merged in follow-up versions. Johan then described in details how serdev works. First, there is a TTY port controller, which instead of registering the traditional TTY character device will register the serdev controller and its slave devices. Thanks to this, one can described in its Device Tree child nodes of the UART controller node, which will be probed as serdev slaves. There is then a serdev API to allow the implementation of serdev slave drivers, that can send and receive data of the UART. Already a few drivers are using this API: hci_serdev, hci_bcm, hci_ll, hci_nokia (Bluetooth) and qca_uart (Ethernet).

We found this talk very interesting, as it clearly explained what is the use case for serdev and how it works, and it should become a very useful subsystem for many embedded applications that use UART-connected devices.

Watch the video and the slides.

GStreamer for Tiny Devices – Olivier Crête

Talk selected by Grégory Clement

GStreamer for Tiny Devices – Olivier CrêteThe purpose of this talk was to show how to shrink Gstreamer to make it fit in an embedded Linux device. First, Olivier Crête introduced what GStreamer is, it was very high level but well done. Then after presenting the issue, he showed step by step how he managed to reduce the footprint of a GStreamer application to fit in his device.

In a first part it was a focus on features specific to GStreamer such as how to generate only the needed plugins. Then most of the tricks showed could be used for any C or C++ application. The talk was pretty short so there was no useless or boring part Moreover, the speaker himself was good and dynamic.

To conclude it was a very pleasant talk teaching step by step how to reduce the footprint of an application being GStreamer or not.

Watch the video and the slides.

Bootlin at Embedded and Kernel Recipes 2017

Kernel RecipesEmbedded RecipesThe Kernel Recipes conference has become over the last few years a very interesting conference, with an original single track format and a limited number of attendees, which fosters communication and networking. Held in Paris, France, it is obviously a conference of choice for Bootlin engineers to attend and speak at. We participated to multiple editions, Bootlin engineer Maxime Ripard gave a talk at the 2014 edition, while Thomas Petazzoni gave a talk at the 2013 edition.

In 2017, the organizers decided to complement the 3-day Kernel Recipes conference with a 1-day Embedded Recipes event, and Bootlin will participate by having two engineers attend those events and give talks:

If you’re interested in attending one of those events, make sure to register on time, there are only 100 seats available!

Bootlin at the Linux Plumbers 2017 conference

The Linux Plumbers conference has established itself as a major conference in the Linux ecosystem, discussing numerous aspects of the low-level layers of the Linux software stack. Linux Plumbers is organized around a number of micro-conferences, plus a number of more regular talks.

Linux Plumbers 2017

Bootlin already participated to several previous editions of Linux Plumbers, and will again participate to this year’s edition that takes place in Los Angeles on September 13-15. Bootlin engineers Boris Brezillon, Alexandre Belloni, Grégory Clement and Thomas Petazzoni will attend the conference.

If you’re attending this conference, or are located in the Los Angeles area, and want to meet us, do not hesitate to drop us a line at info@bootlin.com. You can also follow Bootlin Twitter feed for updates during the conference.

Feedback from the Netdev 2.1 conference

At Bootlin, we regularly work on networking topics as part of our Linux kernel contributions and thus we decided to attend our very first Netdev conference this year in Montreal. With the recent evolution of the network subsystem and its drivers capabilities, the conference was a very good opportunity to stay up-to-date, thanks to lots of interesting sessions.

Eric Dumazet presenting “Busypolling next generation”

The speakers and the Netdev committee did an impressive job by offering such a great schedule and the recorded talks are already available on the Netdev Youtube channel. We particularly liked a few of those talks.

Distributed Switch Architecture – slidesvideo

Andrew Lunn, Viven Didelot and Florian Fainelli presented DSA, the Distributed Switch Architecture, by giving an overview of what DSA is and by then presenting its design. They completed their talk by discussing the future of this subsystem.

DSA in one slide

The goal of the DSA subsystem is to support Ethernet switches connected to the CPU through an Ethernet controller. The distributed part comes from the possibility to have multiple switches connected together through dedicated ports. DSA was introduced nearly 10 years ago but was mostly quiet and only recently came back to life thanks to contributions made by the authors of this talk, its maintainers.

The main idea of DSA is to reuse the available internal representations and tools to describe and configure the switches. Ports are represented as Linux network interfaces to allow the userspace to configure them using common tools, the Linux bridging concept is used for interface bridging and the Linux bonding concept for port trunks. A switch handled by DSA is not seen as a special device with its own control interface but rather as an hardware accelerator for specific networking capabilities.

DSA has its own data plane where the switch ports are slave interfaces and the Ethernet controller connected to the SoC a master one. Tagging protocols are used to direct the frames to a specific port when coming from the SoC, as well as when received by the switch. For example, the RX path has an extra check after netif_receive_skb() so that if DSA is used, the frame can be tagged and reinjected into the network stack RX flow.

Finally, they talked about the relationship between DSA and Switchdev, and cross-chip configuration for interconnected switches. They also exposed the upcoming changes in DSA as well as long term goals.

Memory bottlenecks – slides

As part of the network performances workshop, Jesper Dangaard Brouer presented memory bottlenecks in the allocators caused by specific network workloads, and how to deal with them. The SLAB/SLUB baseline performances are found to be too slow, particularly when using XDP. A way from a driver to solve this issue is to implement a custom page recycling mechanism and that’s what all high-speed drivers do. He then displayed some data to show why this mechanism is needed when targeting the 10G network budget.

Jesper is working on a generic solution called page pool and sent a first RFC at the end of 2016. As mentioned in the cover letter, it’s still not ready for inclusion and was only sent for early reviews. He also made a small overview of his implementation.

DDOS countermeasures with XDP – slides #1slides #2 – video #1video #2

These two talks were given by Gilberto Bertin from Cloudflare and Martin Lau from Facebook. While they were not talking about device driver implementation or improvements in the network stack directly related to what we do at Bootlin, it was nice to see how XDP is used in production.

XDP, the eXpress Data Path, provides a programmable data path at the lowest point of the network stack by processing RX packets directly out of the drivers’ RX ring queues. It’s quite new and is an answer to lots of userspace based solutions such as DPDK. Gilberto andMartin showed excellent results, confirming the usefulness of XDP.

From a driver point of view, some changes are required to support it. RX hooks must be added as well as some API changes and the driver’s memory model often needs to be updated. So far, in v4.10, only a few drivers are supporting XDP.

XDP MythBusters – slides – video

David S. Miller, the maintainer of the Linux networking stack and drivers, did an interesting keynote about XDP and eBPF. The eXpress Data Path clearly was the hot topic of this Netdev 2.1 conference with lots of talks related to the concept and David did a good overview of what XDP is, its purposes, advantages and limitations. He also quickly covered eBPF, the extended Berkeley Packet Filters, which is used in XDP to filter packets.

This presentation was a comprehensive introduction to the concepts introduced by XDP and its different use cases.

Conclusion

Netdev 2.1 was an excellent experience for us. The conference was well organized, the single track format allowed us to see every session on the schedule, and meeting with attendees and speakers was easy. The content was highly technical and an excellent opportunity to stay up-to-date with the latest changes of the networking subsystem in the kernel. The conference hosted both talks about in-kernel topics and their use in userspace, which we think is a very good approach to not focus only on the kernel side but also to be aware of the users needs and their use cases.