Bootlin at the Embedded Linux Conference 2017

Last month, five engineers from Bootlin participated to the Embedded Linux Conference in Portlan, Oregon. It was once again a great conference to learn new things about embedded Linux and the Linux kernel, and to meet developers from the open-source community.

Bootlin team at work at ELC 2017, with Maxime Ripard, Antoine Ténart, Mylène Josserand and Quentin Schulz

Bootlin talks

Bootlin CEO Michael Opdenacker gave a talk on Embedded Linux Size Reduction techniques, for which the slides and video are available:

Bootlin engineer Quentin Schulz gave a talk on Power Management Integrated Circuits: Keep the Power in Your Hands, the slides and video are also available:

Bootlin selection of talks

Of course, the slides from many other talks are progressively being uploaded, and the Linux Foundation published the video recordings in a record time: they are all already available on Youtube!

Below, each Bootlin engineer who attended the conference has selected one talk he/she has liked, and gives a quick summary of the talk, hopefully to encourage you watch the corresponding video recording.

Using SWupdate to Upgrade your system, Gabriel Huau

Talk selected by Mylène Josserand.

Gabriel Huau from Witekio did a great talk at ELC about SWUpdate, a tool created by Denx to update your system. The talk gives an overview of this tool, how it is working and how to use it. Updating your system is very important for embedded devices to fix some bugs/security fixes or add new features, but in an industrial context, it is sometimes difficult to perform an update: devices not easily accessible, large number of devices and variants, etc. A tool that can update the system automatically or even Over The Air (OTA) can be very useful. SWUpdate is one of them.

SWUpdate allows to update different parts of an embedded system such as the bootloader, the kernel, the device tree, the root file system and also the application data.
It handles different image types: UBI, MTD, Raw, Custom LUA, u-boot environment and even your custom one. It includes a notifier to be able to receive feedback about the updating process which can be useful in some cases. SWUPdate uses different local and OTA/remote interfaces such as USB, SD card, HTTP, etc. It is based on a simple update image format to indicate which images must be updated.

Many customizations can be done with this tool as it is provided with the classic menuconfig configuration tool. One great thing is that this tool is supported by Yocto Project and Buildroot so it can be easily tested.

Do not hesitate to have a look to his slides, the video of his talk or directly test SWUpdate!

GCC/Clang Optimizations for embedded Linux, Khem Raj

Talk selected by Michael Opdenacker.

Khem Raj from Comcast is a frequent speaker at the Embedded Linux Conference, and one of his strong fields of expertise is C compilers, especially LLVM/Clang and Gcc. His talk at this conference can interest anyone developing code in the C language, to know about optimizations that the compilers can use to improve the performance or size of generated binaries. See the video and slides.

Khem Raj slide about compiler optimization optionsOne noteworthy optimization is Clang’s -Oz (Gcc doesn’t have it), which goes even beyond -Os, by disabling loop vectorization. Note that Clang already performs better than Gcc in terms of code size (according to our own measurements). On the topic of bundle optimizations such as -O2 or -Os, Khem added that specific optimizations can be disabled in both compilers through the -fno- command line option preceding the name of a given optimization. The name of each optimization in a given bundle can be found through the -fverbose-asm command line option.

Another new optimization option is -Og, which is different from the traditional -g option. It still allows to produce code that can be debugged, but in a way that provides a reasonable level of runtime performance.

On the performance side, he also recalled the Feedback-Directed Optimizations (FDO), already covered in earlier Embedded Linux Conferences, which can be used to feed the compiler with profiler statistics about code branches. The compiler can use such information to optimize branches which are the more frequent at run-time.

Khem’s last advise was not to optimize too early, and first make sure you do your debugging and profiling work first, as heavily optimized code can be very difficult to debug. Therefore, optimizations are for well-proven code only.

Note that Khem also gave a similar talk in the IoT track for the conference, which was more focused on bare-metal code optimization code and portability: “Optimizing C for microcontrollers” (slides, video).

A Journey through Upstream Atomic KMS to Achieve DP Compliance, Manasi Navare

Talk selected by Quentin Schulz.

This talk was about the journey of a new comer in the mainline kernel community to fix the DisplayPort support in Intel i915 DRM driver. It first presented what happens from the moment we plug a cable in a monitor until we actually see an image, then where the driver is in the kernel: in the DRM subsystem, between the hardware (an Intel Integrated Graphics device) and the libdrm userspace library on which userspace applications such as the X server rely.

The bug to fix was that case when the driver would fail after updating to the requested resolution for a DP link. The other existing drivers usually fail before updating the resolution, so Manasi had to add a way to tell the userspace the DP link failed after updating the resolution. Such addition would be useless without applications using this new information, therefore she had to work with their developers to make the applications behave correctly when reading this important information.

With a working set of patches, she thought she had done most of the work with only the upstreaming left and didn’t know it would take her many versions to make it upstream. She wished to have sent a first version of a driver for review earlier to save time over the whole development plus upstreaming process. She also had to make sure the changes in the userspace applications will be ready when the driver will be upstreamed.

The talk was a good introduction on how DisplayPort works and an excellent example on why involving the community even in early stages of the development process may be a good idea to quicken the overall driver development process by avoiding complete rewriting of some code parts when upstreaming is under way.

See also the video and slides of the talk.

Timekeeping in the Linux Kernel, Stephen Boyd

Talk selected by Maxime Ripard.

Stephen did a great talk about one thing that is often overlooked, and really shouldn’t: Timekeeping. He started by explaining the various timekeeping mechanisms, both in hardware and how Linux use them. That meant covering the counters, timers, the tick, the jiffies, and the various POSIX clocks, and detailing the various frameworks using them. He also explained the various bugs that might be encountered when having a too naive counter implementation for example, or using the wrong POSIX clock from an application.

See also the video and slides of the talk.

Android Things, Karim Yaghmour

Talk selected by Antoine Ténart

Karim did a very good introduction to Android Things. His talk was a great overview of what this new OS from Google targeting embedded devices is, and where it comes from. He started by showing the history of Android, and he explained what this system brought to the embedded market. He then switched to the birth of Android Things; a reboot of Google’s strategy for connected devices. He finally gave an in depth explanation of the internals of this new OS, by comparing Android Things and Android, with lots of examples and demos.

Android Things replaces Brillo / Weave, and unlike its predecessor is built reusing available tools and services. It’s in fact a lightweight version of Android, with many services removed and a few additions like the PIO API to drive GPIO, I2C, PWM or UART controllers. A few services were replaced as well, most notably the launcher. The result is a not so big, but not so small, system that can run on headless devices to control various sensors; with an Android API for application developers.

See also the video and slides of the talk.

Bootlin at the X.org Developer Conference 2016

The X.org Foundation hosts every year around september the X.org Developer Conference, which, unlike its name states, is not limited to X.org developers, but gathers all the Linux graphics stack developers, including X.org, Mesa, wayland, and other graphics stacks like ChromeOS, Android or Tizen.

This year’s edition was held last week in the University of Haaga-Helia, in Helsinki. At Bootlin, we’ve had more and more developments on the graphic stack recently through the work we do on Atmel and NextThing Co’s C.H.I.P., so it made sense to attend.

XDC 2016 conference

There’s been a lot of very interesting talks during those three days, as can be seen in the conference schedule, but we especially liked a few of those:

DRM HWComposer – SlidesVideo

The opening talk was made by two Google engineers from the ChromeOS team, Sean Paul and Zach Reizner. They talked about the work they did on the drm_hwcomposer they wrote for the Pixel C, on Android.

The hwcomposer is one of the HAL in Android that interfaces between Surface Flinger, the display manager, and the underlying display driver. It aims at providing hardware composition features, so that Android can leverage the capacities of the display engine to perform compositions (through planes and sprites), without having to use the CPU or the GPU to do this work.

The drm_hwcomposer started out as yet another hwcomposer library implementation for the tegra-drm driver in Linux. While they implemented it, it turned into some generic enough implementation that should be useful for all the DRM drivers out there, and they even introduced some particularly nice features, to split the final screen content into several planes based on the actual displayed content rather than on windows like it’s usually done.

Their work also helped to point out a few flaws in the hwcomposer API, that will eventually be fixed in a new revision of that API.

ARC++ SlidesVideo

The next talk was once again from a ChromeOS engineer, David Reveman, who came to show his work on ARC++, the component in ChromeOS that allows to run Android applications. He was obviously mostly talking about the display side.

In order to achieve that, he had to implement an hwcomposer that would just act as a proxy between SurfaceFlinger and Wayland that is used on the ChromeOS side. The GL rendering is still direct though, and each Android application will talk directly to the GPU, as usual. Only the composition will be forwarded to the ChromeOS side.

In order to minimize that composition process, whenever possible, ARC++ tries to back each application with an overlay so that the composition would happen directly in hardware.

This also led to some interesting challenges, especially since some of the assumptions of both systems are in contradiction. For example, any application can be resized in ChromeOS, while it’s not really a thing in Android where all the applications run full screen.

HDR Displays in Linux – SlidesVideo

The next talk we found interesting was Andy Ritger from nVidia explaining how the HDR displays were supposed to be handled in Linux.

He first started by explaining what HDR is exactly. While the HDR is just about having a wider range of luminance than on a regular display, you often also get a wider gamut with HDR capable displays. This means that on those screens you can display a wider range of colors, and with a better range and precision in their intensity. And
while the applications have been able to generate HDR content for more than 10 years, the rest of the display stack wasn’t really ready, meaning that you had convert the HDR colors to colors that your monitor was able to display, using a technique called tone mapping.

He then explained than the standard, non-HDR colorspace, sRGB, is not a linear colorspace. This means than by doubling the encoded luminance of a color, you will not get a color twice brighter on your display. This was meant this way because the human eye is much more sensitive to the various shades of colors when they are dark than when they are bright. Which essentially means that the darker the color is, the more precision you want to get.

However, the luminance “resolution” on the HDR display is so good that you actually don’t need that anymore, and you can have a linear colorspace, which is in our case SCRGB.

But drawing blindly in all your applications in SCRGB is obviously not a good solution either. You have to make sure that your screen supports it (which is exposed through its EDIDs), but also that you actually tell your screeen to switch to it (through the infoframes). And that requires some support in the kernel drivers.

The Anatomy of a Vulkan Driver – SlidesVideo

This talk by Jason Ekstrand was some kind of a war story of the bring up Intel did of a Vulkan implementation on their GPU.

He first started by saying that it was actually a not so long project, especially when you consider that they wrote it from scratch, since it took roughly 3 full-time engineers 8 months to come up with a fully compliant and open source stack.

He then explained why Vulkan was needed. While OpenGL did amazingly well to cope with the hardware evolutions, it was still designed over 20 years ago, This proved to have some core characteristics that are not really relevant any more, and are holding the application developers back. For example, he mentioned that at its core, OpenGL is based on a singleton-based state machine, that obviously doesn’t scale well anymore on our SMP systems. He also mentioned that it was too abstracted, and people just wanted a lower level API, or that you might want to render things off screen without X or any context.

This was fixed in Vulkan by effectively removing the state machine, which allows it to scale, push things like the error checking or the synchronization directly to the applications, making the implementation much simpler and less layered which also simplifies the development and debugging.

He then went on to discuss how we could share the code that was still shared between the two implementations, like implementing OpenGL on top of Vulkan (which was discarded), having some kind of lighter intermediate language in Mesa to replace Gallium or just sharing through a library the common bits and making both the OpenGL and Vulkan libraries use that.

Motivating preemptive GPU scheduling for real-time systems – SlidesVideo

The last talk that we want to mention is the talk on preemptive scheduling by Roy Spliet, from the University of Cambridge.

More and more industries, and especially the automotive industry, offload some computations to the GPU for example to implement computer vision. This is then used in a car to implement the autonomous driving to make the car recognize signs or stay in its lane. And obviously, this kind of computations are supposed to be handled in a real time
system, since you probably don’t want your shiny user interface for the heating to make your car crash in the car before it because its rendering was taking too long.

He first started to explain what real time means, and what the usual metrics are, which should to no surprise to people used to “CPU based” real time systems: latency, deadline, execution time, and so on.

He then showed a bunch of benchmarks he used to test his preemptive scheduler, in a workload that was basically running OpenArena while running some computations, on various nouveau based platforms (both desktop-grade GPUs, and embedded SoCs).

This led to some expected conclusions, like the fact that a preemptive scheduler is indeed adding some overhead, but is on average worth it, while some have been quite interesting. He was for example observing some worst case latencies that were quite rare (0.3%), but were actually interferences from the display engine filling up its empty FIFOs, and creating some contention on the memory bus.

Conclusion

Overall, this has been a great experience. The organisation was flawless, and the one-track-only format allows you to meet easily both the speakers and attendees. The content was also highly technical, as you might expect, which made us learn a lot and led us to think about some interesting developments we could do on our various projects in the future, such as NextThing Co’s CHIP.

Bootlin at the X Developer Conference

The next X.org Developer Conference will take place on September 21 to September 23 in Helsinki, Finland. This is a major event for Linux developers working in the graphics/display areas, not only at the X.org level, but also at the kernel level, in Mesa, and other related projects.

Bootlin engineer Maxime Ripard will be attending this conference, with 80+ other engineers from Intel, Google, NVidia, Texas Instruments, AMD, RedHat, etc.

Maxime is the author of the DRM/KMS driver in the upstream Linux kernel for the Allwinner SoCs, which provides display support for numerous Allwinner platforms, especially Nextthing’s CHIP (with parallel LCD support, HDMI support, VGA support and composite video support). Maxime has also worked on making the 3D acceleration work on this platform with a mainline kernel, by adapting the Mali kernel driver. Most recently, Maxime has been involved in Video4Linux development, writing a driver for the camera interface of Allwinner SoCs, and supervising Florent Revest work on the Allwinner VPU that we published a few days ago.

Bootlin at the Embedded Linux Conference Europe

The next Embedded Linux Conference Europe will take place from October 11 to October 13 in Berlin, Germany. As usual, the entire Bootlin engineering team will participate, which means this time 10 participants from Bootlin!

Embedded Linux Conference Europe 2016

The schedule for the conference has been published recently, and a number of our talk proposals have been accepted, so we will present on the following topics:

Like every year, we’re looking forward to attending this conference, and meeting all the nice folks of the Embedded Linux community!

“Understanding D-Bus” talk at the Toulouse Embedded Linux Meetup

A few months ago, in May, Bootlin engineer Mylène Josserand presented a talk titled Understanding D-Bus at the Toulouse Embedded Linux and Android meetup.

In this talk, Mylène shared her experience working with D-Bus, especially in conjunction with the OFono and Connman projects, to support modem and 3G connections on embedded Linux systems.

Understanding D-Bus

We are now publishing the slides of Mylène’s talk, they are available in PDF format.

Slides from Collaboration Summit talk on Linux kernel upstreaming

As we announced in a previous blog post, Bootlin CTO Thomas Petazzoni gave a talk at the Collaboration Summit 2016 covering the topic of “Upstreaming hardware support in the Linux kernel: why and how?“.

The slides of the talk are now available in PDF format.

Upstreaming hardware support in the Linux kernel: why and how?

Upstreaming hardware support in the Linux kernel: why and how?

Upstreaming hardware support in the Linux kernel: why and how?

Through this talk, we identified a number of major reasons that should encourage hardware vendors to contribute the support for their hardware to the upstream Linux kernel, and some hints on how to achieve that. Of course, within a 25 minutes time slot, it was not possible to get into the details, but hopefully the general hints we have shared, based on our significant Linux kernel upstreaming experience, have been useful for the audience.

Unfortunately, none of the talks at the Collaboration Summit were recorded, so no video will be available for this talk.

Bootlin at the Embedded Linux Conference 2016

Like every year for about 10 years, the entire Bootlin engineering team will participate to the next Embedded Linux Conference, taking place on April 4-6 in San Diego, California. For us, participating to such conferences is very important, as it allows to remain up to date with the latest developments in the embedded Linux world, create contacts with other members of the embedded Linux community, and meet the community members we already know and work with on a daily basis via the mailing lists or IRC.

Embedded Linux Conference 2016

Over the years, our engineering team has grown, and with the arrival of two more engineers on March 14, our engineering team now gathers 9 persons, all of whom are going to participate to the Embedded Linux Conference.

As usual, in addition to attending, we also proposed a number of talks, and some of them have been accepted and are visible in the conference schedule:

As usual, our talks are centered around our areas of expertise: hardware support in the Linux kernel, especially for ARM platforms, and build system related topics (Buildroot, Yocto, autotools).

We are looking forward to attending this event, and see many other talks from various speakers: the proposed schedule contains a wide range of topics, many of which look really interesting!

Bootlin speaking at the Linux Collaboration Summit

Bootlin engineers are regular speakers at the Embedded Linux Conference and Embedded Linux Conference Europe events from the Linux Foundation, to which our entire engineering team participates each year.

In 2016, for the first time, we will also be speaking at the Collaboration Summit, an invitation-only event where, as the Linux Foundation presents it, “the world’s thought leaders in open source software and collaborative development convene to share best practices and learn how to manage the largest shared technology investments of our time”.

Collaboration Summit 2016

This event will take place on March 29-31 in Lake Tahoe, California, and the event schedule has been published recently. Bootlin CTO Thomas Petazzoni will be giving a talk Upstreaming hardware support in the Linux kernel: why and how?, during which we will share our experience working with HW manufacturers to bring the support for their hardware to the upstream Linux kernel, discuss the benefits of upstreaming, and best practices to work with upstream.

With a small team of engineers, Bootlin has merged over the last few years thousands of patches in the official Linux kernel, and has several of its engineers having maintainer positions in the Linux kernel community. We are happy to take the opportunity of the Collaboration Summit to share some of our experience, and hopefully encourage and help other companies to participate upstream.

2016 Q1 newsletter

Newsletter iconThe Bootlin team wishes you a Happy New Year for 2016, with many new bits to enjoy in your life!

Bootlin is happy to take this opportunity to share some news about the latest training and contribution activities of the company.

Bootlin work on the $9 computer

As announced in our previous newsletter, Bootlin has been working intensively on developing the low-level software support for the first $9 computer, the C.H.I.P by Next Thing Co.

Next Thing Co. has successfully delivered an initial batch of platforms in September to the early adopters, and has started shipping the final products in December to thousands of Kickstarter supporters.

Those products are using the U-Boot and Linux kernel ported by Bootlin engineers, with numerous patches submitted to the official projects and more to be submitted in the coming weeks and months:

  • Support for the C.H.I.P platform itself, in U-Boot and in the Linux kernel;
  • Support for audio on Allwinner platforms added to the Linux kernel;
  • Development of a DRM/KMS driver for the graphics controller found on Allwinner platforms;
  • Significant research effort on finding appropriate solutions to support Multi-Level Cell NANDs in the Linux kernel;
  • Enabling of the NAND storage in Single-Level Cell mode, until the Multi-Level Cell mode can be enabled reliably;
  • Addition of NAND support in the fastboot implementation of U-Boot, which is used to reflash the C.H.I.P.

We will continue to work on the C.H.I.P over the next months, with among other things more work on the graphics side and the NAND side.

Kernel contributions

The primary focus of the majority of our customer projects remain the Linux kernel, to which we continue to contribute very significantly.

Linux 4.2

We contributed 203 patches to this release, with a new IIO driver for the ADC found on Marvell Berlin platforms, a big cleanup to the support of Atmel platforms, improvements to the DMA controller driver for Atmel platforms, a completely new driver for the cryptographic accelerator found on Marvell EBU platforms.

In this cycle, our engineer Alexandre Belloni became the official maintainer of the RTC subsystem.

See details on our contributions to Linux 4.2

Linux 4.3

We contributed 110 patches to this release, with mainly improvements to the DRM/KMS driver and DMA controller driver for Atmel platforms and power management improvements for Marvell platforms.

See details on our contributions to Linux 4.3

Linux 4.4

We contributed 112 patches to this release, the main highlights being an additional RTC driver, a PWM driver, support for the C.H.I.P platform, and improvements to the NAND support.

See details on our contributions to Linux 4.4

Work on ARM 64-bit platform

We have started to work on supporting the Linux kernel on several ARM 64 bits platforms from different vendors. We will be submitting the initial patches in the coming weeks and will progressively improve the support for those platforms throughout 2016 where a major part of our Linux kernel contribution effort will shift to ARM 64-bit.

Growing engineering team

Our engineering team, currently composed of six engineers, will be significantly expanded in 2016:

  • Two additional embedded Linux engineers will join us in March 2016 and will be working with our engineering team in Toulouse, France. They will help us on our numerous Linux kernel and Linux BSP projects.
  • An engineering intern will join us starting early February, and will work on setting up a board farm to contribute to the kernelci.org automated testing effort. This will help us do more automated testing on the ARM platforms we work on.

Upcoming training sessions

We have public training sessions scheduled for the beginning of 2016:

Embedded Linux development training
February 29 – March 4, in English, in Avignon (France)
Embedded Linux kernel and driver development training
March 14-18, in English, in Avignon (France)
Android system development training
March 7-10, in English, in Toulouse (France)

We also offer the following training courses, on-site, anywhere in the world, upon request:

Contact us at training@bootlin.com for details.

Conferences

We participated to the Embedded Linux Conference Europe in Dublin in October 2015, and gave a number of talks:

In addition, our engineer Thomas Petazzoni was invited to the Linux Kernel Summit, an invitation-only conference for the kernel maintainers and developers. He participated to the three days event in Seoul, South Korea. See Bootlin at the Linux Kernel Summit 2015.

At the beginning of 2016, our entire engineering team will be attending the Embedded Linux Conference in San Diego (US), which means that no less than 9 engineers from Bootlin will be present at the conference!

Porting Linux on ARM seminar

In December 2015, we gave a half-day seminar entitled “Porting Linux on ARM” in Toulouse (France). The materials, in English, are now freely available on our web site.

ELCE 2015 conference videos available

ELC Europe 2015 logoAs often in the recent years, the Linux Foundation has shot videos of most of the talks at the Embedded Linux Conference Europe 2015, in Dublin last October.

These videos are now available on YouTube, and individual links are provided on the elinux.org wiki page that keeps track of presentation materials as well. You can also find them all through the Embedded Linux Conference Europe 2015 playlist on YouTube.

All this is of course a priceless addition to the on-line slides. We hope these talks will incite you to participate to the next editions of the Embedded Linux Conference, like in San Diego in April, or in Berlin in October this year.

In particular, here are the videos from the presentations from Bootlin engineers.

Alexandre Belloni, Supporting multi-function devices in the Linux kernel

Kernel maintainership: an oral tradition

Tutorial: learning the basics of Buildroot

Our CTO Thomas Petazzoni also gave a keynote (Linux kernel SoC mainlining: Some success factors), which was well attended. Unfortunately, like for some of the other keynotes, no video is available.