Large Page Support for NAS systems on 32 bit ARM

The need for large page support on 32 bit ARM

Storage space has become more and more affordable to a point that it is now possible to have multiple hard drives of dozens of terabytes in a single consumer-grade device. With a few 10 TiB hard drives and thanks to RAID technology, storage capacities that exceed 16 or 32 TiB can easily be reached and at a relatively low cost.

However, a number of consumer NAS systems used in the field today are still based on 32 bit ARM processors. The problem is that, with Linux on a 32 bit system, it’s only possible to address up to 16 TiB of storage space. This is still true even with the ext4 filesystem, even though it uses 64 bit pointers.

We were lucky to have a customer contracting us to update older Large Page Support patches to a recent version of the Linux kernel. This set of patches are one way of overcoming this 16 TiB limitation for ARM 32-bit systems. Since updating this patch series was a non trivial task, we are happy to share the results of our efforts with the community, both through this blog post and through a patch series we posted to the Linux ARM kernel mailing list: ARM: Add support for large kernel page (from 8K to 64K).

How Large Page Support works

The 16 TiB limitation comes from the use of page->index which is a pgoff_t offset type corresponding to unsigned long. This limits us to a 32-bit page offsets, so with 4 KiB physical pages, we end up with a maximum of 16 TiB. A way to address this limitation is to use larger physical pages. We can reach 32 TiB with 8 KiB pages, 64 TiB with 16 KiB pages and up to 256 TiB with 64 KiB pages.

Before going further, the ARM32 Page Tables article from Linus Walleij is a good reference to understand how the Linux kernel deals with ARM32 page tables. In our case, we are only going to cover the non LPAE case. As explained there, the way the Linux kernel sees the page tables actually doesn’t match reality. First, the kernel deals with 4 levels of page tables while on hardware there are only 2 levels. In addition, while the ARM32 hardware stores only 256 PTEs in Page Tables, taking up only 1 KB, Linux optimizes things by storing in each 4 KB page two sets of 256 PTEs, and two sets of shadow PTEs that are used to store additional metadata needed by Linux about each page (such as the dirty and accessed/young bits). So, there is already some magic between what is presented to the Linux virtual memory management subsystem, and what is really programmed into the hardware page tables. To support large pages, the idea is to go further in this direction by emulating larger physical pages.

Our series (and especially patch 5: ARM: Add large kernel page support) proposes to pretend to have larger hardware pages. The ARM 32-bit architecture only supports 4 KiB or 64 KiB page sizes, but we would like to support intermediate values of 8 KiB, 16 KiB and 32 KiB as well. So what we do to support 8 KiB pages is that we tell Linux the hardware has 8 KiB pages, but in fact we simply use two consecutive 4 KiB pages at the hardware level that we manipulate and configure simultaneously. To support 16 KiB pages, we use 4 consecutive 4 KiB pages, for 32 KiB pages, we use 8 consecutive pages, etc. So really, we “emulate” having larger page sizes by grouping 2, 4 or 8 pages together. Adding this feature only required a few changes in the code, mainly dealing with ranges of pages every time we were dealing with a single page. Actually, most of the code in the series is about making it possible to modify the hard coded value of the hardware page size and fixing the assumptions associated to such a fixed value.

In addition to this emulated mechanism that we provide for 8 KiB, 16 KiB, 32 KiB and 64 KiB pages, we also added support for using real hardware 64 KiB pages as part of this patch series.

Overall the number of changes is very limited (271 lines added, 13 lines removed), and allows to use much larger storage devices. Here is the diffstat of the full patch series:

 arch/arm/include/asm/elf.h                  |  2 +-
 arch/arm/include/asm/fixmap.h               |  3 +-
 arch/arm/include/asm/page.h                 | 12 ++++
 arch/arm/include/asm/pgtable-2level-hwdef.h |  8 +++
 arch/arm/include/asm/pgtable-2level.h       |  6 +-
 arch/arm/include/asm/pgtable.h              |  4 ++
 arch/arm/include/asm/shmparam.h             |  4 ++
 arch/arm/include/asm/tlbflush.h             | 21 +++++-
 arch/arm/kernel/entry-common.S              | 13 ++++
 arch/arm/kernel/traps.c                     | 10 +++
 arch/arm/mm/Kconfig                         | 72 +++++++++++++++++++++
 arch/arm/mm/fault.c                         | 19 ++++++
 arch/arm/mm/mmu.c                           | 22 ++++++-
 arch/arm/mm/pgd.c                           |  2 +
 arch/arm/mm/proc-v7-2level.S                | 72 ++++++++++++++++++++-
 arch/arm/mm/tlb-v7.S                        | 14 +++-
 16 files changed, 271 insertions(+), 13 deletions(-)

This patch series is running in production now on some NAS devices from a very popular NAS brand.

Limitations and alternatives

The submission of our patch series is recent but this feature has actually been running for years on many NAS systems in the field. Our new series is based on the original patchset, with the purpose of submitting it to the mainline kernel community. However, there is little chance that it will ever be merged into the mainline kernel.

The main drawback of this approach are large pages themselves: as each file in the page cache uses at least one page, the memory wasted increases as the size of the pages increases. For this reason, Linus Torvalds was against similar series proposed in the past.

To show how much memory is wasted, Arnd Bergmann ran some numbers to measure the page cache overhead for a typical set of files (Linux 5.7 kernel sources) for 5 different page sizes:

Page size (KiB) 4 8 16 32 64
page cache usage (MiB) 1,023.26 1,209.54 1,628.39 2,557.31 4,550.88
factor over 4K pages 1.00x 1.18x 1.59x 2.50x 4.45x

We can see that while a factor of 1.18 is acceptable for 8 KiB pages, a 4.45 multiplier looks excessive with 64 KiB pages.

Actually, to make it possible to address large volumes on 32 bit ARM, another solution was pointed out during the review of our series. Instead of using larger pages which have an impact on the entire system, an alternative is to modify the way the filesystem addresses the memory by using 64 bits pgoff_t offsets. This has already been implemented in vendor kernels running in some NAS systems, but this has never been submitted to mainline developers.

Back from ELCE 2019: our selection of talks

The Embedded Linux Conference Europe edition 2019 took place a few weeks ago in Lyon, France, and no less than 7 engineers from Bootlin attended the conference. We would like to highlight a selection of talks that Bootlin engineers found interesting. We asked each of the 7 engineers who attended the event to pick one talk they liked, and make a small write-up about it. Of course, many other talks were interesting and what makes a talk interesting is very subjective!

Introduction to HyperBus Memory Devices, by Vignesh Raghavendra

Talk selected by Gregory Clement

Vignesh started his talk by presenting the HyperBus from the hardware point of view. It is a bus using 8 data lines, using either a single or a differential clock as well as a bi-directional data strobe. These last two features clearly indicate that the designers of this bus seek high bandwidth. Two types of memory are available: HyperRAM and HyperFlash and the talk focused on the second one.

The read throughput of the HyperFlash can reach 400MB/s, it is compatible with SPI flash and is an alternative to the octal SPI NOR flashes. Then Vignesh presented the transactions done on the bus for the flash, which is very similar to what is done on SPI bus. He also compares the traditional parallel CFI flashes to the HyperFlash. And finally he describes the 2 types of controllers: Dedicated HyperBus Controllers and Multi IO Serial controllers.

In the last part of the talk, Vignesh presented the recently add kernel features, and future improvements.

This talk was a good introduction to this new bus, covering the hardware parts as well as software support in the Linux kernel.




[PDF]

Open Source Graphics 101: Getting Started, by Boris Brezillon

Talk selected by Paul Kocialkowski

During this talk, Boris provides a comprehensive and accessible overview of the graphics stack that supports GPUs in systems based on the Linux kernel. He also provides insight about the inner workings and architecture of GPUs. Although Boris defines himself as a not (yet) experienced Graphics developer, his talk contains all the elements needed to get a clear first idea on the topic as he covers hardware, kernel and userspace aspects.

To begin with, he explains how the GPU pipeline is split into multiple stages that are needed one after the other to generate a final image from a set of 3D models. The first stage is geometry and involves operations on the vertices that compose the 3D models, followed by rasterization where the view of the 3D scene is materialized on a 2D viewport, producing the end image. He also presents the concept of shaders: they are small dedicated programs that run on the GPU to make each stage configurable and flexible in order to produce the exact wanted result.

He then provides details about how GPU architectures implement massive parallelization to provide efficient results and also details some of the pitfalls that can occur with this approach. After that, Boris presents how the main CPU interacts with the GPU, introducing the concept of a command stream to submit jobs to the GPU.

With all these concepts laid out comes the time for him to present how the software stack is organized to support GPUs. After a general overview, specific aspects are presented. This includes graphics APIs such as OpenGL and Vulkan (and how they follow distinct paradigms) but also covers topics related to Mesa internals such as intermediate representations or windowing system integration. Kernel aspects are not forgotten either and a rationale regarding the (unusual) kernel/userspace separation in place for GPUs is also provided to clarify prominent design choices.

This talk succeeds at providing an introduction to GPUs and 3D rendering that can be understood without specific graphics knowledge while also giving a good idea of how the supporting software stack is organized. It is highly recommended for anyone interested in learning more on these topics!




[PDF]

Learning the Linux Kernel Configuration Space: Results and Challenges, Matthieu Acher

Talk selected by Michael Opdenacker

TuxML (Linux and Machine Learning) is an open-source research project aiming at exploring the Linux kernel configuration space through machine learning. With more than 15,000 configuration parameters, the Linux kernel now has up to 106000 possible configurations. Compare this figure to 1080, the approximate number of atoms in the universe.

As it is not possible to test all such configurations (all the more as each takes about 10 minutes to compile), the goal of the project is to predict “interesting” configurations, that could expose distinct bugs.

Starting from random configurations (from make randconfig), they use statistical learning to eventually pinpoint sets of parameters causing build failures, and avoid testing configurations that are expected to fail. This way, TUXML can be more efficient in exploring the configuration space and find bugs.

That’s typically where researchers can help us engineers and Linux kernel contributors. You need a solid theoretical background in machine learning to process data efficiently.

On Linux 4.13, the research team has managed to explore more than 15,000 different configurations through more than 95,000 hours of computing, eventually to find (and fix) 16 bugs in Linux. Some of these bugs may not come as a surprise for experienced kernel developers, but some others could expose unexpected issues that a human user may not find spontaneously.

They are also trying to use their data to predict the impact of configuration parameters on kernel size, but it turns out that size is hard to predict. At least, they managed to find “influential” options, some expected ones, and some less expected ones, deserving further investigation.

This project looks definitely useful for improving the test coverage of the Linux kernel, by working smarter than trying to compile purely random configurations. Your help is needed for testing, investigating and fixing kernel bugs, and giving your feedback.




[PDF]

Going beyond printk messages, Sergio Prado

Talk selected by Miquèl Raynal

Sergio gave a talk about debugging with an interesting approach: he started by acknowledging that today, printk is very widely used to do serious debugging but that in some cases it would be much more efficient to use other tools. Indeed there are plenty of open source tools available out there so why don’t we use them?

He presented a table indicating, from his point of view, which tool would best fit a given problem and then enumerated a few tips and commands that everybody can use to understand what went wrong in their kernel, for instance after a panic.

Is addr2line the best way to avoid printk messages right after a panic? or maybe the Linux script faddr2line? or even GDB? Maybe you don’t have access to the panic trace yet, in this case you could be interested in looking at pstore or kdump?

Or maybe an issue will more efficiently be hunted with tracing, in this case Sergio shown several static and dynamic options: using tracepoints, kprobes, ftrace, and proposed many others.

Lock-ups and memory leaks are also covered in the slides (see below), but not in the video because unfortunately the 35 minutes slot allowed was not enough for Sergio to detail all these interesting debugging methods. We wish he had more time to give all his feedback around these underused -while powerful- tools!




[PDF]

Supporting Video (de)serializers in Linux: Challenges and Works in Progress

Talk selected by Thomas Petazzoni

Luca Ceresoli’s talk at this ELCE is a good example of an interesting talk, as it combines an introduction to new hardware, what is the status of the support for this hardware in Linux, and what are the challenges to overcome to complete the integration of the hardware support in the kernel, with some open discussion.

Luca’s talk was about the support for video serializers and deserializers, with a focus on camera support. Cameras are usually connected to a system-on-chip using a parallel interface, a MIPI CSI interface or some LVDS interface. However, these interfaces only work for very short distances between the system-on-chip and the camera, and may not work well in electromagnetically noisy environments. For such situations, there are some technical solutions that consists in serializing the camera stream in a fast robust link (typically a coax cable) and then deserialize it before it is captured by the SoC through a standard camera interface. This fast robust link of course transports the stream data itself, but can also transport control information (GPIO status, I2C bus to talk to the remote camera sensor).

There are two main technologies today implementing this: the GMSL technology from Maxim and the FDP-LinkIII technology from Texas Instruments. Luca’s focus is on the latter technology, since that’s what he has been working on for the past months.

After this hardware introduction, Luca gave a status of the different patch series that have been posted by various contributors (himself included) on the Linux kernel mailing lists: some preliminary support for GMSL has been posted by Kieran Bingham, and some preliminary support for FDP-LinkIII has been posted by Luca.

Luca then presented the ideal implementation to support these interfaces, but then quickly dived into the troubles and tribulations: there is no support for stream multiplexing in Video4Linux currently, there is no support for parts of a V4L pipeline going faulty, and there is no support for hotplugging in V4L.

Then, there are some challenges with how to handle the remote I2C bus offered by those serializers/deserializers. Since camera sensors often have the same I2C address, the serializers/deserializers often have some sort of “solution” to this: an I2C switch in the GMSL (de)serializers, and a translation table for I2C addresses in FDP-LinkIII (de)serializers. Luca discussed how these are currently supported in Linux.

At the end of the talk, quite a bit of discussion took place, both about the V4L issues and the I2C issues raised in Luca’s talk. Overall, it was a useful talk if you’re interested in this specific topic.




[PDF]

V4L2: A Status Update

Talk selected by Maxime Chevallier

As someone not very familiar with the V4L2 Framework, I was pleasantly surprised that Hans’ talk was done in such a way that both experienced and beginner developers were able to follow.

Hans started with a quick introduction to the various concepts of video encoding and decoding that needs to be understood to follow the highly technical explanations of the current status of stateless and stateful codecs support.

Besides describing the technical challenge of implementing such support, Hans gave a good overview of the challenges that are faced by the community, focusing on the necessity of having good testing tools, such as the new vicodec driver.

He described the complexity of implementing support for Stateless decoders (where the hardware decoder doesn’t keep track of the state, this has to be done in software), and explained that the new Request API is a good step towards achieving such support, with 2 decoders already supported in the staging area.

Hans then explained the userspace APIs that are to be used when dealing with Stateless decoders, starting some interesting discussions along the way.

All in all, such a talk is a good example of how we can use events such as ELCE to both give good technical insight on existing frameworks, but also to trigger discussions about the ongoing and future work amongst the active developers and maintainers that are brought together by the event.




[PDF]

One Build to Rule Them All: Building FreeRTOS & Linux Using Yocto

Talk selected by Alexandre Belloni

In this talk, Alejandro Hernandez explains how to build FreeRTOS using OpenEmbedded.

He starts by explaining the use case for building both FreeRTOS and a Linux system using the same build system, in this case OpenEmbedded.

He then shows the meta-freertos layer he developped to get OpenEmbedded to build FreeRTOS. The toolchain he used is fairly classic with GCC, binutils, gdb. The main difference is that newlib is used as the C library.
meta-freertos then depends on previous work that has been done, integrating a newlib and libgloss recipes in oe-core. The core of meta-freertos is then a class, freertos-app.bbclass, allowing to abstract many details allowing to build a FreeRTOS application and image for the target. A poky-freertos distribution configuration is also provided.

Alejandro then demoes multiple FreeRTOS applications.

Finally, he goes over multiconfig, the multiple configuration build dependencies, allowing OpenEmbedded to build an image using a configuration but depending on tasks using a different configuration. In other words, this allows to build a Linux system image after building a FreeRTOS application so it can be included in the image. This is very useful in the case Linux is running on the application processor and needs to load FreeRTOS on a smaller processor.




[PDF]

Authenticate and Encrypted Storage on Embedded Linux

Talk selected by Kamel Bouhara

Jan’s talk is introducing us to the current authentication and encryption methods that go on top of the Linux storage stack.

He started reminding us some basic crypto terminologies and then depicted all the existing technologies by the storage they fit into.

For block device storage dm-verity is a good choice to verify integrity of read-only filesystems and the verification is done on each node of a hash tree. For a file or application based verification fsverity is a more relevant tool as it allows on-demand verification.

On raw NAND devices, the integrity should be checked using the UBI filesystem associated to an HMAC or image signature authentication with a root key. This solution can be completed with fscrypt to encrypt specific data on the filesytem.

For the encryption stage, Jan mentioned the ecryptfs project, which is not maintained anymore and could be well replaced by fscrypt which allows to hold several keys in the same filesystem in a multi-user environment. It is therefore a good alternative to dm-crypt which is a block-based based encryption solution used on large block devices and it is not protected against replay attacks using old blocks.

For a TPM based authentication, he recommendeds using the kernel integrity subsystem called IMA/EVM, which is a layout on top of other filesystem, the project Keylime is good example for this: https://sched.co/TLCY.

Jan shared some good practices on how to manage the Master key storage like not using a password based key, if possible use hardware capabilities like ARM TrustZone and OPTEE and use of a verified boot and key wrapping for the master key.




[PDF]

Bootlin opens a new office in Lyon, France

After Toulouse and Orange, Lyon is the third city chosen for opening a Bootlin office. Since September 1st of this year (2017), Alexandre Belloni and Grégory Clement have been working more precisely in Oullins close to the subway and the train station. It is the first step to make the Lyon team grow, with the opportunity to welcome interns and engineers.


Their new desks are already crowded by many boards running our favorite system.

A Kickstarter for a low cost Marvell ARM64 board

At the beginning of October a Kickstarter campaign was launched to fund the development of a low-cost board based on one of the latest Marvell ARM 64-bit SoC: the Armada 3700. While being under $50, the board would allow using most of the Armada 3700 features:

  • Gigabit Ethernet
  • SATA
  • USB 3.0
  • miniPCIe

ESPRESSObin interfaces

The Kickstarter campaign was started by Globalscale Technologies, who has already produced numerous Marvell boards in the past: the Armada 370 based Mirabox, the Kirkwood based SheevaPlug, DreamPlug and more.

We pushed the initial support of this SoC to the mainline Linux kernel 6 months ago, and it landed in Linux 4.6. There are still a number of hardware features that are not yet supported in the mainline kernel, but we are actively working on it. As an example, support for the PCIe controller was merged in Linux 4.8, released last Sunday. According to the Kickstarter page the first boards would be delivered in January 2017 and by this time we hope to have managed to push more support for this SoC to the mainline Linux kernel.

We have been working on the mainline support of the Marvell SoC for 4 years and we are glad to see at last the first board under $50 using this SoC. We hope it will help expanding the open source community around this SoC family and will bring more contributions to the Marvell EBU SoCs.

Report on extensive real-time Linux benchmarks on AT91

The real time page I wrote for Atmel was finally released on the Linux4Sam Atmel Wiki. The purpose of this page was to help new comers to use real time features with Atmel CPUs and to present the state of the real time support.

Here are some figures associated to this work:

  • On this page I present the results of more than 300 hours of benchmarks!
  • During the setup and the tuning tests ran for more than 600 hours.
  • Analysis and formatting took a few dozen hours of work.
  • The benchmarks have been run on 3 boards, 3 flavors of Linux (vanilla, PREEMPT-RT patches, Xenomai co-kernel approach), and 2 kinds of tests (timer-based and GPIO-based)
  • Experiment with Yocto

    I recently had the opportunity to use Yocto. I already practiced quite a lot with OpenEmbedded before. You can see Yocto as a project derived from OpenEmbedded even it is a bit more than that.

    In fact, Yocto is made of Poky (a build system based on OpenEmbedded), a few added build tools (swabber, pseudo, etc.), as well as a set of meta data allowing to create embedded distributions for a number of targets.

    The strength but also the weakness of OpenEmbedded is that it a very flexible build system. It can make production root filesystems, but also a complete distribution with its ready to use package repository, and this for multiple hardware platforms. It makes it a difficult system to get started and get efficient with. Even two years ago, the OpenEmbedded documentation contributed to making it difficult to get started. Indeed, OpenEmbedded did supply some documentation, but which only started to make sense once you start mastering it. This is quite a paradox for a piece of documentation. It lacked the elements allowing developers to understand its operation well.

    With Yocto, I was pleased to realize that substantial progress had been made on this side. The project comes with documentation that is much more exhaustive and above all much more accessible for beginners. Getting started with it is still not completely straightforward, but this time, this is rather because of the complexity and the rich features of the tool.

    In a few hours, I managed to develop a minimalistic BSP (Board Support Package) for a given board (in this case a AT91SAM9G20-EK). The concept of layer allows to have a configuration layer specific to a given piece of hardware. You can even support multiple hardware platforms at once and add specific packages. A layer is indeed just a set of packages and configurations (or configuration overrides). The BSP is just a layer specific to one or several pieces of hardware.

    As you can see, even to support a simple embedded board, there is already a number of concepts to deal with. There are also multiple ways of achieving the same result but which will be easier or more difficult to maintain. The concept of “BSP” for Yocto is therefore a kind of guideline to allow the Yocto community to have a common point of reference. I will try to illustrate the use of a BSP on the AT91SAMG20-EK board here and/or on my Google+ page.

    Another significant progress is optimizing build time for a “minimalistic” target, which went down from more than three hours to just over one hour now. It remains a long time for a very simple target.

    To build a filesystem image with only a few components, Buildroot remains much more appropriate. For systems that require a great number of components, more advanced functionality is then often needed, such as managing a package repository and supporting multiple hardware platforms at once for example. In this case, Yocto remains the best (the only?) option, all the more as this project addresses the traditional weaknesses of OpenEmbedded.

    A new tool to ease kernel maintainer life

    When you are involved in mainlining or maintaining some kernel code, a non negligible part of your time is spent checking patches or patchsets themselves . It is not the most interesting part but it is truly necessary to help merging in kernel code, or to make sure you don’t break anything, for example building with an incompatible configuration.

    Aiaiai developed by Artem Bityutskiy is a tool to do most of this task for you! It uses other checking tools and scripts such as sparse, coccinelle and checkpatch.pl, and comes with its own set of tools and scripts. I don’t know what does “aiaiai” stands for, but in French it sounds like “Ouch Ouch Ouch”, the sound you could make if you forget to use this tool 😉

    PS: On my G+ post, Yegor Yefremov pointed that “aiaiai” means something like “tsk tsk!” (shame on you!) in Russian.

    Building a small Debian root filesystem with Multistrap

    There are several ways to build a root filesystem for an embedded Linux system: Buildroot and Open Embedded are the usual solutions to do this. They allow to fine tune the contents of your filesystem. The drawback is, in both cases, that you need to build everything from sources and this can take from tens of minutes to several hours.

    Sometimes you don’t need all this flexibility and you just want to have a ready-to-use root filesystem, to which you just add a few extra programs. In this case using a distribution is a good solution. So let’s see what we need:

    • A binary distribution
    • Available on several architectures
    • Ability to generate a “small” root filesystem
    • A large choice of packages

    Oh, I think it is a pretty good description of Debian!

    Emdebian is a project to adapt Debian to embedded devices. A good description from the Debian wiki is:

    “In short, what EmDebian does is wrap around the regular Debian package building tools to provide a more fine grained control over package selection, size, dependencies and content to enable creation of very small and efficient Debian packages for use on naturally resource limited embedded targets.”

    And so, pretty recently (2009), Emdebian released Multistrap which is similar to Debootstrap but more appropriate for embedded devices. It seems better by the way it builds a system:

    It works in a completely different way by simply using apt and dpkg, rather than avoiding to use them, which is how Debootstrap works.

    And also more appropriate by its goals:

    It is focused on producing rootfs images for devices, as opposed to chroots for existing machines

    Practical case: build a root filesystem for the USB A9263 board from Calao Systems (arm926ejs based board).

    A drawback of Multistrap is its limitation to Debian, but in fact it is also usable on any distribution based on Debian. In our case, we ran it on an Ubuntu 10.04 system.

    First, let’s install Multistrap:

    $sudo apt-get install multistrap dpkg-dev

    Multistrap needs a config file. For our needs we just use the example one given by Embedian. Let’s name it multistrap.conf:

    [General]
    noauth=true
    unpack=true
    debootstrap=Grip
    aptsources=Grip
    
    [Grip]
    # space separated package list
    source=http://www.emdebian.org/grip
    suite=lenny
    

    Grip is the name of the lightweight Debian distro built by Emdebian.

    Now we can run Multistrap:

    $ multistrap -a armel -d $PWD/RFS -f multistrap.conf
    em_multistrap 0.0.8 using multistrap.conf
    Using foreign architecture: armel
    em_multistrap building armel multistrap on 'amd64'
    INF: Setting ./lib64 -> ./lib symbolic link.
    Getting package lists: apt-get  -o Apt::Architecture=armel -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true -o Apt::Install-Recommends=false -o Dir=/home/mike/celf/multistrap/RFS/ -o Dir::Etc=/home/mike/celf/multistrap/RFS/etc/apt/ -o Dir::Etc::SourceList=/home/mike/celf/multistrap/RFS/etc/apt/sources.list.d/multistrap.sources.list -o Dir::State=/home/mike/celf/multistrap/RFS/var/lib/apt/ -o Dir::State::Status=/home/mike/celf/multistrap/RFS/var/lib/dpkg/status -o Dir::Cache=/home/mike/celf/multistrap/RFS/var/cache/apt/ update
    Get:1 http://www.emdebian.org lenny Release.gpg [197B]
    Ign http://www.emdebian.org/grip/ lenny/main Translation-en_US
    Get:2 http://www.emdebian.org lenny Release [21.4kB]
    Ign http://www.emdebian.org lenny Release
    Ign http://www.emdebian.org lenny/main Packages
    Ign http://www.emdebian.org lenny/main Sources
    Ign http://www.emdebian.org lenny/main Packages
    Ign http://www.emdebian.org lenny/main Sources
    Get:3 http://www.emdebian.org lenny/main Packages [293kB]
    Get:4 http://www.emdebian.org lenny/main Sources [351kB]
    Fetched 665kB in 0s (6,280kB/s)                     
    Reading package lists... Done
    W: GPG error: http://www.emdebian.org lenny Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B5B7720097BB3B58
    W: Duplicate sources.list entry http://www.emdebian.org/grip/ lenny/main Packages (/home/mike/celf/multistrap/RFS/var/lib/apt/lists/www.emdebian.org_grip_dists_lenny_main_binary-armel_Packages)
    Use of uninitialized value within %packages in join or string at /usr/sbin/em_multistrap line 294.
    Use of uninitialized value within %keyrings in join or string at /usr/sbin/em_multistrap line 296.
    apt-get -y  -o Apt::Architecture=armel -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true -o Apt::Install-Recommends=false -o Dir=/home/mike/celf/multistrap/RFS/ -o Dir::Etc=/home/mike/celf/multistrap/RFS/etc/apt/ -o Dir::Etc::SourceList=/home/mike/celf/multistrap/RFS/etc/apt/sources.list.d/multistrap.sources.list -o Dir::State=/home/mike/celf/multistrap/RFS/var/lib/apt/ -o Dir::State::Status=/home/mike/celf/multistrap/RFS/var/lib/dpkg/status -o Dir::Cache=/home/mike/celf/multistrap/RFS/var/cache/apt/ install balloon3-config base-files base-passwd bash bsdutils coreutils debianutils diff dpkg e2fslibs e2fsprogs findutils gcc-4.3-base grep grip-config gzip hostname initscripts libacl1 libattr1 libblkid1 libc6 libcap1 libcomerr2 libdevmapper1.02.1 libgcc1 liblocale-gettext-perl libncurses5 libpam-modules libpam-runtime libpam0g libselinux1 libsepol1 libslang2 libss2 libstdc++6 libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl libuuid1 libvolume-id0 login lsb-base makedev mawk mktemp mount ncurses-base ncurses-bin passwd perl-base procps sed sysv-rc sysvinit sysvinit-utils tar tzdata util-linux zlib1g
    Reading package lists... Done
    Building dependency tree... Done
    The following extra packages will be installed:
      apt debconf debconf-i18n debian-archive-keyring dhcp3-client dhcp3-common gnupg gpgv ifupdown libbz2-1.0 libdb4.6 libncursesw5 libnewt0.52 libpopt0
      libreadline5 libssl0.9.8 libusb-0.1-4 lzma module-init-tools nano net-tools netbase ntpdate readline-common udev wget whiptail
    The following NEW packages will be installed:
      apt balloon3-config base-files base-passwd bash bsdutils coreutils debconf debconf-i18n debian-archive-keyring debianutils dhcp3-client dhcp3-common diff
      dpkg e2fslibs e2fsprogs findutils gcc-4.3-base gnupg gpgv grep grip-config gzip hostname ifupdown initscripts libacl1 libattr1 libblkid1 libbz2-1.0 libc6
      libcap1 libcomerr2 libdb4.6 libdevmapper1.02.1 libgcc1 liblocale-gettext-perl libncurses5 libncursesw5 libnewt0.52 libpam-modules libpam-runtime libpam0g
      libpopt0 libreadline5 libselinux1 libsepol1 libslang2 libss2 libssl0.9.8 libstdc++6 libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl
      libusb-0.1-4 libuuid1 libvolume-id0 login lsb-base lzma makedev mawk mktemp module-init-tools mount nano ncurses-base ncurses-bin net-tools netbase
      ntpdate passwd perl-base procps readline-common sed sysv-rc sysvinit sysvinit-utils tar tzdata udev util-linux wget whiptail zlib1g
    0 upgraded, 87 newly installed, 0 to remove and 0 not upgraded.
    Need to get 15.4MB of archives.
    After this operation, 48.4MB of additional disk space will be used.
    WARNING: The following packages cannot be authenticated!
      libstdc++6 libbz2-1.0 readline-common libncurses5 libreadline5 libusb-0.1-4 zlib1g gpgv gnupg debian-archive-keyring apt debianutils dhcp3-common
      libattr1 libacl1 libselinux1 coreutils lzma dpkg perl-base liblocale-gettext-perl libtext-iconv-perl libtext-charwidth-perl libtext-wrapi18n-perl
      debconf-i18n debconf dhcp3-client sed ncurses-bin lsb-base module-init-tools libssl0.9.8 wget hostname net-tools ifupdown mawk libncursesw5 nano netbase
      libcap1 ntpdate libpam-runtime libpam0g libdb4.6 libpam-modules passwd libvolume-id0 udev libslang2 libnewt0.52 libpopt0 whiptail grip-config
      gcc-4.3-base libgcc1 libc6 base-passwd base-files bash diff libcomerr2 e2fslibs libuuid1 libblkid1 libss2 e2fsprogs findutils grep gzip login mktemp
      mount libsepol1 sysvinit-utils initscripts sysv-rc sysvinit tar tzdata util-linux balloon3-config bsdutils libdevmapper1.02.1 makedev ncurses-base procps
    Authentication warning overridden.
    Get:1 http://www.emdebian.org/grip/ lenny/main libstdc++6 4.3.2-1.1em1 [268kB]
    Get:2 http://www.emdebian.org/grip/ lenny/main libbz2-1.0 1.0.5-1em1 [37.2kB]
    Get:3 http://www.emdebian.org/grip/ lenny/main readline-common 5.2-3.1em1 [3,202B]
    Get:4 http://www.emdebian.org/grip/ lenny/main libncurses5 5.7+20081213-1em1 [165kB]
    Get:5 http://www.emdebian.org/grip/ lenny/main libreadline5 5.2-3.1em1 [108kB]
    Get:6 http://www.emdebian.org/grip/ lenny/main libusb-0.1-4 2:0.1.12-13em1 [13.7kB]
    Get:7 http://www.emdebian.org/grip/ lenny/main zlib1g 1:1.2.3.3.dfsg-12em1 [48.1kB]
    Get:8 http://www.emdebian.org/grip/ lenny/main gpgv 1.4.9-3+lenny1em1 [139kB]
    Get:9 http://www.emdebian.org/grip/ lenny/main gnupg 1.4.9-3+lenny1em1 [533kB]
    Get:10 http://www.emdebian.org/grip/ lenny/main debian-archive-keyring 2010.08.28~lenny1em1 [17.9kB]
    Get:11 http://www.emdebian.org/grip/ lenny/main apt 0.7.20.2+lenny2em1 [514kB]
    Get:12 http://www.emdebian.org/grip/ lenny/main debianutils 2.30em1 [23.4kB]
    Get:13 http://www.emdebian.org/grip/ lenny/main dhcp3-common 3.1.1-6+lenny4em1 [157kB]
    Get:14 http://www.emdebian.org/grip/ lenny/main libattr1 1:2.4.43-2em1 [7,706B]
    Get:15 http://www.emdebian.org/grip/ lenny/main libacl1 2.2.47-2em1 [14.0kB]
    Get:16 http://www.emdebian.org/grip/ lenny/main libselinux1 2.0.65-5em1 [50.0kB]
    Get:17 http://www.emdebian.org/grip/ lenny/main coreutils 6.10-6em1 [1,162kB]
    Get:18 http://www.emdebian.org/grip/ lenny/main lzma 4.43-14em1 [51.0kB]
    Get:19 http://www.emdebian.org/grip/ lenny/main dpkg 1.14.29em1 [405kB]
    Get:20 http://www.emdebian.org/grip/ lenny/main perl-base 5.10.0-19lenny2em1 [905kB]
    Get:21 http://www.emdebian.org/grip/ lenny/main liblocale-gettext-perl 1.05-4em1 [11.0kB]
    Get:22 http://www.emdebian.org/grip/ lenny/main libtext-iconv-perl 1.7-1+b1em1 [11.2kB]
    Get:23 http://www.emdebian.org/grip/ lenny/main libtext-charwidth-perl 0.04-5+b1em1 [6,656B]
    Get:24 http://www.emdebian.org/grip/ lenny/main libtext-wrapi18n-perl 0.06-6em1 [4,444B]
    Get:25 http://www.emdebian.org/grip/ lenny/main debconf-i18n 1.5.24em1 [2,882B]
    Get:26 http://www.emdebian.org/grip/ lenny/main debconf 1.5.24em1 [110kB]
    Get:27 http://www.emdebian.org/grip/ lenny/main dhcp3-client 3.1.1-6+lenny4em1 [185kB]
    Get:28 http://www.emdebian.org/grip/ lenny/main sed 4.1.5-6em1 [23.8kB]
    Get:29 http://www.emdebian.org/grip/ lenny/main ncurses-bin 5.7+20081213-1em1 [70.8kB]
    Get:30 http://www.emdebian.org/grip/ lenny/main lsb-base 3.2-20em1 [5,888B]
    Get:31 http://www.emdebian.org/grip/ lenny/main module-init-tools 3.4-1em1 [44.5kB]
    Get:32 http://www.emdebian.org/grip/ lenny/main libssl0.9.8 0.9.8g-15+lenny8em1 [713kB]
    Get:33 http://www.emdebian.org/grip/ lenny/main wget 1.11.4-2+lenny2em1 [116kB]
    Get:34 http://www.emdebian.org/grip/ lenny/main hostname 2.95em1 [5,808B]
    Get:35 http://www.emdebian.org/grip/ lenny/main net-tools 1.60-22em1 [156kB]
    Get:36 http://www.emdebian.org/grip/ lenny/main ifupdown 0.6.8+nmu1em1 [18.9kB]
    Get:37 http://www.emdebian.org/grip/ lenny/main mawk 1.3.3-11.1em1 [51.2kB]
    Get:38 http://www.emdebian.org/grip/ lenny/main libncursesw5 5.7+20081213-1em1 [187kB]
    Get:39 http://www.emdebian.org/grip/ lenny/main nano 2.0.7-5em1 [83.6kB]
    Get:40 http://www.emdebian.org/grip/ lenny/main netbase 4.34em1 [11.6kB]
    Get:41 http://www.emdebian.org/grip/ lenny/main libcap1 1:1.10-14em1 [7,574B]
    Get:42 http://www.emdebian.org/grip/ lenny/main ntpdate 1:4.2.4p4+dfsg-8lenny3em1 [36.1kB]
    Get:43 http://www.emdebian.org/grip/ lenny/main libpam-runtime 1.0.1-5+lenny1em1 [7,786B]
    Get:44 http://www.emdebian.org/grip/ lenny/main libpam0g 1.0.1-5+lenny1em1 [41.0kB]
    Get:45 http://www.emdebian.org/grip/ lenny/main libdb4.6 4.6.21-11em1 [531kB]
    Get:46 http://www.emdebian.org/grip/ lenny/main libpam-modules 1.0.1-5+lenny1em1 [160kB]
    Get:47 http://www.emdebian.org/grip/ lenny/main passwd 1:4.1.1-6+lenny1em1 [267kB]
    Get:48 http://www.emdebian.org/grip/ lenny/main libvolume-id0 0.125-7+lenny3em1 [18.2kB]
    Get:49 http://www.emdebian.org/grip/ lenny/main udev 0.125-7+lenny3em1 [145kB]
    Get:50 http://www.emdebian.org/grip/ lenny/main libslang2 2.1.3-3em1 [266kB]
    Get:51 http://www.emdebian.org/grip/ lenny/main libnewt0.52 0.52.2-11.3+lenny1em1 [36.7kB]
    Get:52 http://www.emdebian.org/grip/ lenny/main libpopt0 1.14-4em1 [22.3kB]
    Get:53 http://www.emdebian.org/grip/ lenny/main whiptail 0.52.2-11.3+lenny1em1 [11.7kB]
    Get:54 http://www.emdebian.org/grip/ lenny/main grip-config 0.1.2em1 [11.5kB]
    Get:55 http://www.emdebian.org/grip/ lenny/main gcc-4.3-base 4.3.2-1.1em1 [5,496B]
    Get:56 http://www.emdebian.org/grip/ lenny/main libgcc1 1:4.3.2-1.1em1 [23.7kB]
    Get:57 http://www.emdebian.org/grip/ lenny/main libc6 2.7-18lenny4em1 [4,410kB]
    Get:58 http://www.emdebian.org/grip/ lenny/main base-passwd 3.5.20em1 [11.5kB]
    Get:59 http://www.emdebian.org/grip/ lenny/main base-files 5lenny7em1 [49.2kB]
    Get:60 http://www.emdebian.org/grip/ lenny/main bash 3.2-4em1 [364kB]
    Get:61 http://www.emdebian.org/grip/ lenny/main diff 2.8.1-12em1 [59.6kB]
    Get:62 http://www.emdebian.org/grip/ lenny/main libcomerr2 1.41.3-1em1 [6,366B]
    Get:63 http://www.emdebian.org/grip/ lenny/main e2fslibs 1.41.3-1em1 [91.3kB]
    Get:64 http://www.emdebian.org/grip/ lenny/main libuuid1 1.41.3-1em1 [10.8kB]
    Get:65 http://www.emdebian.org/grip/ lenny/main libblkid1 1.41.3-1em1 [21.6kB]
    Get:66 http://www.emdebian.org/grip/ lenny/main libss2 1.41.3-1em1 [10.9kB]
    Get:67 http://www.emdebian.org/grip/ lenny/main e2fsprogs 1.41.3-1em1 [234kB]
    Get:68 http://www.emdebian.org/grip/ lenny/main findutils 4.4.0-2em1 [160kB]
    Get:69 http://www.emdebian.org/grip/ lenny/main grep 2.5.3~dfsg-6em1 [128kB]
    Get:70 http://www.emdebian.org/grip/ lenny/main gzip 1.3.12-6+lenny1em1 [44.0kB]
    Get:71 http://www.emdebian.org/grip/ lenny/main login 1:4.1.1-6+lenny1em1 [50.6kB]
    Get:72 http://www.emdebian.org/grip/ lenny/main mktemp 1.5-9em1 [5,772B]
    Get:73 http://www.emdebian.org/grip/ lenny/main mount 2.13.1.1-1em1 [69.3kB]
    Get:74 http://www.emdebian.org/grip/ lenny/main libsepol1 2.0.30-2em1 [96.4kB]
    Get:75 http://www.emdebian.org/grip/ lenny/main sysvinit-utils 2.86.ds1-61em1 [17.8kB]
    Get:76 http://www.emdebian.org/grip/ lenny/main initscripts 2.86.ds1-61em1 [33.6kB]
    Get:77 http://www.emdebian.org/grip/ lenny/main sysv-rc 2.86.ds1-61em1 [13.7kB]
    Get:78 http://www.emdebian.org/grip/ lenny/main sysvinit 2.86.ds1-61em1 [46.8kB]
    Get:79 http://www.emdebian.org/grip/ lenny/main tar 1.20-1+lenny1em1 [148kB]
    Get:80 http://www.emdebian.org/grip/ lenny/main tzdata 2010j-0lenny1em1 [749kB]
    Get:81 http://www.emdebian.org/grip/ lenny/main util-linux 2.13.1.1-1em1 [293kB]
    Get:82 http://www.emdebian.org/grip/ lenny/main balloon3-config 0.6 [2,400B]
    Get:83 http://www.emdebian.org/grip/ lenny/main bsdutils 1:2.13.1.1-1em1 [17.0kB]
    Get:84 http://www.emdebian.org/grip/ lenny/main libdevmapper1.02.1 2:1.02.27-4em1 [44.1kB]
    Get:85 http://www.emdebian.org/grip/ lenny/main makedev 2.3.1-88em1 [15.8kB]
    Get:86 http://www.emdebian.org/grip/ lenny/main ncurses-base 5.7+20081213-1em1 [16.4kB]
    Get:87 http://www.emdebian.org/grip/ lenny/main procps 1:3.2.7-11em1 [160kB]
    Fetched 15.4MB in 3s (4,819kB/s)
    Download complete and in download only mode
    I: Calculating obsolete packages
    I: Extracting apt_0.7.20.2+lenny2em1_armel.deb...
     -> Processing conffiles for apt
    I: Extracting balloon3-config_0.6_all.deb...
    I: Extracting base-files_5lenny7em1_armel.deb...
     -> Processing conffiles for base-files
    I: Extracting base-passwd_3.5.20em1_armel.deb...
    I: Extracting bash_3.2-4em1_armel.deb...
     -> Processing conffiles for bash
    I: Extracting bsdutils_1%3a2.13.1.1-1em1_armel.deb...
    I: Extracting coreutils_6.10-6em1_armel.deb...
    I: Extracting debconf-i18n_1.5.24em1_all.deb...
    I: Extracting debconf_1.5.24em1_all.deb...
     -> Processing conffiles for debconf
    I: Extracting debian-archive-keyring_2010.08.28~lenny1em1_all.deb...
    I: Extracting debianutils_2.30em1_armel.deb...
    I: Extracting dhcp3-client_3.1.1-6+lenny4em1_armel.deb...
     -> Processing conffiles for dhcp3-client
    I: Extracting dhcp3-common_3.1.1-6+lenny4em1_armel.deb...
    I: Extracting diff_2.8.1-12em1_armel.deb...
    I: Extracting dpkg_1.14.29em1_armel.deb...
     -> Processing conffiles for dpkg
    I: Extracting e2fslibs_1.41.3-1em1_armel.deb...
    I: Extracting e2fsprogs_1.41.3-1em1_armel.deb...
     -> Processing conffiles for e2fsprogs
    I: Extracting findutils_4.4.0-2em1_armel.deb...
    I: Extracting gcc-4.3-base_4.3.2-1.1em1_armel.deb...
    I: Extracting gnupg_1.4.9-3+lenny1em1_armel.deb...
    I: Extracting gpgv_1.4.9-3+lenny1em1_armel.deb...
    I: Extracting grep_2.5.3~dfsg-6em1_armel.deb...
    I: Extracting grip-config_0.1.2em1_all.deb...
    I: Extracting gzip_1.3.12-6+lenny1em1_armel.deb...
    I: Extracting hostname_2.95em1_armel.deb...
    I: Extracting ifupdown_0.6.8+nmu1em1_armel.deb...
     -> Processing conffiles for ifupdown
    I: Extracting initscripts_2.86.ds1-61em1_armel.deb...
     -> Processing conffiles for initscripts
    I: Extracting libacl1_2.2.47-2em1_armel.deb...
    I: Extracting libattr1_1%3a2.4.43-2em1_armel.deb...
    I: Extracting libblkid1_1.41.3-1em1_armel.deb...
    I: Extracting libbz2-1.0_1.0.5-1em1_armel.deb...
    I: Extracting libc6_2.7-18lenny4em1_armel.deb...
     -> Processing conffiles for libc6
    I: Extracting libcap1_1%3a1.10-14em1_armel.deb...
    I: Extracting libcomerr2_1.41.3-1em1_armel.deb...
    I: Extracting libdb4.6_4.6.21-11em1_armel.deb...
    I: Extracting libdevmapper1.02.1_2%3a1.02.27-4em1_armel.deb...
    I: Extracting libgcc1_1%3a4.3.2-1.1em1_armel.deb...
    I: Extracting liblocale-gettext-perl_1.05-4em1_armel.deb...
    I: Extracting libncurses5_5.7+20081213-1em1_armel.deb...
    I: Extracting libncursesw5_5.7+20081213-1em1_armel.deb...
    I: Extracting libnewt0.52_0.52.2-11.3+lenny1em1_armel.deb...
    I: Extracting libpam-modules_1.0.1-5+lenny1em1_armel.deb...
     -> Processing conffiles for libpam-modules
    I: Extracting libpam-runtime_1.0.1-5+lenny1em1_all.deb...
     -> Processing conffiles for libpam-runtime
    I: Extracting libpam0g_1.0.1-5+lenny1em1_armel.deb...
    I: Extracting libpopt0_1.14-4em1_armel.deb...
    I: Extracting libreadline5_5.2-3.1em1_armel.deb...
    I: Extracting libselinux1_2.0.65-5em1_armel.deb...
    I: Extracting libsepol1_2.0.30-2em1_armel.deb...
    I: Extracting libslang2_2.1.3-3em1_armel.deb...
    I: Extracting libss2_1.41.3-1em1_armel.deb...
    I: Extracting libssl0.9.8_0.9.8g-15+lenny8em1_armel.deb...
    I: Extracting libstdc++6_4.3.2-1.1em1_armel.deb...
    I: Extracting libtext-charwidth-perl_0.04-5+b1em1_armel.deb...
    I: Extracting libtext-iconv-perl_1.7-1+b1em1_armel.deb...
    I: Extracting libtext-wrapi18n-perl_0.06-6em1_all.deb...
    I: Extracting libusb-0.1-4_2%3a0.1.12-13em1_armel.deb...
    I: Extracting libuuid1_1.41.3-1em1_armel.deb...
    I: Extracting libvolume-id0_0.125-7+lenny3em1_armel.deb...
    I: Extracting login_1%3a4.1.1-6+lenny1em1_armel.deb...
     -> Processing conffiles for login
    I: Extracting lsb-base_3.2-20em1_all.deb...
    I: Extracting lzma_4.43-14em1_armel.deb...
    I: Extracting makedev_2.3.1-88em1_all.deb...
    I: Extracting mawk_1.3.3-11.1em1_armel.deb...
    I: Extracting mktemp_1.5-9em1_armel.deb...
    I: Extracting module-init-tools_3.4-1em1_armel.deb...
     -> Processing conffiles for module-init-tools
    I: Extracting mount_2.13.1.1-1em1_armel.deb...
    I: Extracting nano_2.0.7-5em1_armel.deb...
     -> Processing conffiles for nano
    I: Extracting ncurses-base_5.7+20081213-1em1_all.deb...
     -> Processing conffiles for ncurses-base
    I: Extracting ncurses-bin_5.7+20081213-1em1_armel.deb...
    I: Extracting net-tools_1.60-22em1_armel.deb...
    I: Extracting netbase_4.34em1_all.deb...
     -> Processing conffiles for netbase
    I: Extracting ntpdate_1%3a4.2.4p4+dfsg-8lenny3em1_armel.deb...
     -> Processing conffiles for ntpdate
    I: Extracting passwd_1%3a4.1.1-6+lenny1em1_armel.deb...
     -> Processing conffiles for passwd
    I: Extracting perl-base_5.10.0-19lenny2em1_armel.deb...
    I: Extracting procps_1%3a3.2.7-11em1_armel.deb...
     -> Processing conffiles for procps
    I: Extracting readline-common_5.2-3.1em1_all.deb...
    I: Extracting sed_4.1.5-6em1_armel.deb...
    I: Extracting sysv-rc_2.86.ds1-61em1_all.deb...
    I: Extracting sysvinit-utils_2.86.ds1-61em1_armel.deb...
    I: Extracting sysvinit_2.86.ds1-61em1_armel.deb...
    I: Extracting tar_1.20-1+lenny1em1_armel.deb...
     -> Processing conffiles for tar
    I: Extracting tzdata_2010j-0lenny1em1_all.deb...
    I: Extracting udev_0.125-7+lenny3em1_armel.deb...
     -> Processing conffiles for udev
    I: Extracting util-linux_2.13.1.1-1em1_armel.deb...
     -> Processing conffiles for util-linux
    I: Extracting wget_1.11.4-2+lenny2em1_armel.deb...
     -> Processing conffiles for wget
    I: Extracting whiptail_0.52.2-11.3+lenny1em1_armel.deb...
    I: Extracting zlib1g_1%3a1.2.3.3.dfsg-12em1_armel.deb...
    I: Unpacking complete.
    Get:1 http://www.emdebian.org lenny Release.gpg [197B]
    Ign http://www.emdebian.org/grip/ lenny/main Translation-en_US
    Get:2 http://www.emdebian.org lenny Release [21.4kB]
    Ign http://www.emdebian.org lenny Release
    Ign http://www.emdebian.org lenny/main Packages
    Ign http://www.emdebian.org lenny/main Sources
    Ign http://www.emdebian.org lenny/main Packages
    Ign http://www.emdebian.org lenny/main Sources
    Hit http://www.emdebian.org lenny/main Packages
    Hit http://www.emdebian.org lenny/main Sources
    Fetched 198B in 0s (2,020B/s)
    Reading package lists... Done
    W: GPG error: http://www.emdebian.org lenny Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B5B7720097BB3B58
    
    Multistrap system installed successfully in /home/mike/celf/multistrap/RFS/.

    Let’s explain the parameters:

    • armel is the architecture, for our example: arm in little endian
    • -d $PWD/RFS is the output directory containing the root filesystem. Be careful to pass an absolute path.
    • -f multistrap.conf is the name of the configuration file

    If you look at RFS/dev, you will see that there are no device files in it. This will be a problem at boot time, unless you build a kernel with the below options:

    CONFIG_DEVTMPFS=y
    CONFIG_DEVTMPFS_MOUNT=y
    

    Thanks to these parameters, the kernel will automatically mount a tmpfs filesystem on /dev, and will populate it with devices present on the system.

    Now we have a root filesystem, but we still need to run the package configuration scripts to make it usable. The packages were installed, but their configuration scripts couldn’t be executed, because they can only run on the target architecture.

    The easiest way to do this is to use NFS. On the host side, you need to export the root filesystem directory through NFS. On the target side you have to select /bin/sh for the init process. A typical kernel command could be:

    console=ttyS0,115200 root=/dev/nfs nfsroot=192.168.0.1:/path/RFS rw ip=192.168.0.20 init=/bin/sh

    Then boot your board. You should reach a command line.

    First mount /proc:

    mount -t proc nodev /proc
    

    Then configure your packages using this command line:

    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin  dpkg --configure -a 

    You will get a few questions about localization for tzdata, then the packages will be configured.

    Finally go back to your host to change the RFS/etc/inittab file by uncommenting the below line and modifying it according to your serial console configuration (usually 115200). For example, replace

    #T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100

    by

    T0:23:respawn:/sbin/getty -L ttyS0 115200 vt100

    As we built this root filesystem as a normal user we may have problems with some setuid programs, which need to be owned by the root user. So let’s change the ownership of some directories:

    sudo chown root:root -R RFS/bin RFS/usr/bin RFS/sbin RFS/usr/sbin

    The last trick is to delete the root password by modifying RFS/etc/passwd, by replacing

    root:x:0:0:root:/root:/bin/bash

    by

    root::0:0:root:/root:/bin/bash

    You can now reboot your system without the init=/bin/sh kernel parameter. We now have a ready to use embedded Linux root filesystem with the power of Debian.

    Other things to fix and do:

    • Configure the /etc/resolv.conf file to be able to connect to the Internet.
    • Configure the gateway in the ip= kernel parameter (ip=client-ip:server-ip:gw-ip:netmask:hostname:device:autoconf). For example:
      bootargs=console=ttyS0,115200n8 root=/dev/nfs ip=192.168.2.100::192.168.2.1:255.255.255.0:emdebian:eth0:off nfsroot=192.168.2.1:/home/mike/work/celf/
    • Install other packages such as mtd-utils and vim
    • Install kernel modules (either manually or through a kernel package)
    • Add a new user
    • Create a minimum /etc/fstab file

    You now have a Debian system for which it is very easy to add new software, and which can be configured in a very familiar way. That’s great to make product prototypes, small, low-power and secure servers for home or office use, and in some cases, even real products.