Open Source Summit Europe 2022

Running out of desk space for your debug devices? Do you hate wasting time trying to figure out which serial port your board is hanging off? How about resetting the power of your board - do you have to lean over lots of other stuff to re-plug the power cable? Maybe you have a huge collection of embedded devices which keep growing or perhaps you have struggled because the board you want to use is with a colleague across the globe and you cannot wait for it to be shipped? These issues were all solved when Chris created a board farm at a minimal hardware cost using LabGrid. This beginner-level presentation will introduce you to a device-agnostic way to share boards across the globe and will help make your development simpler. Expect a live demo of how simple it is to remotely power-on a board, flash an image, access the console and even help verify peripherals, all without leaving your console! Practical tips of the hardware used, the Ansible recipes used to setup the software stack, connecting new boards and integration of boards in the farm to image-building CI pipelines in GitLab and GitHub will also be shown.

KernelCI is a project focused on testing the upstream Linux Kernel on different hardware with an open testing philosophy and high modularity. Thanks to this approach, KernelCI is expanding its testing ecosystem by allowing new tests and trees to be easily integrated into KernelCI. We will talk about two main ways of growing the current KernelCI ecosystem: Adding to the KernelCI code Using KCIDB (KernelCI’s results aggregation system) From a CIP project (Civil Infrastructure Platform) testing member and KernelCI CIP instance mentor, you will get an overview of the effort of the CIP project to merge its current testing system into KernelCI, and how it is possible to collaborate and send test results to KernelCI using KernelCI’s common reporting system KCIDB. The CIP project tests merged into KernelCI are currently used for testing the CIP Kernel. The CIP Kernel is currently used for example for isar CIP core and EMLinux (an embedded Linux distribution). This talk will give an overview of what we learned from making GKernelCI (Gentoo Kernel automatic testing system) and CIP testing systems collaborate with KernelCI. What we did, what is still missing and what is planned in the future. We hope that this experience will help future collaboration with the KernelCI project.

During this presentation, the listener will learn how software dinosaurs (meant positively) like Debian can be combined with modern tools like Ansible, git and Mender by applying best practices like GitOps and infrastructure as code. The outcome will be a highly scalable, robust and traceable embedded fleet setup that is easy to operate with a reduced number of tools at a global scale using a modest server infrastructure. During the presentation a concrete example of managing an individual IoT device will be shown. The IoT device will furthermore receive an automatically built OS image that gets dispatched over the air. Hardware-in-the-loop testing guarantees the quality of the OS image. Finally a sample IoT fleet of seven devices (5 Raspberry Pis, 1 Compulab IoT gateway, 1 Variscite SOM on a development board) will be managed using a git repository instead of a typical IoT fleet management tool. The presentation done by Matthias Lüscher shall encourage the audience to discover tools and best practices from other domains and apply them successfully to embedded projects.

With great labs comes great responsibility! In this presentation, we will walk through the challenges of maintaining and growing the Collabora LAVA lab, which now counts more than 100 devices of several types running thousands of test jobs every day. Our main goal is to build an ecosystem of devices that open source projects can use for automated testing on various hardware. Successful CI initiatives such as KernelCI and MesaCI have been leveraging the Collabora lab for their automated upstream testing and development workflows, with more than a hundred thousand test jobs running every month. Maintaining and scaling up on such a large lab comes with its own sets of requirements and challenges. As coverage increases in the project pipelines, the impact of any unreliability or failure in the hardware pool multiplies. Continuous monitoring and recovery mechanisms are crucial to ensure good device availability and reliability. We will discuss the lifecycle of devices inside the lab from the bring up to the day-to-day care, as well as the ongoing efforts to make the Collabora LAVA lab more robust as it grows.

Testing of software on real hardware is an important part of the development cycle. As there is no co-located automated testing event this year, we want to use this BoF to continue the exchange between the people involved in testing for Embedded Linux. Compared to the Kernel Testing & Dependability MC at Plumbers, the BoF will focus more on the practical side of running a board farm. We'll start with a quick summary of recent developments related to board farms, lab control frameworks, KernelCI and automated testing in general. Then, we can share (good or bad) experiences with different approaches and tools or just for discussion in general. Possible discussion topics for this BoF are: - control software (Lava, tbot, pdudaemon, labgrid, the board farm REST-API, ...) - board farm architectures - controller to board mapping (1:N vs. 1:1) - useful hardware tools - gaps in our tools - plans for the next year - possible areas for collaboration - … We'll be using a shared document to gather topics before and during the BoF and collect notes during the discussions: https://docs.google.com/document/d/1oxgzOvWu6vbC4SoJq6y-NbqhlnEED73ZeCp7hpSU7QU/edit?usp=sharing

Marek is doing day-to-day testing of the Linux kernel 'next' branch since 2018. Since v4.15 Linux kernel release there are almost 250 commits with his 'Tested-by' and 90 commits with his 'Reported-by' tags. In his speech, Marek will present his system used for the daily tests, show how does it work and what kind of tests are being performed. Then he will show how he identifies the regressions, how to prepare a report about such regression and how to help kernel developers to fix it. Marek will also show how a typical bisection process does look like, how to do it automatically and how to interpret the results. He will also share some practical hints and tips related to the process of finding regressions and identifying broken commits. In the summary Marek will briefly compare his solution with other kernel testing robots and systems.