Open Source Summit Europe 2022

With great labs comes great responsibility! In this presentation, we will walk through the challenges of maintaining and growing the Collabora LAVA lab, which now counts more than 100 devices of several types running thousands of test jobs every day. Our main goal is to build an ecosystem of devices that open source projects can use for automated testing on various hardware. Successful CI initiatives such as KernelCI and MesaCI have been leveraging the Collabora lab for their automated upstream testing and development workflows, with more than a hundred thousand test jobs running every month. Maintaining and scaling up on such a large lab comes with its own sets of requirements and challenges. As coverage increases in the project pipelines, the impact of any unreliability or failure in the hardware pool multiplies. Continuous monitoring and recovery mechanisms are crucial to ensure good device availability and reliability. We will discuss the lifecycle of devices inside the lab from the bring up to the day-to-day care, as well as the ongoing efforts to make the Collabora LAVA lab more robust as it grows.