There’s also SkiffOS (https://github.com/skiffos/SkiffOS).

It’s a project that uses buildroot to create a small Linux for a specific device that’s only used to start a container.

I’ve wanted to try it sometime after getting headaches with both Buildroot and Yocto. Particular adding more libraries tends to break things.

That doesn't sound very performant.

Why not? The only overhead I can see is some storage and memory overhead due to duplicate libraries, and some possible small startup time penalty? Containers are just normal Linux processes after all, it's not like there's a VM involved

Dont forget having multiple versions of a library will blow up your cache.

Just to the degree that there's more machine code competing for the same limited page cache... nothing about having multiple versions of one library is worse for cache than having the same amount of machine code across different libraries. It's not ideal, but considering how lean you could make the host system, it's a minimal cost.

My experience with buildroot is that it's really slow to compile because it doesn't compile packages in parallel (so you'll only get the parallelism of an individual package's build system, with sequential stuff inbetween), and you end up recompiling from source a whole lot because it doesn't do dependency tracking between packages so if you change a library, you either have to manually remember to recompile the whole chain of dependents, or do a clean build. Yocto, on the other hand, compiles packages in parallel and tracks which packages need to be recompiled due to a changed recipe or config file.

Buildroot has package-parallel builds when using BR2_PER_PACKAGE_DIRECTORIES (see https://buildroot.org/downloads/manual/manual.html#top-level...). It's for some reason still marked as experimental in the docs but it has been solid for me for many years.

The lack of dependency tracking isn't great but other than working around it like you described just using ccache has worked pretty well for me. My Buildroot images at work do full recompiles in under 10 minutes that way.

Meanwhile the Yocto projects I've worked on used to have a ton of chaff that causes partial rebuilds with trivial changes to take longer than that. This probably isn't an inherent Yocto/BitBake thing but the majority of Yocto projects out there seems to take a very kitchen-sink approach so it's what you'll end up having to deal with in practice.

I think, in a lot of cases, the choice between Buildroot and Yocto comes down to "which one does the SoC vendor support."

Yocto is synonymous with low-end IoT these days, and causes more problems than it solves in the long-term for many folks.

Also, bootstrapping your own application launcher shell on a raw kernel is usually not a difficult task (depending on vendor firmware.) Some folks just drop a full Lua environment for an OS that fits in under 2.7MB ISO even with a modern kernel.

Nir Lichtman posted a tutorial for mere mortals here:

https://www.youtube.com/watch?v=u2Juz5sQyYQ

Highly recommended exercise for students =3

I think that's fair, but it does depend on what you want the relationship with your SoC vendor and the Yocto community to be. A lot of SoCs have pretty good community support in Yocto (and probably Buildroot), and using a community-maintained BSP meta layer will make things easier for you in some ways. SoC vendors aren't always great at following Yocto best practices. Plus, unless you have excellent support contracts with your vendor and are prepared to use it, you'll probably go to the Yocto community for support with weird Yocto issues you run into; and Yocto developers are (understandably) much more helpful if you say you use mainline Linux with a BSP maintained by the Yocto project than if you use a vendor's kernel fork with a BSP maintained by the SoC vendor.

Buildroot is easy to use even without vendor support

buildroot to bringup, yocto to ship