diff --git a/content/blog/2023/12-22-alpine-linux-homelab/index.md b/content/blog/2023/12-22-alpine-linux-homelab/index.md new file mode 100644 index 0000000..dab289b --- /dev/null +++ b/content/blog/2023/12-22-alpine-linux-homelab/index.md @@ -0,0 +1,168 @@ +--- +title: Alpine Linux Homelab +date: 2023-12-22 +tags: + - computers +--- + +A few days ago, my root filesystem crashed. + +This isn't as big a deal as it would have been before containers: +because the filesystem was happy enough that docker could start, +all of my services were able to run, since they live on another filesystem. +Weird. +But still, +it needed to be repaired, +so I could run things like `git` (which had become corrupted). + +A few of my pals have been using Alpine Linux for a while. +I've used similar distributions--tinycore and OpenWRT--so +I sort of knew what Alpine was about. +And I've used Alpine a whole lot in container images, +because it's just so gosh darn small. +So I dove in on that. + +2 hours later, I had a whole working system, +running my containerized services just like before. +This is an absolutely bonkers fast redeployment, +and I owe a lot of that to how dead simple Alpine is. + +Setup +----- + +I've got Alpine booting on my Raspberry Pi in "diskless" mode. +That means it loads the entire operating system into RAM. +It then mounts my 12TB btrfs array, +which has all the containerized services, +and launches podman to start them all up. + +If I need to make system-wide changes, +I can just edit files in `/etc` and run +`lbu commit`, which will pick up changes and write them out +as a tarball +to my normally read-only SD card. +This is a Big Deal when you're booting from an SD card, +because those things can't handle lots of writes. +I've also got a service running that copies the tarballs +onto the 12TB btrfs system, +so I should be able to get back online in under an hour if I lose this card. + +I moved from Docker to Podman because the latter uses about half the space on the root filesystem. +That's important to me, +because the root filesystem is in RAM. + + +RAM Disk +-------- + +Running a production machine entirely from RAM may seem foolish, +especially on a Raspberry Pi. +But it's actually pretty svelte: + + Filesystem Size Used Available Use% Mounted on + tmpfs 3.8G 108.8M 3.7G 3% / + +That's right, the entire host OS, +including all the junk needed for networking and podman, +takes up 108MB of the 8GB on this system. +That's just 1.35%! + +I haven't checked, +but I have a sneaking suspicion that base Ubuntu uses more RAM than that, +with all the services it kicks up. + + +Podman +------ + +At my previous job, +I played around with podman a little. +It is more or less a replacement for Docker, +although it doesn't have swarm mode. + +I gave it a shot in this conversion, +after being irritated that Docker itself frequently used over 10% of the CPU, +and quite a lot of RAM. + +It turned out that podman not only doesn't appear to show up in the CPU usage, +but it also takes about half the space on the RAM disk as Docker did. +Running podman as root just worked out of the box: +I had to create two files in `/etc` to map my uid and gid +to get it working for my user account, +but now that's working nicely too. + +I'm not using podman pods, though. +I looked into this once, +and it seemed to require learning a new schema for all kinds of things, +and maintaining hundreds of lines of YAML. +I know runit and bourne shell pretty well, +and I know how to launch individual podman containers with a shared bridge network, +so I'm just doing that. + + + +A Beefier System +---------------- + +All of my services are running in containers. +At some point I realized: +why not put my login shell in a container, too? +So I did. + +My work building that is in my +[toolbox repository](https://git.woozle.org/neale/toolbox), +but the summary is that I'm running a fat Alpine inside my lean host Alpine. +It still uses lbu and apk for preserving state, +but I can stretch out and install bash, vim, and even emacs. +I have a gigantic magnetic disk behind it, +I don't have to worry about storage space used by the OS in this setup. + +Every time the container starts, +it reinstalls all the packages I have set up, +from the cached package repository I bind-mounted into the container from the btrfs array. +If I install new packages, +or update existing ones, +that updates the cache. + +The tooling to do all this is minimal. +I encourage you to +[check out the repository](https://git.woozle.org/neale/toolbox) +if you're interested. +Most of the work is in the +[init script](https://git.woozle.org/neale/toolbox/src/branch/main/sbin/toolbox-init), +and most of the time getting *that* working correctly +was just me realizing that Alpine already had mechanisms for everything I wanted. + + +Runit +----- + +When I managed my own (tiny) Linux distribution, +I became a fan of Dan Bernstein's "runit" init system. +Alpine was happy to let me set that up, +even providing a little `init.d` script to have their built-in openrc start runit. + +Because runit uses files and filesystem pipes, +I can `sv restart /host/etc/services/foo` inside my "fat Alpine" container, +to restart the foo service *on the host OS*. +This lets me use my fat editor on service files, +and restart things without hopping onto the host OS. + + +Closing +------- + +Moving from Ubuntu to Alpine gave me the following benefits: + +* Boot time is down +* Memory use is down (I think) +* CPU use is down, and as a corrolary, so is energy use +* Absolute certainty that my system configuration is preserved in backup, + because every time I boot, I restore the system (into RAM) from that backup +* I get to keep all my creature comforts like bash, btop, vim, emacs, and the like +* If my root filesystem ever crashes again, I can just make a new Alpine SD card, + copy a backup tarball onto it, and I get my system back + +Thanks to Nick Moffitt for blabbing about Alpine enough to get me interested, +and for providing some setup instructions (which I eventually completely undid) +to get me started quickly!