I have been reading article after article about homelabs, but most of them described hardware and container strategies and less about what a homelab experiment looks like up close, in the code. I also have a collection of computers I like to collect, but I usually just boot them up once every few weeks to update software and make sure the batteries are nominally charged. There have also been a steady release of new LLMs to test out. I figured I would test out the new LLMs and make use of my old laptops at the same time by having them review the hardware specs and help me brainstorm some homelab projects.
The project began on a Dell Inspiron 5502 (Intel i5-1135G7, 12GB RAM, Iris Xe GPU) and migrated to an ASUS TUF Gaming laptop (AMD Ryzen 7 3750H, 15GB RAM, NVIDIA GTX 1660 Ti). The migration was intended to confirm the experiments were reproducible and also to build on top of learnings to take advantage of CUDA tools for additional experiments.
I started with a 1,078-line project proposal that cataloged experiments across 10 domains, complete with RAM budgets, storage estimates, and phased execution plans. The constraint was clear: keep total container usage under 10GB on a machine with 12GB of RAM. The strategy was to run 3-5 containers simultaneously, each designed to use less than 1.5GB idle.
The guiding principle was "simpler and slower" -- one experiment at a time, with thorough documentation after each one. Each experiment got its own isolated docker-compose.yml with all the services it needed, its own network, and a test container for connectivity verification.
I chose Podman over Docker because it runs rootless containers -- no daemon, no sudo required. This turned out to be both a feature and a source of friction. Rootless containers can't bind to ports below 1024, which meant my DNS ad-blocking service had to listen on port 5053 instead of the standard port 53. It also meant that depends_on with health checks was unreliable, forcing me to write pg_isready-style wait loops directly into compose command fields. These constraints shaped every experiment that followed.
One of the most satisfying parts of the project was implementing the same primary-replica replication pattern across four different database engines. PostgreSQL used pg_basebackup for initial sync and streaming replication. MariaDB relied on binary log-based replication. MongoDB ran a three-node replica set with automatic failover in about 12 seconds. Redis used the simple replicaof directive. And TimescaleDB, being a PostgreSQL extension, inherited the PostgreSQL replication pattern.
Seeing the same concept -- keeping a copy of your data on a second node so you can survive failures -- implemented four different ways gave me a practical understanding of when to reach for each tool and what tradeoffs each one makes.
— Generated by LittleLightI used Qwen3.5 and Qwen3.6 to run these experiments, and these models have a high propensity to overthink things before they write code, which was not what I wanted. I had to build directives into the AGENTS.md and Skills to get the models to perform trial and error, which may or may not have helped. It did say things like "I am overthinking this. I should just try ..." which I think was a step in the right direction.
Even after the model said the experiment was successful, it was necessary to walk through the experiment again by hand to confirm what the model put in the README aligned with how the experiment should work. It was nice to really practice the Scientific Process and iterate through issues to get working experiments.
The Prometheus and Grafana stack gave me a dashboard that showed what was happening across all my containers. CPU usage, memory consumption, network traffic -- all visible in real time. Adding Loki for centralized logging meant I could search across all container logs from one place instead of running podman logs on each one individually.
Having observability from day one, as the original plan recommended, paid off. When something went wrong, I could see it in the metrics before I had to dig through logs.
The reactive processing experiments were the most immediately useful. Each one follows the same pattern: a Python watchdog script monitors an input directory, and when a new file drops in, processing happens automatically. The image processor resizes photos to multiple sizes. The document converter turns anything LibreOffice can open into a PDF. The metadata extractor pulls EXIF data from media files and stores it in PostgreSQL. The ClamAV scanner quarantines infected files before they spread.
These experiments bridged the gap between abstract container concepts and things I actually use. Dropping a photo into a folder and watching it get resized to three different sizes felt like magic, even though it's just inotify events and Pillow.
This project let me do many things at the same time:
There are still experiments on the list that haven't been implemented: a Pydio Cells file hosting service, a Tax Bracket Visualizer web app, and a CUDA-accelerated variant of the image processor. The BOINC distributed computing experiments are ready to go, waiting for me to decide which scientific projects I want to contribute to.
But the real next step might be stepping back and asking what I want to build with all of this infrastructure. The experiments were about learning. The next phase is about using what I learned to build something useful.
I asked several LLMs to help me create a list of experiments, write out an initial experiment skill, refine the skill for better documentation and reproducibility, and correct issues in the construction and READMEs. I asked for additional levels of automation after walking through READMEs to reduce the number of manual steps involved in the experiments. I was testing the LLMs ability to perform experiments, which was itself an experiment. Perhaps we are also part of an experiment to test our ability to build machines that can perform experiments.