Testing my own backup guide found four bugs

November 2, 2026 · 6 min read · testing, backup, linux, lxd

New here? Start here for what my homelab is and how this site is organized.

When someone follows a guide, they copy a command, paste it, and expect it to work. If step 4 fails, they don’t know if it’s their machine or my mistake. Most people stop there. Some keep going and end up with half a setup they don’t understand, which is worse.

So I made a rule for this site: every guide is run from scratch on fresh Ubuntu machines before I publish it. Not on my own server. My server has been set up for months, and I have tweaked things along the way that I have long forgotten. If a guide works there, it only proves it works on my server.

The backup guide was the hardest one to test so far. It needed three runs and found four bugs.

How the test works

The test runs on a spare PC with LXD, which makes small virtual machines. For the backup guide it makes two:

• A NAS. A plain Ubuntu machine with an NFS share that only the server is allowed to use, set up the way the guide tells you to set up yours.
• A server. A fresh Ubuntu machine that first gets the earlier guides on this site: the firewall and updates, then monitoring with email alerts. So it looks like the server of a reader who followed them in order.

Alert emails go to a tiny fake mail server inside the test machine. It saves each email to a file, so the test can check that the right alert arrived, and nothing goes out to the internet.

The part I like most: the test doesn’t have its own copy of the commands. A small script reads the guide itself, pulls out every command box, and runs them exactly as written. If I fix the guide, the test picks up the fix. There’s no second copy that can drift away from what readers see.

Then the test breaks things on purpose and checks that the guide’s promises hold:

• It runs a backup and checks what’s inside the archive.
• It changes a file and restores just that one file.
• It turns the NAS off and waits for the “backup failed” email.
• It fakes an old backup and waits for the “backup missing” email, then checks that it clears.
• It deletes a Grafana dashboard and restores Grafana from the backup.
• It runs the monthly restore test.
• It reboots, checks that everything came back, then runs the guide’s undo steps and checks that nothing is left.

Alerts take a few minutes to fire, so one run takes about 35 minutes. I run it on two Ubuntu releases, the two latest long-term releases.

Run one: two bugs

Most of it worked. The backup, the single-file restore, the full restore and the “NAS is off” email all did what the guide said. Two things didn’t.

The “Backup missing” alert fired right after the first good backup. Here’s how that part works. When a backup succeeds, the script writes the time to a small text file. Node exporter, the program that feeds server numbers to Prometheus, reads files like that from one folder and passes them on. The alert fires if that time is too old, or missing.

Node exporter never saw the file. My monitoring guide changes node exporter’s settings line to turn on an extra feature. That new line replaced the default one, and the default was what told node exporter to read that folder. Each guide worked on its own. Used together, in the order I recommend, they broke. The fix is one extra command in the backup guide that adds the folder back to the settings line.

The restore test said “no dashboards” when the dashboard was right there. The guide includes a monthly test that opens the backup and checks that Grafana’s data is really in it. My first version counted rows in the database table where Grafana used to keep dashboards. The current Grafana keeps them somewhere else, so the count was zero.

The dashboard was in the backup. The test was just looking in the wrong place. A test that fails when everything is fine is less dangerous than one that passes when it shouldn’t. But it’s still bad, because it teaches you to ignore it, and then you ignore it on the day it’s right. Now the test starts a throwaway copy of Grafana from the backup and asks it, through its own API, how many dashboards it sees. That works whatever Grafana does with its tables next.

Run two: two more

Fixing those let the test get further, and it found two more.

The backup time still didn’t show up. The backup script starts with umask 077. That makes everything the script creates private to root, which is exactly what I want for an archive full of passwords and keys. But it also made the little time file private. Node exporter runs as its own user, not root, so it couldn’t read the file.

Both choices were right on their own. A security setting quietly broke the monitoring. The fix is a chmod 644 on that one file, with a comment in the script so nobody “fixes” it back later. I also added a line to the guide’s troubleshooting section: check that the file is readable by everyone.

The throwaway Grafana crashed with “permission denied”. The restore test starts it as the grafana user. When you run something with sudo, it starts in the folder you were in. As root, that was /root, and the grafana user isn’t allowed to read /root. Grafana gave up before it even started.

From the monthly timer, it would have worked, because systemd starts everything from /. It only failed when run by hand with sudo. That’s exactly when a reader would run it, right after setting it up, to check that it works. They would have seen a failed restore test on day one and lost trust in the whole guide. The fix is a cd / before starting Grafana.

Run three

Everything passed on both releases. The backup time showed up in Prometheus. “Backup missing” stayed quiet, fired on the fake old backup, and cleared again. The restore test passed and found the dashboard. The NAS-off email arrived. The single-file and full restores worked. After the reboot, everything came back by itself. After the undo steps, nothing was left.

Then I published the guide.

It’s not a one-off

This keeps happening. When I tested the monitoring guide the same way, it found five bugs. One alert rule wasn’t quoted properly for its file format. One check ran two seconds too early. The undo steps left folders behind. Some alerts sent extra emails nobody needed. And a package the installer recommends failed at every boot. Each one would have cost a reader an evening.

What I took from it

None of these bugs would have shown up on my own server. A fresh machine is the only honest reader of a guide.

All four bugs were in the gaps between things:

• Between two guides. Each worked alone; together they didn’t.
• Between three users. Root, node exporter and Grafana each see different files.
• Between two ways of running the same script. A timer and sudo start in different places.

That’s where I look first now. When I write a step, I ask: who runs this, from where, and what did an earlier guide already change? And when a test fails, I check the test before I check the guide. Sometimes the test is the bug.