djspace

My server took 2 minutes 15 seconds to boot. Now it takes 23.

November 2, 2026 · 6 min read · linux, ubuntu, systemd, nfs

New here? Start here for what my homelab is and how this site is organized.

My server almost never reboots. It runs my monitoring and a few other services, and at the time it also answered DNS for the whole house. When it’s down, nobody at home can open a website. So I avoided rebooting it, and I never paid attention to how long a boot took.

Then an update left a “reboot required” flag that sat there for days. Before I rebooted, I wanted to know what I was walking into. I ran my health check script, which tests that every service, mount and alert is working. It came back with 18 OK and 3 FAIL. Then I asked systemd how long the last boot took, with systemd-analyze.

Two minutes and 15 seconds. That’s a long time for a machine that does nothing special at boot, and every one of those seconds was a second with no DNS in the house.

Finding what it waited for

systemd-analyze splits boot time into two parts. The kernel part was 12 seconds, which is normal. The other 2 minutes 3 seconds came after the kernel, while systemd started services. Something in there was slow.

systemd-analyze blame lists services by how long they took. systemd-analyze critical-chain is more useful: it shows the chain of things the boot actually waited for, in order. At the top of that chain was systemd-networkd-wait-online, with a delay of about two minutes.

A service waiting for nothing

That service has one job. It holds up the boot until the network is “online”, so that anything that needs the network doesn’t start too early. By default, “online” means every network interface it manages.

My server is wired. But it also has a Wi-Fi card I have never used. That card never connects to anything, so it never comes online. The service waited for it until it timed out, on every single boot.

The common advice on forums is to disable the service. I didn’t want to. My server mounts folders from the NAS over the network, and network mounts are meant to wait for “network online” before they start. Without the service, they could try to mount before the network is up and fail.

So I kept it, and told it which interface matters:

sudo systemctl edit systemd-networkd-wait-online

[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --interface=<wired-interface> --timeout=30

The empty ExecStart= line isn’t a typo. It clears the original command, so the second line replaces it instead of adding to it. ip -br link shows your wired interface’s name. The 30-second timeout is a safety net: if the cable is unplugged, the boot waits 30 seconds, not two minutes.

The bigger problem I found on the way

While I was looking at what happens at boot, I read my /etc/fstab, the file that says what to mount at startup. My backup folder on the NAS had the right options. My media folder, on the same NAS, only said defaults.

That’s a trap. With defaults, a network share is treated like a local disk the system can’t live without. If the NAS is off or slow when the server starts, Ubuntu can stop in emergency mode and wait for someone to fix it.

It gets worse. On Ubuntu the root account is locked, so emergency mode won’t even give you a shell. You get a message and a prompt you can’t use. Think about a power cut: the power comes back, the server boots faster than the NAS, the mount fails, and the server sits there stuck until I get to it with a keyboard. If I’m away, the house has no DNS until I’m back.

Three options fix it:

defaults,_netdev,nofail,x-systemd.mount-timeout=30

If the NAS is down, the server now boots without the media folder, my monitoring sends a “NAS share missing” alert, and I mount it by hand once the NAS is back. That’s a much better failure than a dead server.

Small cleanups while I was there

Two more things came along for the ride. The system journal had grown to 2.4 GB. I capped it at 500 MB with a small file in /etc/systemd/journald.conf.d/. And I turned off two services a server doesn’t need: printer discovery (cups-browsed) and the modem manager.

Making it safe to try

None of these changes is big. But a mistake in fstab is exactly the kind of thing that stops a machine from booting, so I wanted to be careful.

I put everything in one script. By default it only shows what it would change. With --apply, it backs up every file before touching it. After changing fstab, it runs findmnt --verify, which checks the file for errors. If that check fails, the script puts the old fstab back by itself and stops before doing anything else. At the end, it prints the exact commands to undo every change.

Then I wrote down how to get back in if the server didn’t boot at all. Since emergency mode is useless with a locked root, the plan was: hold Shift at power-on to get the GRUB menu, press e to edit the boot entry, add init=/bin/bash to the end of the linux line, and boot. That gives a root shell without a password. From there, remount the disk read-write, copy the fstab backup back, and reboot. If even that failed, a Ubuntu USB stick would do the same job.

I never needed any of it. But having it written down before I started made the next step much less stressful.

The test reboot

I rebooted on purpose, late at night, when nobody at home was using anything. When it came back, I ran systemd-analyze again.

23 seconds, down from 2 minutes 15. The part after the kernel went from 2 minutes 3 seconds to 11 seconds. The journal was down to 474 MB.

Then I ran the health check again. All 21 checks passed: the services, both NAS mounts, the server’s address and route, the internet, DNS and ad blocking, monitoring, and no reboot pending. Two of the three earlier failures had been about the reboot itself, and they cleared once it was done.

What I took from it

A server that never reboots hides its boot problems until the worst moment. That’s usually a power cut, when you’re not home, and the machine has to come back by itself with nobody watching.

The 2-minute delay was annoying. The fstab line was dangerous. And I only found it because I went looking at boot for a different reason.

So now I reboot on purpose after big changes, at a time I pick. I run the same health check before and after. And before any change that could stop a machine from booting, I write down how I’ll get back in. It’s much nicer to find a problem at a time I picked, with a plan in my notes, than at the time the power company picks.