My gateway ran at 95 °C for two days, and nothing told me
New here? Start here for what my homelab is and how this site is organized.
It was a little before six in the morning, and the UniFi app was so slow I thought the internet was down. Every page of it took ages to open.
The internet wasn’t down. Websites loaded fine on my laptop. A ping to the gateway came back in half a millisecond. DNS answered right away. Only the management side was crawling. That was the first clue: whatever was wrong, it wasn’t the part of the gateway that moves packets.
The graphs had it all along
I already send the gateway’s numbers to Grafana. A small tool called UnPoller reads them from the UniFi controller and stores them in a database, and I had a nice dashboard for it. So I opened it and looked back.
The story was right there:
- For weeks the CPU had sat around 11%, and the chip around 80 °C.
- One afternoon two days earlier, the CPU climbed to 67%.
- By that evening it was at 90%, and the chip was between 95 and 97 °C.
- It stayed there, non-stop, for almost two days.
Memory was at 78 to 80%, and the gateway was using close to a gigabyte of swap. Network traffic was close to zero, both on the internet side and inside the house. So it wasn’t a download, a backup or a busy device. Something inside the gateway was working hard for no reason.
The data was there the whole time. I had the graphs. I didn’t have an alert, so nobody looked. For two days my gateway ran hotter than I would ever choose to run it, and I found out only because a web page was slow.
What was eating the CPU
I logged in to the gateway over SSH and ran top. The usual suspects on a gateway are routing and the intrusion prevention engine. It wasn’t either of them.
It was the UniFi Network application, the Java program behind the dashboard, at 283% CPU. That’s almost three cores busy, on a small box, for a program that mostly draws web pages and collects statistics.
Its log was full of the same line, over and over: “could not acquire lock … before timeout”. It was trying to save statistics for the gateway and one of my switches, and it couldn’t. That’s a symptom of an overloaded app, not a cause. I looked for what had started it, but the log had already rotated. The older lines were gone.
The only clue I had: about three hours before the CPU went up, I had turned on a new firewall zone for my servers. Maybe that was it, maybe not. I wrote it down and moved on.
Restart the app, not the box
The quick fix would have been to reboot the gateway. I didn’t, for two reasons. A reboot cuts the internet for everyone in the house for a few minutes. And it wipes the state I might want to look at later.
UniFi runs the Network application as its own service, separate from routing, Wi-Fi and DNS. So I saved the error log first, then restarted only that service:
systemctl restart unifi
Nobody in the house noticed anything. Two minutes later the CPU was at 34%, the load was coming down, and the chip was at 78 °C. All my UniFi devices were reporting again, and my server’s health check passed.
The two alerts
Then I wrote the alerts I should have had from the start. They are Grafana rules in a file, so I can keep them with my other configs and reinstall them in one step:
- Gateway CPU high: the 30-minute average is above 70%, for 5 minutes.
- Gateway too hot: the 15-minute average is above 90 °C, for 10 minutes.
I picked the numbers from my own graphs. My normal was about 11% and 80 °C, so 70% and 90 °C are far enough above normal to mean something, and far enough below the incident to catch it early.
I used averages on purpose. A firmware update or a speed test can push the CPU to 100% for a minute, and I don’t want an email for that. I would ignore that kind of email after the third time, and then the alert would be worthless. Two days at 90% is a different story.
The setup script has a dry-run mode, like all my scripts. Before changing anything, it printed what the new rules would have seen during the incident. The 30-minute average had peaked well above the line, so the CPU alert would have fired on the first evening. That’s the test I wanted: not just “the rule loads”, but “the rule would have caught the real thing”. Then I applied it. The script saved the old rules file, restarted Grafana, checked that Grafana came back, and printed how to undo it.
It came back
Six days later, the CPU alert emailed me in the morning.
I was in the middle of a big change: moving my server to a new network zone. The console was slow again, and one switch sat in “Getting Ready” for about seven minutes. I logged in to the gateway. Same thing: the Network app at 164% CPU, the load around 4.5, a gigabyte of swap. The gateway had been up for 42 days.
I restarted the app, and two minutes later the chip was back at 78 °C. From the email to the fix took less than two hours, on a busy morning. Last time it took two days, and luck.
What I still don’t know
I never found the root cause. Both times, it started during or soon after a big change to the network. That’s a pattern, not proof. If it happens a third time, I’ll look at a firmware update first, then a scheduled weekly restart of the Network app. A weekly restart is a workaround, but it’s a cheap one, and the alerts will tell me if it’s not enough.
One more thing changed after this. I needed SSH on the gateway to see what was going on, and it had stayed on since. A few days later I turned it off. Now I turn it on for an incident, fix the problem, and turn it off again. The password lives in my password manager, not in my head or a text file.
What I took from it
A graph without an alert is a history book. It tells you what happened, after you already found out the hard way.
I had spent time building nice dashboards, and they were useless in the moment that mattered, because I wasn’t looking at them. Now, for everything I care about, I ask two questions:
- Would I see this on a graph?
- Would anything tell me if I didn’t look?
If the answer to the second one is no, the job isn’t finished. And when I write the alert, I set the line from my own normal, and I check it against a real bad day before I trust it.