Always UPS everything. But also always have a simple backdoor. I generally have 1 little desktop like a NUC running some basic Win10 OS and an install of remote software like TeamViewer. It is connected to my hardware router right after the ISP router and a backup connection. Used to be LTE everywhere, now I’m half and half on Starlink. It is then also connected to the router ports needed for management but inactive.

If I have to, remote into the NUC over Starlink. I can then reboot my main ISP box. I can eventually get into my router and enable those ports which are pre-plugged in. From there I can then access all the stack management and all the IPMI ports like iLo. It’s a virtual interface through a virtual interface. It is slow, and painful. But it works.

And it works 99.99% of the time. But even then, I’ve had to do a call of shame and walk one of my friends through which button to press as I’m on the other side of the world. In my case it was also power related but the UPS I had decided to overheat. In reality over the summer, the temps were high. But also it is a super awesome double conversion UPS. The line voltage into the UPS was dropped to below standards from the utility because their grid was overworked with everyone’s AC’s. So the UPS saw this as a line failure, kicked in the double conversion and ran happily. But it did not count as a power failure, so none of my services scaled back. Essentially it was delivering 3KW of juice from the wall through a double conversion making the whole thing super hot. Eventually it shut down for safety automatically, just pulled the plug. My NUC is on a separate little backup along with the modems and an auto transfer switch which did its thing. But there was no way to press the reset button on the UPS for a critical safety shutdown like that. It had to be in person.