The 25-Minute Drive Where I Imagined Every Worst-Case Scenario
I was asleep when the phone rang; it was 9:30 am.
One of the employees was calling to tell me the system was down. They couldn't access it.
My first thought was that it was probably something simple. Maybe the network switch needed a restart. Maybe there was an outage. Maybe it was one of those incidents that sounds serious until you discover a cable came loose somewhere.
Then they told me the local network wasn't working either. They couldn't access the system locally.
That changed things.
I started asking the obvious questions. Was the server on? Were the lights on? Did it look like it was running?
They confirmed that it was powered on.
I didn't want to believe there was anything seriously wrong. I kept looking for an explanation that didn't involve the server itself.
So I asked them to connect a monitor.
Our bare-metal server was sitting there with a black screen.
And then, finally, through the blurry WhatsApp video call, I saw it.
Loading Linux 6.8.0-138-generic ...
error: cannot allocate kernel buffer.
Loading initial ramdisk ...
error: you need to load the kernel first.
God.
There is something uniquely uncomfortable about seeing a machine you depend on fail before the operating system even gets a chance to start.
I instructed them to try booting another kernel through GRUB. Maybe the latest kernel was broken. Maybe we could get in using the previous version and deal with the problem later.
Same error.
Both kernels failed.
At that point, I was in shock.
And, of course, it had to be Thursday
In Libya, Thursday has a very particular feeling. It's the last working day of the week for many businesses, a bit like Friday in much of the Western world.
People want to finish their work, close out whatever they have pending, and head into the weekend.
Nobody wants to spend that day staring at a server that refuses to boot.
And nobody wants to tell employees that the business system they need to do their work is unavailable.
I could feel the clock ticking. People had work to finish, and I didn't even know yet whether I was dealing with a routine boot failure or something much worse.
This wasn't just a machine I could troubleshoot whenever I got around to it. It was running business-critical systems, and I was suddenly trying to work out whether the data was safe.
Twenty-five minutes of worst-case scenarios
I got ready and headed to the office.
The drive was about 25 minutes.
Twenty-five minutes of thinking about every possible scenario, trying to reason through a problem I hadn't even properly diagnosed yet.
What if the NVMe had failed?
What if the filesystem was corrupted?
What if the machine wouldn't recover?
What if the data was gone?
And then the thought that made everything worse:
Our Google Cloud backup wasn't active yet. The subscription had been delayed.
The local backup, an HDD sitting alongside the NVMe, was supposed to be my last line of defense.
If the primary drive was dead, that HDD was what I had left.
You know that feeling when your brain starts debugging every possible disaster before you've even arrived at the machine?
That was the entire drive.
The 4 GB USB stick problem
I got to the office, ready to start recovering the server.
And then I realized something rather embarrassing.
I hadn't brought a USB stick.
So I had to ask someone to go out and get one from whatever stationery shop was closest.
Eventually, they came back with a 4 GB USB stick.
I already had an Ubuntu Server ISO, but it was too large for it.
Of course.
At that point, I didn't need a full operating system installer anyway. I needed something I could boot quickly, inspect the hardware, access the filesystems, and hopefully figure out what had happened.
So I had to find and download SystemRescue instead.
Eleven minutes.
Eleven minutes waiting for that ISO to download, followed by flashing it onto the USB stick. Somehow, those eleven minutes felt infinite.
Another little obstacle, another thing to solve before I could even start solving the actual problem.
When you're troubleshooting a production machine, the recovery environment is supposed to be one of the tools you already have ready. Instead, I was improvising with whatever I could get my hands on.
Eventually, I had a bootable SystemRescue USB. Wandered around in BIOS and UEFI settings to bypass the security layers and SecureBoot.
Finally, I could start investigating.
First, find out what actually failed
I booted SystemRescue and started investigating.
The NVMe was detected. The LVM volumes were there. The root filesystem was accessible. The kernel images and initramfs files existed.
Then I checked the drive's SMART report.
SMART overall-health self-assessment: PASSED
Critical Warning: 0x00
Media and Data Integrity Errors: 0
Available Spare: 100%
For the first time that morning, I had something reassuring to look at.
Those results don't guarantee that an SSD is flawless, but they didn't suggest the kind of drive failure I'd been imagining during the drive.
The problem was happening at the bootloader stage. GRUB couldn't allocate a kernel buffer, and the subsequent attempt to load the initial RAM disk failed because the kernel hadn't loaded.
Both installed kernels, 6.8.0-136-generic and 6.8.0-138-generic, were affected.
I didn't want to jump straight into reinstalling the operating system. I wanted to understand what had failed and recover without putting the data at additional risk.
SystemRescue gave me access to the existing installation. Eventually, I found an alternate boot route that got me further into the system.
The relief was real.
The NVMe wasn't reporting obvious media errors. The installation was accessible. The machine wasn't simply a dead box with an unreadable disk.
I could finally stop imagining the worst long enough to focus on the actual fault.
And then I checked the backup
There was still one thing I hadn't verified.
The local HDD backup.
It existed. The files were there. The backup system had been set up for exactly this kind of situation.
But the most recent backup was from August 31.
It was October 22.
Almost two months of potential changes weren't covered by that recovery point.
The Google Cloud backup was supposed to provide another layer of protection, but the subscription delay meant that layer wasn't active yet.
So the backup I had been counting on as my last resort was far older than I had realized.
That was a different kind of shock.
I'd started the morning afraid that the primary storage might have failed and taken the company's data with it. Instead, I discovered that the recovery strategy itself had a gap I hadn't fully appreciated.
The HDD was there, but its presence wasn't the same as having a current backup.
Having a backup isn't the same as being able to recover
It's easy to think about backups as a checkbox.
Do we have another disk? Yes.
Do we have a backup job? Yes.
Is there somewhere else to store the data? Yes.
But those answers don't tell you whether you can recover from a failure today.
A backup is only as useful as its recovery point, its integrity, and your ability to restore it when you need it.
A backup from August 31 is better than having nothing at all. But in October, it represents a very different level of risk than a backup that completed last night.
And if nobody notices that a backup has stopped updating, the problem can sit quietly in the background for weeks.
Everything appears fine until the moment you actually need it.
What I'm taking away from this
This incident has made one thing painfully clear: backup infrastructure needs monitoring and verification just as much as production infrastructure does.
The local HDD backup needs a reliable schedule, with visible success and failure reporting.
The Google Cloud backup needs to become operational, and its completion needs to be verified rather than assumed.
Recovery points need to be checked. Restores need to be tested. And a failed or stale backup should become an incident of its own, not something discovered during a completely unrelated failure.
The goal isn't to build infrastructure that never fails. That's unrealistic.
The goal is to make sure that when something fails, the damage is contained and recovery is possible.
This morning, I was lucky. The NVMe's health report was reassuring, the installation was accessible, and I found a way to investigate the boot failure without reinstalling everything.
But I don't want the next incident to depend on luck.
Because the most important thing I learned today wasn't about GRUB, kernel images, or Linux recovery.
It was that I had been treating a backup as my safety net without checking how current that safety net actually was.
The server going down was the incident. The backup from August 31 was the wake-up call.
A side note
Honestly, I wouldn't recommend waking up this way. One minute you're peacefully unconscious, the next you're mentally calculating how much of the company's data might have just vanished, wondering whether your backups are actually backups, and planning your own career funeral before you've even brushed your teeth or got dressed. All things considered, a bad dream would've been much more considerate.
Debuglore is where I document the failures, investigations, and lessons behind the systems I build and maintain. Not just what broke, but what it revealed about the way I engineer things.