Yesterday morning at 6am, after my pounding ride on the Peloton (7-week streak, 5-days a week so far this year), I reached for my phone and discovered many a missed call from Ryan. Oh dear this can’t be good.

Ringing him back, camera off (as no one needs to see me in that post ride state – Ryan said later “I’ve seen worse”), I learn that one of our databases had gone offline and automatically failed over to the secondary backup.

My wife, says to me “Ryan needs to ring with good news in the morning every so often”.

Looking at her in disbelief, “a database has fallen over!!” I answered with a WTF look in my face.

Not missing a step, she retorts “well just pick it back up. Maybe put one of those necklaces around it to detect falls”. At this point, I realized she was teasing me, given she knows exactly how this world works. It was the light relief that was required.

All joke aside, we actually do have our “fall detection necklaces” around all of our infrastructure, and because of these alerts, we know when things start to go off balance. Monitoring is a tricky beast and often over looked, with the customer calling in “you guys having a problem?” being the real alert mechanism.

When Ryan and I were doing a due-diligence engagement, one of the questions we would ask during our time with the engineering team “how do you know, now, that everything is operating normally?“. It was a question that would often be met with fear as they process the fact, that they didn’t.

Monitoring seems easy, but it isn’t. It requires effort, consideration, and if you are looking after a system that isn’t of your making, you first need to learn its normal rhythms of peaks and lows. Knowing when to sound the alarm is an art. We’ve all walked past a car-alarm blaring without a second glance, numb to the racket, relegating it to an annoyance.

If you are not careful, then all those email/text/slack alerts can go unanswered if they are firing off for the wrong reasons. You get alarmed-fatigued, numb to the alarms.

Make sure your “fall detection necklace” is actually doing its job – so when you get the call at 6am in the morning, you know to really answer it (camera off).

AI Disclaimer: Gemini Nano Banana Pro was used to generate the photo – from the ’84 Ghostbusters with me as Bill Murray