Of course I struggeled on the second day to write a blog post. The whole universe aligned to make it hard on me ;)
No kidding, this day was rough. Getting woken up at 3am after only two hours of sleep by a couple hundred failed system checks is nerv wrecking. Especially if you could have mitigated most of it. But this is what happens if you don’t properly sort in announced maintenance windows of your datacenter provider and network equipment gets updated.
It would have been ok if the replication on two databases didn’t break on top of that. This created another huge pile of work. Luckily everything is alright again and our lovely customers weren’t affected.
Tonight there is another maintenance window with the next batch of routers getting updates. Not a chance I’ll be surprised again.