Do you have a story to tell? April 27th is National Tell a Story day

In my first job out of college (early days of the dotcom; back in the 1900's) I was an analyst on a floor of about 40-60 UNIX developers, architects, analysts, etc. All working together on a big enterprise product®® with an early, yet ROBUST version control system.

Anyway, a friend of mine, entry level UNIX Build Admin was doing some maintenance and cleaning up directories - as one does. He navigated ../../../ <return> and without noticing where that landed him
typed

rm -f *.* 

There may have even been a -r in there but I'm not sure that mattered much because he was in the main root directory.
For anyone here who doesn't know command line unix...
     rm = Remove
     -f = 'force' (don't ask any further questions)
     star dot star = wildcard filename . wildcard filetype (all of them)

The effect was nearly instant - dev-gopher-heads popped up in all the cubes and the last 3hrs of that day was largely ineffective for most devs and *really busy* for the build team. Luckily the lead on that team was pretty much a rockstar and he was able to restore the entire build structure that was deleted up to within like an hour before the rm command. Nearly everyone was back and running right where they were by the next morning.

It probably could have been worse - what made it so memorable was how it was almost like a power-outage; so fast.

----
®® customizing the absolute bejeezus out of the thing...they may still be working on it honestly. I worked there for about 5 years of an 18month project that was probably only 1/2 way done when all the contractors (like me) were released one month after 9/11. Taught me a lifelong lesson about customizing enterprise software. 🤔🤔🤔

Thankfully, no one has given me the ability to cause outages, but I love these types of stories. I've noticed a shift online about highly public outages–sure, some jokes about blaming the interns, but also others sharing their mishaps in solidarity. 

 

Back in my marketing days... two months after I got hired full time from my internship... 

My biggest, most visible whoopsie was sending the previous product release email to all 180,000 people in the marketing database... instead of the new release update to the relevant customers. AND on that same morning, sent an email in German with a typo to investors and analysts.

Unfortunately, the product release was an operational email, so it did not include an unsubscribe link. We got flooded with (understandably) upset replies, and the CEO was furious about both. (Swiss startup that immediately regretted hiring marketers in San Francisco.) 

In my defense, I was woken up by the CEO and CMO calling me repeatedly at 6am on a holiday to send this out because my colleague went MIA. 

 

We all learned some lessons that week!

Setting up better systems: This was a "bus factor" situation, where I was scrambling in a tool I didn't know, with a workflow that did not make any logical sense, and following the docs is how I pressed the "wrong" buttons.

Testing: We didn't have a good testing procedure, so we didn't catch this in advance.

Email hygiene: There are laws about unsubscribe links, and email templates needed to have those as standard. I take those links seriously, and it was clear many others do too. 

Urgency: This was not an emergency. The only consequence of delaying the send by a day would have been... missing the CEO's arbitrarily set deadline. The rush caused a bigger mess to clean up.

Boundaries: I didn't set them, and was just hired full time from my internship, so I didn't think I could push back. That set a precedent, and later I wound up working late at night on Christmas Eve doing a banal task with that CEO. 

 

I'm still close friends with the employee who went MIA, and a decade later, I still tease him about this.