At the end of yesterday, I connected with friends I hadn’t been in touch with for far too long. I scheduled lunch with former CNET CEO Shelby Bonnie in a few weeks. My Twitter followers increased. We added new about.me users. Many of my friends logged in and sent me notes about how blown away they were with what we had built since buying back about.me. Turns out I’ve got a lot of good friends who have our back. I learned a lot about our team, marveled at their maturity and skill in the moment of crisis and fell in love with them all over again. But that’s not the way the day started……
It was a stunning, peaceful Saturday morning – the kind of morning when the heat is dry and the sound is very still (sorry to those of you who live in the brutally cold East Cost weather but this is San Francisco in late January). After a morning run in the Presidio, I made myself some Blue Bottle Coffee in my Chemex, hit play on John Coltrane’s Giant Steps and settled in to play around with one of our newest features that enables about.me users to invite friends to join our service.
I clicked send. Nothing happened. So I clicked it one more time and it wasn’t clear anything happened. Hmm. So I decided to send my team an email to see what was broken. When I opened my email, literally 2-3 minutes later, I had a crazy number of unread emails flowing in (like “I love you, man, but I’ve gotten ~30 of these so far….”). It’s just awesome to getting an email from someone you admire like Marc Andreessen letting you know you’ve sent them 20, oops, actually 24 emails in the past few minutes.
And then Twitter alerts started popping up on my screen (and each time I clicked, I had less followers than I had minutes before). My phone rang. It rang again. And again. And then iMessage started popping up alerts (like “Please, like really please make it stop”). And then I had that “aw f*&^” moment when I realized I’d jammed all my contacts with messages, more messages and even more messages. I had triggered a weird bug because my email contact list is quite large and it would time-out, and then re-activate, creating a continuous loop. It also somehow tied into my WordPress blog which was a separate issue (to be clear, not a WordPress issue but one on our end) that’s too complicated to explain. NTL, I was on my way to sending 47,402 emails in a span of 4 minutes, an average of 23.6 emails per person. BAM. BOOM. AARGHH. F*&^.
I sent out an urgent note to our team. My co-founder Ryan, calm as always, called and told me to 1) make my tweets private; 2) start deleting all the repetitive tweets; and 3) disconnect my WordPress blog from the automatic tweet feature. On his end, he disconnected the feature we were testing. I then spoke to Matt, Founder of WordPress, who calmly told me where to go to disconnect the automatic tweet feature. Just in case you missed it, there’s a theme in there, they were calm, I wasn’t.
Afterward, it stopped. Kinda. I watched as my Twitter followers continue to melt away, I lost about 400 in a few hours. I turned my attention back to email and oh my, so many messages, many from me :). I wrote a simple acknowledgement and apology response, copied and started pasting it in reply after reply after reply. Then I went back to Twitter, announced “It’s just a hunch but I think there are a lot of people who are going to delete me from their address book.” Then I let everyone on Twitter know “oy, even my family deleted my account.” And then, people on Twitter started interacting, retweeting and favoriting those tweets and voila, I started adding new followers. And people invited friends to join about.me and many of them logged in and updated their pages.
Meanwhile, the amazing about.me team started doing forensics around how many emails were sent to individual addresses in order to get a pattern to any duplicate sends. We exchanged inputs in real time (exactly how did we do this before we had tools like Slack!?!) and figured out the root cause and, more importantly, how to make sure that didn’t happen again.
I read the hundreds of email replies to my apology and this is what was said more often than anything else “dude, don’t worry, everyone’s been there before.” It reminded me how amazing our community can be in a moment of crisis. We are lucky.
And then we all went back to our beautiful, peaceful Saturday. BAM. BOOM. AARGHH. F*&^. PHEW. SIGH. SMILE.