BAM. BOOM. AARGHH. F*&^. PHEW. SIGH. SMILE……….. Everyone’s Been There Before

Screen shot 2014-01-26 at 9.16.27 PMAt the end of yesterday, I connected with friends I hadn’t been in touch with for far too long.  I scheduled lunch with former CNET CEO Shelby Bonnie in a few weeks. My Twitter followers increased.  We added new users.  Many of my friends logged in and sent me notes about how blown away they were with what we had built since buying back  Turns out I’ve got a lot of good friends who have our back.  I learned a lot about our team, marveled at their maturity and skill in the moment of crisis and fell in love with them all over again. But that’s not the way the day started……

It was a stunning, peaceful Saturday morning – the kind of morning when the heat is dry and the sound is very still (sorry to those of you who live in the brutally cold East Cost weather but this is San Francisco in late January).  After a morning run in the Presidio, I made myself some Blue Bottle Coffee in my Chemex, hit play on John Coltrane’s Giant Steps and settled in to play around with one of our newest features that enables users to invite friends to join our service.

Screen shot 2014-01-26 at 9.02.34 PM

I clicked send.  Nothing happened.  So I clicked it one more time and it wasn’t clear anything happened. Hmm.  So I decided to send my team an email to see what was broken. When I opened my email, literally 2-3 minutes later, I had a crazy number of unread emails flowing in (like “I love you, man, but I’ve gotten ~30 of these so far….”). It’s just awesome to getting an email from someone you admire like Marc Andreessen letting you know you’ve sent them 20, oops, actually 24 emails in the past few minutes.

Screen shot 2014-01-28 at 5.09.08 PMAnd then Twitter alerts started popping up on my screen (and each time I clicked, I had less followers than I had minutes before).  My phone rang.  It rang again.  And again. And then iMessage started popping up alerts (like “Please, like really please make it stop”).  And then I had that “aw f*&^” moment when I realized I’d jammed all my contacts with messages, more messages and even more messages.  I had triggered a weird bug because my email contact list is quite large and it would time-out, and then re-activate, creating a continuous loop.  It also somehow tied into my WordPress blog which was a separate issue (to be clear, not a WordPress issue but one on our end) that’s too complicated to explain.  NTL, I was on my way to sending 47,402 emails in a span of 4 minutes, an average of 23.6 emails per person. BAM. BOOM. AARGHH. F*&^.


Screen shot 2014-01-26 at 9.01.01 PM

I sent out an urgent note to our team.  My co-founder Ryan, calm as always, called and told me to 1) make my tweets private; 2) start deleting all the repetitive tweets; and 3) disconnect my WordPress blog from the automatic tweet feature.  On his end, he disconnected the feature we were testing.  I then spoke to Matt, Founder of WordPress, who calmly told me where to go to disconnect the automatic tweet feature.  Just in case you missed it, there’s a theme in there, they were calm, I wasn’t.

Afterward, it stopped.  Kinda.  I watched as my Twitter followers continue to melt away, I lost about 400 in a few hours.  I turned my attention back to email and oh my, so many messages, many from me :).  I wrote a simple acknowledgement and apology response, copied and started pasting it in reply after reply after reply.  Then I went back to Twitter, announced “It’s just a hunch but I think there are a lot of people who are going to delete me from their address book.”  Then I let everyone on Twitter know “oy, even my family deleted my account.”  And then, people on Twitter started interacting, retweeting and favoriting those tweets and voila, I started adding new followers. And people invited friends to join and many of them logged in and updated their pages.

Screen shot 2014-01-26 at 9.49.36 PM Screen shot 2014-01-26 at 9.52.02 PM
























Meanwhile, the amazing team started doing forensics around how many emails were sent to individual addresses in order to get a pattern to any duplicate sends.  We exchanged inputs in real time (exactly how did we do this before we had tools like Slack!?!) and figured out the root cause and, more importantly, how to make sure that didn’t happen again.

I read the hundreds of email replies to my apology and this is what was said more often than anything else “dude, don’t worry, everyone’s been there before.”  It reminded me how amazing our community can be in a moment of crisis. We are lucky.

And then we all went back to our beautiful, peaceful Saturday. BAM. BOOM. AARGHH. F*&^. PHEW. SIGH. SMILE.

3 thoughts on “BAM. BOOM. AARGHH. F*&^. PHEW. SIGH. SMILE……….. Everyone’s Been There Before

  1. Like Josh, I’m a bit disappointed that I wasn’t in the list 🙂

    Generally, it does happen to everyone, and it did happen to me (not that specifically, but similar issues!) and I found that I’m a LOT less aggressive when this happens to a friend of mine, because I understand how they feel and it’s also obvious that it’s a bug and not something that they meant to do!

  2. Tony

    Well we’ve all had stuff explode on us – the worst thing is if you don’t know that it’s exploded and I suspect quite a few people, including me, emailed you to see if you knew 🙂

    And like others, I actually logged in to both of my accounts to tweak stuff (and discovered an interesting “catch 22” issues in the process)


  3. Sometimes an error is an opportunity to look at the issue in a new way and ignite conversations I suppose. YA and I really like ABOUT.ME…gives me the opportunity to connect with some amazing people……and “Luck is where opportunity meets preparation” Denzel Washington

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s