Leftover Takeout

Some Thoughts About A/B Testing

At lunch today, Andrew reminded me of a story that I had long forgotten.

When he was at USV, he attended a board mtg for a portfolio company that we have in common.

It was the early days of that company and their initial product hadn’t been out that long.

The topic of A/B testing came up and Andrew reminded me that I came out against A/B testing in that meeting.

The sad but true part of todays conversation is that I don’t recall that meeting or that particular conversation.

But I would say the same thing to any early stage company with their initial product.

When you have thousands or even tens of thousands of active users, the product comes from the founders vision and desire. It doesn’t come from a focus group or some survey. And it didn’t come from an a/b test of some landing page or the home page.

It comes from the heart.

a/b testing in the early days is never going to be as good as pursuing the founders vision for making the product everything that it’s supposed to be. Fix those annoying bugs. Make it faster. Make it easier. Make it right.

Now, that doesn’t mean you shouldn’t listen to your users. Their feedback is vital. We all know, as a perfect example, that may of Twitter’s best features came from their users (e.g @mentions and retweets).

But a/b testing is optimizing around the edges around a terribly small group in the early days. It’s not worth the energy in my opinion.

As a web service grows significantly, a/b testing goes from not helpful to critical. Analytics, a/b testing, linking new features to metric driven objectives become an essential tool in the arsenal.

gbattle sez:

I respectfully disagree with you here Bijan, but it may be more in method than spirit.

For one, A/B or split testing is just that, a test of a hypothesis, a scientific assessment of the founder’s ability to express his or her vision through the lens of customer behavior. It is not a focus group. It is not a survey. It is not a conversation. You are watching what people do and quantifying it, not listening to what they say.

Split testing provides data that supports or refutes a hypothesis via the purposeful segmentation of customer experience to generate data. What you do with that data is another matter entirely. One can optimize or overfit to a split tested model just as readily as one can become entrenched in the myopic hubris of his or her heart and gut. The real questions are, how informed was the founder’s final decision and, over time, what is the founder’s predictive capacity with respect to his or her heart/gut/instinct? It is in these two questions that data becomes a key factor, but not an exclusive one.

Maybe I’m missing something here, but I don’t understand how one can be an advocate for listening to early stage qualitative user feedback (comments, suggestions, “get out of the office”, etc.) but not an advocate for tracking and testing those qualitative outcomes for quantitative feedback to fuel the iteration of product, exactly what A/B testing was built to help accomplish.

If I replaced “A/B testing” with the term “analytics” or just “gathering data” in your post, I’m not sure you’d feel as confident about devaluing their importance for early stage startups. The scoreboard doesn’t lie, and founders must choose which metrics to listen to and which to ignore at any stage. Making data collection and data analysis part of your DNA early on isn’t a waste of time. I’d argue it’s easier to form that culture more earlier than later.

I get that you want early stage founders to avoid analysis paralysis, blind optimization and wasting precious cycles and resources with little return value, but I think we’d agree these afflictions can apply to any company independent of growth phase.