When Data Is Wrong

I love data. When I was a business analyst at Microsoft, I enjoyed unearthing insights that moved the business forward. But data worries me too – it can be misunderstood, and it can be misused.

Sometimes data points to an obvious direction that may be incorrect. For example, I believe the quality of movies has deteriorated since studios started making changes to movies from audience surveys. The data is used to convince directors and writers their instinct is wrong. But what people say is the best ending may actually leave less of an impression – the result is less memorable movies, and ultimately people not wanting to go to the theater any more. Meanwhile, the studios look at data showing a decline in theater attendance and attribute it to home theater systems. And so, they turn to surveys even more.

While data is key to the success of any startup, it can also lead you astray. When we launched the alpha of LazyMeter, reviews were mixed – while 50% loved it the other 50% couldn’t figure out how to use it. The data told us that an analogy to a music player we were using didn’t make sense for a to-do list. The clear suggestion was to change from iconography to text; for example, to change our play button to the text “Today”. We resisted because the iconography is the point of the product – it’s what enables us to make task management faster than pen and paper. Switching to text also would have made us appear like the many other task managers we sought to replace. We followed our gut, not the obvious action suggested by the data, and went about finding another way to improve the experience for new users.

We added documentation. We played with tooltips. We tweaked the active and inactive states for the navigation buttons. All efforts improved our adoption rates, but they still left much to be desired.

And then we realized something. The problem was not that people didn’t understand how to use our product. The problem was they didn’t understand what the product was doing for them. We weren’t explaining our product on our homepage. ‘Play means today’ makes a lot more sense if you know LazyMeter is your to-do list, one day at a time. We did a simple redesign with a wordpress template, focusing primarily on messaging. And we haven’t heard from a confused user since. Responses immediately went from “I don’t get it” to “this is simple and intuitive”. Registration and adoption rates went through the roof – without any major changes to the usability of the product.

While it was critical to have data showing us where users were getting stuck, the obvious suggestion to change the usability would have been a mistake. It’s important to realize data can paint the wrong picture. If it feels wrong, dig deeper.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: