Three Ways The Tech Industry (Still) Misuses Metrics
By Ash Huang
As one designer puts it, “It’s easy while moving fast and breaking things to trust data blindly.”
Editor’s Note: This story contains one of our 10 Best Business Lessons of 2016. Click here to read the full list.
I once had a project manager ask me to redesign a home page. Without going over why we were redesigning it, he launched into his plan of how it all should go: Each section’s size would correspond with what percentage of clicks they received on Google analytics. The feature header would take up exactly 63% of the page, the sidebar 21%, and everything else would be divided up with the remainder.
After screaming silently to myself inside an empty conference room, I kindly suggested to him that we design the page based on how we wanted users to act.
Measuring things is a vital part of developing a successful product. By extension, it’s crucial to everything the technology industry does. And it helps explain why we seem so eager to debate the relative business merits and drawbacks of data.
It’s true that measuring things properly can show us smart ways forward and monitor the health of a system. But it’s no less the case that hailing quantitative data as the best or only answer is a sweet, sweet honey trap that can lead to bizarre and ill-informed decisions–and one that continues to ensnare the very tech companies and professionals at the front lines of data innovation. Here’s how.
In 1999, NASA famously crashed a $125 million spaceship because its scientists accidentally used both metric and English units in their programming. If literal rocket scientists can make such huge mistakes, then just about anybody can mess up a line of tracking code, no matter how smart or experienced they are.
Tech has a history of mistrusting qualitative data like user interviews and experiential information. It can seem soft or illogical, or have too small a sample size. However, we need to regard quantitative data with the same skepticism and double-check those results with the same level of scrutiny. A friend of mine ran an A/B test sending the exact same email to each group, and one variant received 5% to 6% more clicks.
It’s easy while moving fast and breaking things to trust data blindly. Any spike or drop we see in data should immediately prod us to ask, “Can that be true? How can we verify or disprove this?”
Companies also make the mistake of prioritizing short-term gains over long-term ones–sometimes heavily. A common modern example is the sign-up wall. Go to almost any blog or e-commerce site and you’ll be bombarded with an overlay modal. The main messaging is some variation of “Give us your email and let us email you until the end of time,” along with a smaller “No, thanks, I hate saving money.”
Fast Company contributor Brian Scordato recently pointed out the staggering illogic of this particular fad:
The retailer has done something monumental to get your attention: Of all the millions of things you could be doing, by some stroke of acquisition genius, you’re on their site. You’ve handed them a megaphone and allowed them to put it to your ear. It’s their one opportunity to prove they know the person on the other end . . . to prove they know what’s important to you.
And what they’ve decided is important to you is price.
While I’m sure this tactic gives some businesses a bump in newsletter sign-ups (why else would they do it?), no user walks away from that interaction thinking, “What a lovely company. I’d better tell all my friends right now!” Meanwhile, creating valuable content, while more time-consuming, might result in better retention–not to mention users who will actually turn into paying customers.
Quantitative data tells us “how much,” but it rarely tells us “why.” It doesn’t help us identify new opportunities the way qualitative user studies (still) do. I’ve worked with plenty of companies that bemoan a low retention rate post-sign-up. But after talking to users, they’re usually surprised to find that people often bounce because they misunderstand what a product does, and how it would add value to their lives–something the business never actually communicated in its rush to offer a 15% discount.
This realization can alter company’s priorities drastically, which has direct and powerful consequences for the ways websites, apps, and tech products are designed–whether that means changing button colors or designing flows that gives people the right information at the right time. In my experience, dismissing qualitative data can focus whole teams on 1% to 2% bumps when a systemic change could move the needle 20% or more.
It’s easy to chase arbitrary metrics. We’re only human–tech professionals included–so playing the “make the number go up” game soothes our starved and hoarding lizard brains. The best metrics, though, are gathered as evidence of goals.
For instance, a common metric companies track is how long a user spends on each session. Products are fighting for users’ attention, and often each display ad that scrolls by means more money in the bank. It’s simple and seems to accurately indicate user interest.
But what if we were Google?
Google search’s magic is that it sifts through the entire Internet and brings you that one special search result. If it’s working correctly, a user will actually spend very little time on a results page. But they may come back and search over and over again. If we only measured how long a user spent in product, we would never get the whole story.
In fact, we’d be dangerously tempted to de-prioritize search-result quality and instead design the page as an information casino to keep eyes on it as long as possible–because that’s what the data incontrovertibly pointed to.
The most sustainable growth strategy is to build a compelling, simple product and a lively, valuable community therein. Part of Pinterest’s early success was the blogger crowd that Ben Silbermann cultivated, often one-on-one. Early Twitter was a bizarre and devoted community of people who collectively invented the hashtag and the @-mention:
I’m hardly the first tech professional to argue that data is only a tool and needs to be treated that way. But that plea for balance usually comes from outside the tech sector, from industries where qualitative measures have long been prized and metrics kept at arm’s length. Neither are the ideal attitude. But if the tech industry really wants to remain the vanguard in innovation that it sees itself as being, the task of putting data back in balance with the human touch has to start here.
Early Twitter might’ve got this intuitively right, but it’s something the industry is still learning. If the recent controversy over the way Facebook surfaces news stories in its Trending Topics feature is any indication (originally, human “curators” were reportedly tasked with training an algorithm), we still have a way to go.
The good news? We may not need new heaps of data to help us get there. We’ll just need people.