Data Is Good; Decisions Are Better
Years ago, I was working with a developer on setting up Google Analytics on a relaunched website. We were talking about what we were going to tag on the site, and he said, “I can get you as much data as you want, but you won’t actually use it all, so let’s figure out what you actually need.”
That conversation stuck with me for two reasons.
First, he was right. Despite my saying I wanted as much data as possible, the truth was, I wasn’t going to use everything in the dashboard. I was honestly probably only going to use 8-10 primary data points and then everything else was just interesting to know.
Second, the key point of focusing on data that I actually needed was an important qualifier for me to hear. Data for data’s sake doesn’t help. Data’s just a tool; on its own, it doesn’t do anything. But when used to make decisions, it can be very important.
For example, if you’re pricing your ads, it would help to have data on average expected impressions, click-throughs, viewability, etc. If you’re testing where to put an in-line newsletter sign-up, having bounce rate and scroll depth data could be helpful.
Data is good. There is no denying that.
But it cannot make decisions for you. And the issue with the proliferation of data is that people start to treat it less like a tool in your box and more like a crutch. People start to get paralyzed.
This data paralysis just fundamentally slows things down. Here’s an example using that in-line subscription module I mentioned above…
Let’s say you don’t have scroll depth data. How can you decide where to put the CTA? An “analytical” answer might be to first get scroll depth data, perhaps by acquiring a new tool, then analyze that data, and finally make a decision on where to put the call to action. That sounds smart. But it’s wrong.
This brings me to an important rule.
Pick the right data
In the above example, there is a multitude of data points that you could use. Here are just a few:
- Scroll-depth before bounce
- Total bounce
- PVs/visitor
- Conversion rate of subscription module
All of these are useful and having them doesn’t hurt, but the only one that actually matters is conversion rate. But this is where the paralysis comes into play. You can’t get conversion rate without having the module on the page, but you have no data to tell you where you should put it to begin with.
“How can I make a decision without having any data?” someone might ask.
If you don’t have data, you have to first stop and figure out what matters most to you. Once you have that, go and get the right data.
The only thing you really care about in the above example is how many people are subscribing to the newsletter. So, the conversion rate of the subscription module is the only thing that matters. Therefore, how do you get that?
The right answer is to throw the in-line CTA anywhere on the page. It doesn’t actually matter where because once you have it on the page, you’ll concretely know what’s working and what isn’t. From there, you can move it around and compare the conversion rate data.
How do you pick the right data? Start at the end and then work backwards. To do that, ask what the expected outcome is if everything was already working. From there, ask what is the actual piece of data you’d want. And then ask how to get that piece of data as quickly as possible.
This is a more straightforward approach than the alternative, which is trying to analyze disparate pieces of data and then making a decision. It might lead you to the same place, but it’ll be slower.
So why doesn’t this happen?
Technical debt
The development team is whom I have often butted heads with the most about moving quickly to capture data. And I don’t blame them. The scenario that I described above requires them to do suboptimal work that we might change. That’s annoying.
It can also be dangerous.
There’s a term in software development known as “technical debt.” This is the cost of having to do additional work fixing the mistakes that were made in an effort to be fast. That debt can quickly become a burden and slow things down even more than data analysis. This is rampant in media.
Think about doing a kitchen remodel. You hire a general contractor and they break the whole plan down for you into three things you can manipulate:
- Low Cost
- Quick Delivery
- High Quality
You can only have two of them. If you’re willing to pay whatever it takes, you can get better materials and hire more people to work on the project. If you’re going to be cheap about it, but still want amazing materials, it’s going to take a lot longer to get done.
In media, many publishers don’t have a ton of budget to work with. That means engineering teams don’t get to choose. And since there’s more work than resources, these teams are forced to move quickly. Therefore, quality suffers. You accrue technical debt. Shortcuts happen.
But not all shortcuts are created equal. We have to strike a balance technical debt and business needs. In the above example, throwing an inefficient CTA on a page might require rework, but it’s also the best and quickest way to get the right data. Rather than spending more time working on a perfect solution, get an imperfect one up and then iterate. It might take more total time, but it’s a higher quality of time.
Here’s the real problem with technical debt for media companies. Because there is often more work than resources to do it, publishers never pay back the debt. And this is why developers get so damn angry. Technical debt makes their lives miserable because they have to maintain inefficient systems. And then things get built on top of inefficient systems. And before you know it, you’ve got a Frankenstein tech stack.
Anytime you’re sacrificing quality for cost and delivery, you have to then make a concerted effort to go back and fix things once the tests have been completed. Otherwise, interest (aka, developer resources) is charged against that debt. Nothing is free in this world. But at the same time, developers have to understand they are building for the growth of the business, not to always have perfect code. Sometimes we accrue debt; we have to get comfortable with that.
Data is good, decisions are better
Because of how much data is available, people have become increasingly dependent on it. It has gotten to the point where some people feel like they answer to the data. “Well, the data doesn’t say we can do that.”
The data doesn’t say anything. The data is just data. You have to interpret the data and decide what to do from there.
This brings me to the final point… When stuck between whether or not to make a decision, I find action is the right choice. But there’s a framework to do this better. For this, I turn to Jeff Bezos’ shareholder letter published in 2016:
One common pitfall for large organizations – one that hurts speed and inventiveness – is “one-size-fits-all” decision making.
Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don’t like what you see on the other side, you can’t get back to where you were before. We can call these Type 1 decisions.
But most decisions aren’t like that – they are changeable, reversible – they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.
Too often, we treat every decision like there is no going back. But the reality is that if we throw up the lead capture form and we hate it, we can always remove it. We can move quickly with imperfect data, gather the right data, and then continue making decisions from there.
Data is a tool, but at the end of the day, you have to decide. Don’t let the lack of data paralyze you from moving forward, so long as there’s a door back.
Thanks for reading today’s newsletter. Let me know your thoughts either by replying or by joining the AMO Slack channel. Have a great weekend!
