Measuring What Matters Before and After

When you make a change to your ecommerce setup whether it’s a UX update, an app integration, a new product bundle layout or a whole new checkout flow the instinct is often to launch, move on, and hope the numbers follow.

But if you’re not measuring what happens before and after, how do you know whether the change had the intended outcome?

We see this a lot. A system change gets made. It works well technically. Everyone’s happy with the look and feel. But there’s no baseline, no follow-up, and no clear KPI tied to the reason it was done in the first place. So, it’s difficult to answer the question “Did that actually help?” .

The problem often isn’t getting access to the data, but the lack of clarity around what the change was supposed to achieve and how to measure it.

Measuring data before and after a change doesn’t need to be complicated. It just needs to be thought about early, and ideally before the work starts. Big changes, especially at a system or platform level, should deliver more than a cleaner user experience. They should support your business growth.

That’s where clear, job-specific measurement comes in. Not every change is trying to achieve the same thing.

Not all metrics are created equal

Historically, the default has been to measure whether the conversion rate has gone up. But what if the work wasn’t really aimed at that?

If you're a subscription brand preparing for acquisition, your goal might not be short-term conversions, it might be increasing customer lifetime value (LTV) or retention rate. If you're testing a new product category, maybe you're more interested in repeat purchase.

Or maybe the goal is to collect more direct customer data to help you understand how the change affects user behaviour across different parts of the site. If you're expanding into new markets, your focus right now might be on engagement and brand stickiness rather than profit.

In other words, the "why" behind a change should shape the "what" you measure.

What should you measure?

It depends on your specific situation. Here’s what some of our clients might look at in practice:

• Homepage rework?

  Look at engagement with featured collections, scroll depth, and bounce rate from key entry points.

• Subscription flow update?

  Track how users engage with different subscription options, where they drop out during sign-up, and how many stay subscribed after 30 days.

• Navigation changes?

  Check product discovery rate, click depth, and how many sessions result in a product view.

• Introducing a loyalty scheme?

  Focus on repeat purchase rate, points redemption behaviour, and average time between orders.

The point is that every change should have an intended impact, and you need a way to measure whether it’s actually achieving that goal.

The missing link: reporting

Many brands invest time in optimisation and UX improvements. But they don’t have a consistent way of tracking what happened after. That’s not a failure of intent. It’s usually a lack of process.

We’ve seen it often: a project is signed off, pushed live, and then the team moves straight onto the next thing.

That’s understandable. But over time it leads to a disconnect between effort and outcome. If you don’t check the results, you can’t learn what worked - or what didn’t. And you can’t tie progress back to business goals or your investment in the change.

So our recommendation is simple: build reporting in from the start.

• Decide what success looks like

• Choose the KPI that reflects that

• Set a realistic timeframe

• And agree how you’ll track it, even if it’s just a manual check-in a month later

It doesn’t need to be complex. It just needs to be intentional and meaningful.

One job, one goal

Not every project will affect every metric, and it doesn’t need to. If a system adaptation only impacts mobile users, then you only need to track mobile behaviours. If it’s only relevant for new customers, don’t put returning users in the mix.

The tighter the focus, the more valuable the data insights.

And remember: different cohorts care about different things. First-time buyers vs. subscribers. UK vs. international. Bargain-hunters vs. repeat customers. When you define your audience clearly, it’s much easier to spot, measure and understand real change.

Our take

We believe in optimisation that’s measurable, meaningful, and firmly grounded in commercial goals. You don’t need to track every possible data point possible. It’s about tracking the right metrics, for the right reason.

So next time you’re making a change, ask:

• Why are we doing this?

• What are we expecting to happen?

• Who will it affect?

• And how will we know if it’s worked?

If those questions are clear from the start, the numbers will tell a much more useful story at the end.

Are you looking for clearer ways to measure the impact of your ecommerce changes? We help teams build testing and reporting into their workflow, so meaningful insights become part of the process, not an afterthought.