The Mistakes We Keep Seeing After Companies Buy Experimentation Software

The Tool Nobody Turns On

A company spends weeks evaluating experimentation software, sits through the demos, wins budget approval, and signs the contract. Three months later, nobody can say whether a single test has shipped.

This outcome is common. Vertice analyzed more than $75 billion in tracked enterprise software spend and found that 65% of SaaS licenses were unused or underused as of 2026, up from 62% a year earlier. Another 14% went completely untouched.

Experimentation software carries the same risk, and arguably more of it. People open a project-management tool to check a task.

Nobody opens a testing tool by accident. If no one configures it, owns it, or checks it against the site's real traffic, it can sit idle indefinitely and renew quietly every year.

We see this pattern constantly in audits, and it happens before any of the execution problems we have written about elsewhere.

A team can run disciplined hypotheses, sound sample sizes, and clean implementation, but that discipline never pays off if the tool never switches on.

What Actually Happens in the First 90 Days

Software rarely causes the failure. Usually, one or more of four specific gaps opens in the weeks after the purchase decision, before anyone runs a test.

No one owns it: Whoever championed the budget request, usually a marketing or growth lead, buys the tool and then hands it off without naming anyone to run tests. Everyone assumes someone else is driving.

Nobody checks traffic until after signing: Our own research into why testing programs fail found that roughly 1 in 6 eCommerce stores we audit lack the traffic for statistically valid A/B tests, using a benchmark of about 15,000 monthly visitors or 300 monthly orders.

Teams should run that math during evaluation. Too often they run it months into a subscription they use badly.

No first-test deadline exists: Without a date attached to "ship the first test," configuration work competes with every other priority on the team's plate and loses. Weeks become a quarter, and the quarter ends in a renewal decision nobody wants to make.

Configuration goes to whoever has time, not whoever owns CRO: A developer with a free afternoon installs the snippet and leaves the rest half done.

Goals stay undefined, and segments stay unbuilt. The tool runs, technically, and produces nothing useful. No demo reveals these four gaps.

They surface 90 days later, when someone asks what the tool has produced.

Introducing the Experimentation Activation Score™

We built a diagnostic for this exact window, the same way our AI Pick Score™ and Snippet Drag Score™ measure other specific mechanisms.

This one measures whether a team will actually use a newly purchased experimentation tool before a single test runs.

We use this as our own internal diagnostic, and we don't claim that measuring software adoption is new.

Product-led-growth teams have tracked generic "activation scores" for years. Our version ties the idea to the four gaps that derail testing programs, and it builds its bands around a real first-test deadline instead of an arbitrary usage metric.

The score runs from 0 to 100 and draws on three weighted inputs.

Ownership Clarity (35%): asks whether one named individual, rather than a team or department, has explicit responsibility for running tests through the tool. A tool with no named owner scores zero here, whatever else is true.

Traffic & Eligibility Readiness (35%): asks whether anyone checked the site's traffic and order volume against the tool's statistical requirements before signing, or whether the team discovered the problem afterward.

Time-to-First-Test (30%): counts the days between contract signature and the first live test. The longer that gap stretches, the more the tool looks like shelfware to everyone watching the budget.

80 to 100, Activation-Ready: Someone owns the tool, the team checked traffic in advance, and a first test is live or has a committed date within 30 days.

50 to 79, Stalled: The team misses at least one input, usually ownership or a deadline. The subscription runs while the testing program sits dormant.

0 to 49, Shelfware Risk: The team misses two or more inputs. In our experience, a tool at this band rarely ships a first test without an outside push.

Translating a Low Score Into Wasted Spend

A Stalled or Shelfware Risk score costs money, and the example below is illustrative, not a measured case.

Suppose a company's experimentation tool falls into the 65% that Vertice classes as unused or underused. Every month without a first test costs a month of subscription fees and produces nothing measurable.

Testing software adds a second cost on top of that: competitors who did activate their programs keep compounding what they learn, while the stalled program learns nothing.

We want to be precise about what this math shows. Vertice published the 65% underuse figure and the 14% fully unused figure for SaaS broadly.

Applying those patterns to a specific company's experimentation tool is a reasonable extrapolation, not a result anyone measured for a particular tool or client.

Why This Sits Before the Decision Lifecycle™

Our research on why most A/B testing programs fail draws on 1,103 real client experiments and documents what breaks once a program is running: unshipped winners, missing roadmaps, and proxy metrics that stand in for revenue.

That research introduced the Convertcart Experimentation Decision Lifecycle™, a six-stage model that starts with choosing what to test.

The Experimentation Activation Score™ doesn't compete with that framework. It measures the stage before stage one.

A team can follow the Decision Lifecycle™ perfectly, and a perfect process still produces nothing if the tool sat unconfigured for four months first.

The two diagnostics run in sequence by design.

What This Looks Like in Practice

We have no published case study that isolates activation. Our existing case studies document optimization results after a testing program was already running, not the rollout period before it, so we won't force a number we can't back up.

We can describe what we see across our own audits.

The four gaps in this post (missing ownership, late traffic checks, no deadline, rushed configuration) show up repeatedly in accounts that come to us after a prior testing tool stalled. That is a consistent pattern we observe, not a measured, citable statistic, and we want to keep that difference clear.

Closing the Gap: How to Activate Experimentation Software Correctly

Most of this work happens before the first test runs, and it costs little next to the subscription it protects.

Name an individual owner while you are still negotiating the deal, not after you sign. A tool without a named owner on day one tends to stay that way.

Run the traffic math during evaluation, not three months into the subscription. Our own research puts the floor at about 15,000 monthly visitors or 300 monthly orders, so measure your site against that.

Set a first-test deadline within 30 days of signing, and treat it like any other launch date on the team's calendar.

Give configuration work to whoever owns CRO, not whoever has a free afternoon. Someone who understands what the tool should measure needs to set up goals, segments, and tracking.

Revisit the Activation Score at 30, 60, and 90 days. If the score still reads Stalled on day 90, escalate it instead of waiting for renewal.

So, What's the Bottom Line?

Buying experimentation software was never the hard part. Activating it is, and that work lands in the 90 days nobody budgets for.

Vertice's data shows 65% of SaaS licenses going unused or underused, and testing tools give us no reason to expect better.

That is an ownership and deadline problem, not a software problem, and teams can fix both before they waste a dollar of subscription cost.

If you already run a testing program and want to know what breaks next, our research on why most A/B testing programs fail makes the natural next read.

If you'd rather skip the activation risk entirely, CRO360 runs testing as a managed service, so you have no internal tool to activate.

Want to know where your own testing program stands? We'll score it against all three inputs and tell you exactly what's missing.

A Few FAQs About Activating Experimentation Software

1. Why do companies buy A/B testing software and then never use it?

Usually because nobody took ownership, nobody checked traffic eligibility before the purchase, and no deadline forced a first test out the door. The tool itself rarely causes the problem.

2. How much SaaS software goes unused?

Vertice's 2026 data puts the share at 65% of licenses idle or barely used, with 14% fully untouched. Testing tools likely follow a similar pattern, though those figures cover SaaS broadly.

3. How much traffic do I need before buying an A/B testing tool?

Aim for at least 15,000 monthly visitors or 300 monthly orders. Below that, tests routinely return inconclusive results no matter how good the tool is, so check this before you sign.

4. What's the difference between the Activation Score and the Decision Lifecycle framework?

The Activation Score looks at the weeks before testing starts: ownership, traffic readiness, and time-to-first-test. The Decision Lifecycle covers what happens once testing is underway. One comes before the other.

5. How long should it take to launch a first A/B test after buying the software?

Aim for 30 days or less once you have confirmed ownership and traffic eligibility. Past that point, the tool starts drifting toward shelfware regardless of its capability.