Skip to content
EN
BOOK A CALL
Conversion Rate Optimization

A/B Testing Services: Test Design, Sample Size and Analysis

The hard part of A/B testing is not setting up the tool; it is making sure the result is not down to chance. Tests stopped early, run on too little traffic or overlapping with a sale week declare a 'winner', and the change that ships then delivers nothing. Our A/B testing service starts with a traffic calculation, and we write down the duration and success criteria before the test goes live.

Book a growth call Read the details
your-website.comBuy nowA · currentyour-website.comBuy nowB · new
Illustrative view: illustrative view of two page versions in an A/B test.
Purpose and appearance

What do we use this service for?

We run A/B tests to prove that a change to the site actually increases sales before rolling it out to everyone. Decisions on design and price display rest on measured results, and changes that would lose sales are caught on a small share of visitors first. Every result feeds the hypothesis for the next test.

Funnel stages it serves

  1. Awareness
  2. Consideration
  3. Conversion
  4. Loyalty

How to calculate A/B test sample size

Three values determine how much traffic a test needs: the current conversion rate, the smallest difference we want to detect, and the margin of error we accept. For example, on a page with a 2 percent conversion rate, detecting a 10 percent relative lift (from 2 to 2.2 percent) at 95 percent significance and 80 percent power requires roughly 80,000 visitors per variant.

This calculation often changes what gets tested. If traffic is not enough, instead of small changes such as button color, we test changes likely to make a big difference: price display, the number of checkout steps, offer structure. If needed, we also move the primary metric to a higher step in the funnel, for example from purchase to checkout start.

How long should an A/B test run?

We calculate the duration from the required sample and always round it up to full weeks, because weekday and weekend visitors behave differently. We do not check results every day and stop the test when significance appears; that approach markedly increases the chance of declaring a false winner. We also account for periods specific to the shopping calendar in Türkiye:

  • We do not launch new tests during the November sales period, Black Friday or the weeks before religious holidays
  • In categories affected by payday and the start of the month, we plan tests to cover at least one monthly cycle
  • We do not change the ad setup sending traffic to the same page while the test is running
  • If the share of visitors assigned to variants deviates from plan (SRM), we discard the result

Google Optimize alternatives: choosing an A/B testing tool

Google Optimize shut down on 30 September 2023, and GA4 has no built-in A/B testing tool. Today we work with standalone tools such as VWO, AB Tasty, Optimizely or Convert and send variant information to GA4 as an event parameter. That way we can read results in GA4 and order data as well as in the tool's own report.

Because client-side tests change content while the page loads, they can cause a brief flicker, and the anti-flicker snippet can hurt page speed, especially LCP. For critical tests such as checkout flow and pricing, we recommend a server-side setup where the variant is chosen while the page is rendered on the server. We decide which setup to use based on the share of revenue that passes through the page under test.

How we prioritize A/B test ideas

Test ideas come from funnel analysis and session recordings. We rank them using the ICE score described in growth experiments; for site tests, we score Impact based on the volume of drop-off at the step the change affects. We run site tests separately from ad tests: for creative and messaging tests, we use the platforms' own experiment tools and ad creative analysis. A winning ad message, in turn, often provides the next hypothesis to test on the landing page.

A/B Testing: when do we recommend it?

We recommend it

  • E-commerce and subscription sites that can collect enough visitors per variant
  • Teams looking for evidence before making major decisions on price display, checkout flow or offer structure
  • Companies that want to settle design debates with data rather than opinions

We don't recommend it

  • Small interface tests on sites with only a few thousand monthly visitors; results would take months
  • Fixing an obvious bug; bugs are not tested, they are fixed
  • Setups where it has not been verified that the variant event reaches GA4; results cannot be read against order data

How we manage A/B Testing

  1. HypothesisWe write the change, the reason for it and the expected impact in a single sentence.
  2. Sample calculationWe calculate the required number of visitors and duration based on the current rate and target difference.
  3. Setup and quality controlWe check variants on every device and browser and verify the variant event sent to GA4.
  4. Launch and waitWe do not act on results before the planned duration ends; we only run SRM and error checks.
  5. Readout and documentationWe read results by device and record every test, winner or loser, in the learning archive.

A/B Testing: metrics and requirements

Metrics we track

  • Primary conversion rate
  • Revenue per visitor
  • Confidence interval
  • Sample ratio mismatch (SRM)
  • Average order value

What you need to get started

  • An A/B testing tool license, or developer support for server-side testing
  • A variant parameter defined as a custom dimension in GA4
  • Sufficient, steady traffic to the page being tested
  • A decision to freeze page and campaign changes during the test

Official sources

The Brand Move CRO Team

To learn more about A/B Testing or other conversion optimization services, contact The Brand Move Conversion Optimization team today.

Contact us now
Explore further

Other Conversion Rate Optimization services

Conversion Rate Optimization overview

Funnel Analysis

Funnel drop-off analysis in GA4: we find the step, device and channel where visitors leave and set priorities with data.

Explore

Related pages

Frequently asked questions

A/B Testing: frequently asked questions

The test looks like a winner at 90 percent confidence. Can we ship it?

Not if the planned sample and duration have not been reached. Interim results fluctuate, and the difference seen in tests stopped early usually shrinks later. Once the planned duration is complete, we decide based on the threshold and primary metric set in advance.

What if our site does not have enough traffic for A/B testing?

We test bigger changes that are likely to move the needle, such as price display, the number of checkout steps or offer structure, instead of small tweaks like button color. If needed, we move the primary metric to a higher step in the funnel, for example from purchase to checkout start.

Can we run more than one test at the same time?

Yes, on different pages and in areas that do not affect each other. If we test within the same user flow, for example on both the cart and checkout pages, we either separate users or run the tests in sequence to control for interaction.

What happens if the result is not significant when the test ends?

We record the test as inconclusive; we do not extend it in search of significance. An inconclusive test tells us the two versions cannot be told apart at this traffic level. We then write a new hypothesis with a bigger change or move the primary metric higher in the funnel.

Will an A/B testing tool slow down our site?

Client-side tools change content while the page loads, which can cause flicker, and the anti-flicker snippet can hurt page speed, especially LCP. For critical tests such as checkout and pricing, we recommend server-side testing, where the variant is chosen before the page reaches the browser.

Let's run a first A/B Testing review together.

In a free call we review the drop-off points in your funnel and your landing pages, then send you the first three ideas to test in writing.

Book a growth call No sales pressure. Just findings.