25.10.19

The effect of ITP 2.3 on split testing

Hans Hoogenboom

Senior CRO strategist

The update to Apple’s Intelligent Tracking Prevention (ITP version 2.3) protocol in September 2019 sent analysts into a tailspin over worries it could skew data. And with good reason. But the fallout reaches much further than your analytics; influencing the split-testing results so integral to data-driven business decisions. Find out how it could be affecting yours…

Why is ITP 2.3 affecting A/B testing?

In the most common browser-side method of multivariate testing (MVT), A/B testing and personalisation, the experimentation platform used relies on cookies in the user’s browser to determine whether or not they have already seen the experience and to include them in experiments.

If cookies are removed by Safari after a period of time, the experimentation platform can’t recognise the user effectively; they’re then labelled as a first-time visitor and prevented from seeing the most relevant experience.

How is ITP 2.3 affecting test data?

Simply put, if a user comes to a site and qualifies to be in an A/B test then the experimentation platform will randomly assign (bucket) to experience A or B. Any goals completed by this user will be attributed to the variation, determining its success. This decision is stored through a cookie, so that if the user comes back the platform can serve them the same experience.

If, due to ITP, the same user loses cookies and returns to the site in a different session, the experimentation platform will treat them as a new user and randomly re-bucket.

This can mean that they see a very different experience; counting in both data sets and skewing the data. And if the user doesn’t convert it could decrease the conversion rate unnecessarily.

Alternatively the user could also see the same experience; again bad news for your data as they’ll be counted twice in one experience, reducing conversion rate and other metrics.

Whether throwing out false positives or false negatives, both scenarios can skew data and impact your ability to make the right data-driven decisions.

Will my split-testing experiments be affected by ITP 2.3?

If you’re using any of the following for A/B testing, your data could be impacted by ITP:

Third-party cookies

As ITP doesn’t allow for 3rd party cookies, you’re likely to be heavily impacted and should consider moving to other solutions

First-party cookies

These cookies will be removed after one day from the user’s browser, which is likely to affect your data and user experience.

Local storage

This data will be removed after seven days in the user’s browser, so could impact your test data and user experience if the path to conversions is generally longer than a week.

Server-side testing

This is currently the safest option, but make sure your platform has good support for server-side bucketing and segmentation.

How to avoid data skewing caused by ITP 2.3

There are a few methods that could help you avoid data skewing when A/B testing:

Server-side testing solutions

Servers are not subject to browser privacy rules so can help segmentation without relying on browser cookies.

Example of server-side testing solutions: Monetate, Optimizely (server-side), Conductrics.

Hybrid testing solutions

These solutions can leverage server-side and other platforms, in combination with browser-side, to reduce the effect of ITP.

Example of hybrid testing solutions: Optimizely X, Adobe Target, VWO

Local storage for leverage

Local storage works in a similar way to cookies in that it can store data in the user’s browser. However, this does come with a few restrictions, such as not being able to transfer across sub-domains. In ITP 2.3 the local storage data could be removed after as little as seven days, so your data may still be impacted.

 

Need help with conversion rate optimisation? Contact results@icrossing.co.uk.

Continue reading

POV

/

05.03.2019

Apple’s cookie clampdown: ITP 2.1

SEE FULL POV

POV

/

10.07.2019

How to measure the impact of ITP 2.x

SEE FULL POV

POV

/

07.10.2019

Intelligent Tracking Prevention 2.3: The neverending ITP story

SEE FULL POV

ix-chevron-bg

Contact

Are you ready to make a digital step-change?

We believe that moving too slowly in digital is the biggest risk your business faces. If you are ready to move faster in digital, we are here to help.

GET IN TOUCH