The update to Apple’s Intelligent Tracking Prevention (ITP version 2.3) protocol in September 2019 sent analysts into a tailspin over worries it could skew data. And with good reason. But the fallout reaches much further than your analytics; influencing the split-testing results so integral to data-driven business decisions. Find out how it could be affecting yours…

In the most common browser-side method of multivariate testing (MVT), A/B testing and personalisation, the experimentation platform used relies on cookies in the user’s browser to determine whether or not they have already seen the experience and to include them in experiments.

If cookies are removed by Safari after a period of time, the experimentation platform can’t recognise the user effectively; they’re then labelled as a first-time visitor and prevented from seeing the most relevant experience.

Simply put, if a user comes to a site and qualifies to be in an A/B test then the experimentation platform will randomly assign (bucket) to experience A or B. Any goals completed by this user will be attributed to the variation, determining its success. This decision is stored through a cookie, so that if the user comes back the platform can serve them the same experience.

If, due to ITP, the same user loses cookies and returns to the site in a different session, the experimentation platform will treat them as a new user and randomly re-bucket.

This can mean that they see a very different experience; counting in both data sets and skewing the data. And if the user doesn’t convert it could decrease the conversion rate unnecessarily.

Alternatively the user could also see the same experience; again bad news for your data as they’ll be counted twice in one experience, reducing conversion rate and other metrics.

Whether throwing out false positives or false negatives, both scenarios can skew data and impact your ability to make the right data-driven decisions.

If you’re using any of the following for A/B testing, your data could be impacted by ITP:

Third-party cookies

As ITP doesn’t allow for 3rd party cookies, you’re likely to be heavily impacted and should consider moving to other solutions

First-party cookies

These cookies will be removed after one day from the user’s browser, which is likely to affect your data and user experience.

Local storage

This data will be removed after seven days in the user’s browser, so could impact your test data and user experience if the path to conversions is generally longer than a week.

Server-side testing

This is currently the safest option, but make sure your platform has good support for server-side bucketing and segmentation.