Senior CRO strategist
The update to Apple’s Intelligent Tracking Prevention (ITP version 2.3) protocol in September 2019 sent analysts into a tailspin over worries it could skew data. And with good reason. But the fallout reaches much further than your analytics; influencing the split-testing results so integral to data-driven business decisions. Find out how it could be affecting yours…
In the most common browser-side method of multivariate testing (MVT), A/B testing and personalisation, the experimentation platform used relies on cookies in the user’s browser to determine whether or not they have already seen the experience and to include them in experiments.
If cookies are removed by Safari after a period of time, the experimentation platform can’t recognise the user effectively; they’re then labelled as a first-time visitor and prevented from seeing the most relevant experience.
Simply put, if a user comes to a site and qualifies to be in an A/B test then the experimentation platform will randomly assign (bucket) to experience A or B. Any goals completed by this user will be attributed to the variation, determining its success. This decision is stored through a cookie, so that if the user comes back the platform can serve them the same experience.
If, due to ITP, the same user loses cookies and returns to the site in a different session, the experimentation platform will treat them as a new user and randomly re-bucket.
This can mean that they see a very different experience; counting in both data sets and skewing the data. And if the user doesn’t convert it could decrease the conversion rate unnecessarily.
Alternatively the user could also see the same experience; again bad news for your data as they’ll be counted twice in one experience, reducing conversion rate and other metrics.
Whether throwing out false positives or false negatives, both scenarios can skew data and impact your ability to make the right data-driven decisions.
If you’re using any of the following for A/B testing, your data could be impacted by ITP:
As ITP doesn’t allow for 3rd party cookies, you’re likely to be heavily impacted and should consider moving to other solutions
These cookies will be removed after one day from the user’s browser, which is likely to affect your data and user experience.
This data will be removed after seven days in the user’s browser, so could impact your test data and user experience if the path to conversions is generally longer than a week.
This is currently the safest option, but make sure your platform has good support for server-side bucketing and segmentation.
There are a few methods that could help you avoid data skewing when A/B testing:
Servers are not subject to browser privacy rules so can help segmentation without relying on browser cookies.
Example of server-side testing solutions: Monetate, Optimizely (server-side), Conductrics.
These solutions can leverage server-side and other platforms, in combination with browser-side, to reduce the effect of ITP.
Example of hybrid testing solutions: Optimizely X, Adobe Target, VWO
Local storage works in a similar way to cookies in that it can store data in the user’s browser. However, this does come with a few restrictions, such as not being able to transfer across sub-domains. In ITP 2.3 the local storage data could be removed after as little as seven days, so your data may still be impacted.
We believe that moving too slowly in digital is the biggest risk your business faces. If you are ready to move faster in digital, we are here to help.