Creative Testing on Meta in a Privacy 1st World (Part 1)

iOS 14.5 was arguably the largest data privacy shift the digital ads space has ever experienced. While the pain was most immediately felt by app advertisers, the impact on measurement permeated to web advertisers as well. Supporting stat: Across our client portfolio about 70% of web impressions are allocated to iOS devices in social channels, and are therefore subject to the measurement + targeting challenges that are an extension of losing device ID access). Now, nearly 18 months after the original release of iOS 14.5 and just over a year since the system update push that occurred in June 2021, the platforms (specifically FB + Google) have largely re-calibrated themselves to the diminished signal environment. What that means for the advertiser is an entirely new playbook for how to approach social channels across campaign architecture, audience management, optimization, event configuration, ad operations, and importantly: creative testing & deployment. 

This is a multiple installment series on how the game has changed for creative strategy across social channels, including 1) best practice testing process, architecture & velocity 2) production cadence & breadth and 3) production strategy & team structure.

Platform	Facebook	Google	TikTok
App	Article 1	Article 3	Article 5
Web	Article 2	Article 4

To understand why creative strategies had to change across the core social platforms, we need to first comprehend the way that signal is being relayed back to channels in a privacy first world. You as advertisers are probably familiar with the way that diminished event volume has impacted both measurement & targeting (as well as general direct response performance), but creative optimization and testing leverages these same signals for optimization & measurement. Strategic calibration for creative has generally lagged behind the more immediate challenges introduced by iOS 14 (namely, measurement first & efficiency maximizing campaign strategy second). The signal chain is, of course, different for both app + web. I’ll take these one by one:

(App), aka SKAdNetwork aka  ‘SKAD’

Sometime in summer 2020 Facebook explained that they would limit iOS 14 campaigns to 9 per account – where does this number come from? To make sense of this you need to first understand what a SKAD postback returns to the ad network upon an app-side conversion:

• Ad Network
• 6-bit Conversion Value
• Install/Re-install
• Claim type (Click vs. View)
• Campaign ID

Thats it. No geo information, no device information, no ad set / ad group or ad information, no other exogenous metadata. 

Each Ad Network (Facebook included) is relegated to 100 campaign IDs. Recall their 9 campaign limit: the suspicion has been that Facebook allocates the 100 campaign IDs evenly across each iOS 14 UI campaign, meaning that each campaign is allocated roughly 11 campaign IDs. So that’s just 11 degrees of freedom to cover the conceivable dimensions of optimization. Including, but not limited to:

• Audience
• Geo
• Device
• User Age
• User Gender
• Audiences
• Ad Set
• Creative (including size, length, type etc.)

A quick illustration on how FB might leverage multiple campaign IDs to goalseek through just the age dimension:

SKAD Campaign	SKAD Campaign ID 1	SKAD Campaign ID 2	SKAD Campaign ID 3
FB Campaign	FB Campaign 1	FB Campaign 1	FB Campaign 1
Age Bucket	18-25	26-40	41-55
Spend	$100	$100	$100
Conversions	25	15	7
CPA	$4	$6.67	$14.29

So, while the advertiser does not explicitly see the mechanics behind this, the end result is that their campaign “FB Campaign 1” is optimizing to younger traffic due to efficiency differences reported by SKAD campaign IDs. 

Now, lets look at how you might structure a creative testing campaign if you were to take a page from the 2019 Facebook ad testing playbook:

Campaign	Concept 1
Ad Set	Variant 1		Variant 2		Variant 3		Variant 4		Variant 5
Ad	1	2	1	2	1	2	1	2	1	2
	3	4	3	4	3	4	3	4	3	4

With 2019’s signal fidelity, both Facebook & the advertiser could observe the deterministic relationship between a large count of creative variants and sub variants with high volume actions like installs; all at relatively low cost. You might recall common media buying naming conventions of the day that included metadata like background color, size, length, copy subject, copy variant etc. 

For all intents and purposes this structure is dead in 2022. Remember you have 11 campaign IDs to cover all potential dimensions of optimization including and outside of creative. If you were to roll this structure out in a SKAD framework (1x Campaign, 5x Ad Sets, 4x Ads = 20 Distinct Assets), its almost certainly the case that Facebook itself does not know which Ad was responsible for driving an install or conversion. 

To overcome this, the platforms have been estimating conversions at the ad level based on a black box of app side data that advertisers do not have direct insight into (of course basic things like click probability, but also metrics like “screen time” or how long the ad remained on the user’s screen, how far into and when the user clicked on a video, “thumbstopping”, etc.). In practice this methodology works to some degree, depending on how complex the setup and how detailed of a creative insight you are targeting. 

But ultimately, conversions are identified by a sole campaign ID that FB maps to a campaign in the UI. Hypothetically, if the advertiser were to throw in an audience dimension to this test (Lookalike vs Broad lets say), the total unique data points increase to 40. This is why mass variant testing (spray & pray), micro variant testing (e.g. background colors, typefaces etc.), and “personalized creative” for certain customer archetypes are each less tenable as a practice. Both you as the advertiser and Facebook as the platform cannot overcome the fundamental creative testing limitations introduced by the SKAD framework.

But, you can solve for it with a proper testing structure and strategic approach to production.