Welcome to the first instalment of ASO Giraffe's creative testing series, where we demystify the creative testing process step by step.
Part 2: Asset Creation
Part 3: Experimentation
Today we're chatting to Jorge Canga, games marketer and founder of Kaizen Internet. We asked him to take us through the thought process behind some hypotheses he made for a mobile real-time strategy (RTS) war-themed game he worked on. We recreated the original assets with a bit of Giraffe flavour to illustrate the changes, but the concepts, process and numbers are real. To set the scene - Jorge is tasked with improving the performance of a strategy game he’s responsible for. He’s identified the App Store assets - namely, the icon and the screenshots, as weak points in an otherwise strong and well-optimised funnel.
The Hypothesis is where all great experiments start. In the context of creative optimisation, a hypothesis could be an idea about what type of messaging may appeal to your target customers, what visual elements may be the most appealing, or perhaps that making text more legible will lead to more conversions (it can be as simple as that).
Starting off with the screenshots, Jorge had a feeling that the assets they were using on the stores could be improved
"The company was reluctant to make changes since CRO wasn't part of the marketing strategy. I hypothesised that through utilising warmer colors, including more action and changing the format (portrait vs landscape), we could make more visually appealing assets that would drive performance increases."
For Jorge, this screenshot was not selling the game well enough. The screenshot emphasised the RTS nature of the game, but offered very little in terms of visual stimulation or context about the game itself. Whilst the message is legible and big, which is great for App Store assets because the text becomes very small on phones, Jorge wanted a more compelling value proposition, and also to convey the intensity and action of the game to fit better with the genre and match expectations of the target audience.
Jorge wanted to add more action elements, while keeping visual clutter to a minimum. He thought that landscape screenshots could help with this, so here’s what he came up with:
Original screenshot style vs. updated screenshot style:
Jorge hypothesised that combining gameplay elements with two of the game’s characters would capture the user’s attention. Combining this with a more aggressive typeface and colourful explosions in the background, he hoped to make the game more enticing for users
With the assets created from Jorge's hypothesis, they ran a fake app store page test sending traffic to the portrait and landscape variations.
The results were conclusive - with 1077 visitors, the conversion rate for the landscape images was a 64% improvement over the baseline, Driving 104 installs compared to the 64 of the baseline variation. And these installs came from a valuable channel, paid ads.
With such a big improvement in conversion, the amount of visits necessary to get a valid experiment was greatly reduced.
With Jorge’s strong hypothesis, he improved the conversion rate of his page by +64%! A statistically significant improvement that helped him justify the importance of A/B testing assets and earned more resource for App Store experimentation, a great result! From this, Jorge went with confidence onto his second test...
Hypothesis #2
Next up, Jorge wanted to apply the same thought process to his app icon.
The original icon was flat and not dynamic, though it looks pretty cool with a giraffe inside, in reality this icon was fairly uninspiring. The mech suit communicated ‘war’ to users, the game genre, but Jorge hypothesised that with a more dynamic icon, he could continue to reinforce the genre, catch the users’ eye and communicate the nature of the game in a way that aligned better with user expectations
Original icon style vs. updated icon style:
Utilising the same thinking that drove the screenshot A/B test. Jorge hypothesised that combining gameplay elements with two of the game’s characters would capture the user’s attention.
To validate his hypothesis, Jorge again ran this as an A/B test using spoof landing pages. He sent traffic to both variations, and when the results came in, they were conclusive, with a statistically significant improvement that validated his hypothesis.
The results came out conclusive again - a 36% improvement in installs, with conversion rate improving from 8.7% to 11.8%
From these two tests, conversion rate for the page improved from 5.9% to 11.8%, a great improvement that will have strong implications all the way down the funnel.
For Jorge, the tests were a success, and these were his takeaways from the process:
“CRO is a huge part of the funnel and has a direct impact on User Acquisition. In total, across all our tests, we managed to reduce our CPI by 30% through creative iterations. This meant more installs, more spend, and faster growth”.
"Think outside of the box, don't stick to what you see on the stores. Try new stuff that nobody else is doing it. Early bird gets the worm".
"No one is right 100% of the time. Whilst we’ve highlighted some successful tests here, not all our tests were successful. Also bear in mind that even if you see good results, it doesn't mean they can't be improved further."
Many thanks to Jorge Canga for the case study!
