In this post, we’ll look at an example case of how a/b testing can benefit you during the game development phase.
You have been assigned to design and develop a new game. You have two great options on the table. You have to make a choice between a runner or platformer concept. However, the choice isn’t easy since they both seems to have similar scalability in terms of audience and market size. Luckily, we have a/b testing to help us out. Let’s look at how we can overcome the first problem of choosing the right concept to further develop.
1. Marketability
Because we want to base our decision on actual data from a live crowd, we may start by creating look-alike store pages that illustrate the concepts. Let’s make four screenshots of each concept, as well as other metadata such as title, icon, and so on. We can also include an additional variation that’s an existing game of ours. This way we can benchmark the results of the concpets to see if any of them are worth to further develop.
Because the two concept variations are not real games, we may send people to a survey when they click the ‘install’ button. This allows us to learn even more about the audience and their preferences.
Because we’re interested in marketability in this case, we’ll use the same budget for both concepts and see which one gets more installs.
Now that the a/b test is complete, we can examine the results. We can see that the platformer concept received far more installs than the runner concept. To support our decision, the platformer concept look-alike page outperformed the benchmark game that we included in the test as well.
Testing different concepts with look-alike pages
Awesome, we were able to base our decision on actual user behaviour. What’s next?
Now that we’ve decided on a concept, we can try out different themes to see which one works best for our game. Our plan was to make a platformer game with a cowboy theme, however we’re not sure if that’s the best option. So now we’re up to our second a/b test.
2. Theme test
We planned to develop cowboy themed platformer game, but since it’s not used by other platformer games we want to be sure there’s an actual market opportunity. Based on competitor analysis, we can see that zombies and knights are popular themes. So let’s include them as hypothesises on our theme test.
Once again, we create look-alike pages that describes the different hypothesis.
We used the same target audience and budget for all variations in this test, but different ads.
When we analyze the test findings, we discover that cowboys were an unused theme for a reason. The clear winner in this test was knights, which makes our job easier because we can confidently conclude that knights are the right theme for us.
Testing different themes with icons and screenshots.
We can now move on to the last important question in the process: what art style will we use in our game.
3. Art style
After deeply analysing the competitors we can conclude that other platformer games use mainly cartoon like or pixel art style. We are also happy with the findings and want to test both art styles. Also, after discussing internally we want to add a third variation as a wild card to see if realistic art style would work.
This time, the results were mixed. When it came to conversion rate, the realistic art style clearly won, but the cartoon-like art style had more installs. After further analysis, we can conclude that realistic art styles are more difficult to find an audience for, while for cartoon-like art style it’s relatively easy to find the audience. This is why, despite outperforming in terms of conversion rate, realistic received far fewer installs. And since we are looking for the art style with better marketability, we’ll choose the cartoon-like art style.
Different art styles and KPI’s
Awesome, now let’s develop this game!
“Huh, what’s that?”
“We’ll add interactive stories as additional feature.”
“No, we didn’t test that.”
And so, we did one more test for our game.
4. Feature test
We planned to include interactive stories in the game from the start. But, just to make sure we’re on the right track, let’s run one more a/b test. We want to compare the interactive stories to more typical platformer features, so we make screenshots that illustrate the interactive as a feature alongside more standard platformer features like enemies to kill and items to collect.
With the finding we can now be sure that the interactive stories was in fact the best option to go with.
Different additional features and KPI’s
We’re all set to test for the game development phase after the feature test. By testing we have been able to base our decisions on the most important people, the potential players who will actually play the game.
But don’t worry! There’s plenty of testing to be done. In the next post we’ll go over how to optimize store pages to match the target audience and get the most out of people who visit our page.
As always, if you have any questions, please don’t hesitate to reach out! .
