7 Concept validation myths busted is our version of MythBusters. While our mustaches aren’t as impressive as Jamie Hyneman’s, we still have some first-hand insights into concept validation in mobile game development. We’ve worked with many studios from the beginning of development to help make decisions along the way with the help of A/B testing. The best way to start is by running initial marketability tests with look-alike app store product pages to determine whether there is a market for the concept.
Here are some myths we’ve stumbled upon:
1. “A/B testing is only useful for established games”
Busted: Concept validation through A/B testing is beneficial for games at any stage of development. The earlier you start the better insights you get to support development. For new games, it provides critical insights into market interest and potential success before significant resources are invested. For established games, it can help refine and optimize aspects of the game and marketing strategies to maintain or boost engagement and revenue. We always say it is better to fail fast to afford to fail twice.
2. “A/B testing is too expensive for small developers”
Busted: While A/B testing can seem daunting, the cost doesn’t need to be astronomical. The return on investment can be substantial, as it helps prevent costly mistakes by identifying what resonates with the audience early on. Moreover, some tools offer scalable pricing based on the size of your user base, making it accessible for smaller studios. Here are some rough numbers for what an example test can cost:
3. “Concept validation only involves testing visuals”
Busted: During the development phase the most commonly tested aspects are game theme, art style, characters, and even gameplay mechanics. The earlier you are, the better is to test big elements. Sometimes the tests can even help determine what skills you should invest in when building a new team. We have a great example of this in our case study with Funcraft. Testing thev different elements helps create a comprehensive understanding of what attracts and retains players.
4. “A/B testing is time-consuming and slows down development”
Busted: When integrated early on into the development process, A/B testing can actually save time in the long run by preventing the need for major revisions later. Automated tools and streamlined workflows help make the process efficient, and for teams to test and iterate quickly without significant delays.
5. “Positive test results guarantee success”
Busted: While positive A/B test results are encouraging, they do not guarantee overall success. They indicate potential and guide development, but other factors such as user experience, game quality, and marketing strategies also play critical roles in a game’s success. Concept validation is a tool among many that contribute to a well-rounded development approach. It also doesn’t erase the need for human intuition and experience when building the next hit game.
6. “A/B testing should only be done close to launch”
Busted: Early and ongoing A/B testing can help development decisions and improve the game’s overall design and marketing strategy. Testing different concepts early in the development process allows for more flexibility and reduces the risk of last-minute changes or post-launch disappointments. After launch it is important for app store optimization to keep the app store product page relevant and appealing.
7. “A/B testing can be done without proper planning”
Busted: Effective A/B testing requires careful planning, including clear objectives, well-defined hypotheses, and a thorough understanding of the metrics to be measured. Skipping market research and hypothesis-building can lead to misleading results and poor decision-making. Structured and strategic testing helps reduce the risk of inconclusive tests.
