Are you looking to boost your conversion rates and drive more installs? If so, you need to master the art of hypothesis formulation and validation through User Acquisition (UA) activities. With well-crafted hypotheses, you can create highly effective creatives that grab the attention of your target audience and drive engagement. In this article, we'll explore the power of hypotheses in the mobile gaming industry and show you how to conduct UA activities to validate your ideas.
Table of content
What is a Creative Hypothesis?
Creative hypotheses are assumptions, usually made by the Creative Development team (responsible for images and videos for game promotion and advertising, but not including in-game UI) which are validated through advertising campaigns on Ad Channels or Store Optimization.
Let's take an example:
Objective: Increase the conversion rate on the store by changing the game's icon
Hypothesis: The new icon (icon test) would result in a better conversion rate than the old icon
Testing method: A/B test
Testing period: 9 days
Results: After 9 days, the new icon on the Play Store had a significantly better conversion rate (up to 21.3%) compared to the current icon in the game. Based on these results, the Product team decided to replace the current icon with the tested icon.
The chart shows the conversion rate of the current and new icons.
Blue line: old icon; Orange line: new icon
(Source: Amanotes)
After the testing process, the new icon has a high conversion rate
(Source: Amanotes)
In this case, the new icon was significantly different from the current one in terms of color, expression, and posture of the character. For new games in the initial stage (Soft Launch), it is common to test multiple options with significant differences like the example mentioned above.
After a while, testing with significant differences did not yield good results. This will drive to testing of another option with smaller changes, such as changing only one element (positioning, angle, color, or size of the character, etc.).
Let's take another example in Magic Tiles 3. The initial hypothesis was that “changing the screenshot to a Halloween theme would yield better results”.
Purpose: Increase conversion rate on the store by changing screenshots
Hypothesis: New screenshots (Halloween theme) generate a better conversion rate compared to the old screenshots
Testing method: A/B test
Testing period: 5 days
Result: The new screenshots generated a better conversion rate, increasing by +5.7%
The chart shows the conversion rates of the old and new screenshot sets.
Blue line: the current set, Orange line: the new set
(Source: Amanotes)
Upon observation, we noted that both screenshot sets share the same layout. However, in the latest test, modifications were made to the elements of the new Halloween-themed screenshot set, including changes to the font and background color, to accentuate the seasonal effect. As a result of this change, the conversion rate for the new set has increased by +5.7% when compared to the current set. This outcome confirms our hypothesis, and thus we have decided to replace the existing screenshot set with the new one.
The formulation of hypotheses plays a vital role in increasing conversion rates, and download numbers, and optimizing the user experience on both Products and Ads. By checking hypotheses through the UA process, we can explore trends and preferences of users and apply them to the development of products and creatives to better match the customers.
Hypothesis development process
The process of developing and formulating hypotheses is an important part of the process of testing a creative, also known as the Ad creative life cycle:
Step 1: Market Research
Before conducting the hypothesis process, it's important to "do your homework" and research to collect as much data related to the target field. There are 2 approaches for this step:
Approach 1: Learning the case studies of other successful games and applying their common mechanics and genres to develop more reliable hypotheses.
Approach 2: Staying up-to-date with current trends, which often result in sudden high conversion rates, and analyzing various elements such as concepts, ideas, colors, and sounds for the test variables.
As there is no universal success formula, keep collecting data, experiment with different variables, and continue to analyze customer preferences and market trends to refine and improve the hypotheses
Step 2: Hypotheses Development
After obtaining insights from market research, we create hypotheses and methods to test them. Determining the testing methods is the specific task of the User Acquisition (UA) team. The creatives team should also be aware of this to ensure objective results or high reliability. Some factors that affect test results include test duration, sample size, test subjects, campaign creation, platform, etc. The test subjects' creative typically includes:
Layout
Art Style: 2D, 3D, Pixel, realistic
Perspective: Top-down, bottom-up, head-on, from behind, etc.
Color: Bright, dark, neutral
CTA: Play now, install now, Try it now, etc.
Step 3: Creative Production
At this point in the process, our focus is on generating designs or modifying the creative's format, such as graphics, videos, and playable content, in alignment with the hypothesis stated earlier.
For instance, our hypothesis proposes that incorporating a CTA (call to action) in the creatives will lead to a positive outcome, such as an increase in downloads or conversion rates (CR). Therefore, all creatives produced adhere to this hypothesis by featuring the same CTA phrase, such as "play now" or "install now." In case the chosen CTA phrase fails to yield satisfactory results, we will not draw hasty conclusions. Instead, we will test a few alternative CTA phrases for a more accurate and dependable conclusion.
Step 4: Testing
After producing the creatives, the UA team, consisting of the PO, Creative Analyst, and other relevant stakeholders, will utilize them by uploading them to the store or Ad channel for testing. The prevalent method used for testing is A/B testing, which involves comparing two versions of the same creative to identify the better-performing one. Typically, the ideal testing period is approximately two weeks (14 days). However, in certain circumstances where the results are too negative or the budget constraints are limiting, the testing period may be shortened to 3-6 days.
Step 5: Result analysis
Once the testing period concludes, we receive relevant information and data that we use to create new hypotheses or perform retests in other stages, countries, and so on. There are several approaches to address the situation, and we make a decision based on the specific case whether to conduct further testing or begin with a new hypothesis. We repeat this process until we receive a positive outcome that aligns with the approved hypothesis.
We believe that creative hypotheses are the foundation of successful games. By testing our ideas and constantly iterating, we're able to stay ahead of the curve and deliver hit games that keep players coming back for more
Phương Nguyễn - Liveops Creative Project Coordinator at Amanotes
Key takeaways
Hypotheses should be accompanied by a means of proof or measurement through numerical results
Sufficient testing time and sample size are necessary to provide accurate conclusions, typically with a minimum testing period of 5-7 days and an ideal duration of 14 days
While there may be similarities between games in terms of factors, a hypothesis that holds true for one game may not necessarily be applicable to another. Games in the mature stage often yield less favorable testing results than new games. For example, changing the game icon may lead users to mistake it for a new game rather than the old one, making hypotheses optimizing icon factors less effective
To systematically verify hypotheses, we begin by researching and referencing ideas from games with similar mechanics. By identifying commonalities and synthesizing a hypothesis, we can determine whether it is applicable to our game
Be mindful of factors affecting hypothesis verification, including the testing method, target audience, timing, platform, and seasonal events (festivals, holidays, etc.)
Author: Phương Nguyễn - Liveops Creative Project Coordinator at Amanotes