Featured article
It feels like everybody is using AI these days. From mundane purposes like writing emails and getting news summaries to more serious tasks like getting legal or even medical advice, people are trusting the AI chatbots with important decisions.
Even in my highly specialized field of mobile ad monetization, it’s becoming common to get questions from clients or colleagues, referring to something that ChatGPT (or some other AI chatbot) told them.
For me, that begs the question: how good are these chatbots actually? Should I be worried about my job being replaced by AI anytime soon?
Methodology
To figure this out, I’ve made a list of 40 ad monetization questions, with the idea to use these questions to test different AI models and see how well they perform compared to my knowledge.
The questions are a mix of several things:
- Basic knowledge even junior ad monetization managers need to know
- Common questions that we get from clients in our daily work
- Advanced knowledge I would expect from seniors
- And just to spice things up a bit, a few highly difficult problems that are meant to push AI to its limits
A snapshot of some of the questions is shown below, though it's more accurate to describe them as broader topics. Some of them required multiple questions to be covered in full.
The exact way in which the questions are asked is super important when interacting with AI, and it’s called prompting.
Prompting is quite complex and would require an article all of its own (and maybe we’ll write one in the future!).
What’s important to know is that if you want to utilize AI effectively, you need to learn about prompting. It’s an important skill that allows you to squeeze the maximum out of your questions.
The Importance of Prompting in Our Experiment
I want to point out the exact way in which the questions are asked is super important when interacting with AI, and it’s called prompting.
Prompting is quite complex and would require an article all of its own (and maybe we’ll write one in the future!).
What’s important to know is that if you want to utilize AI effectively, you need to learn about prompting. It’s an important skill that allows you to squeeze the maximum out of your questions.
How Did We Test AI in Ad Monetization?
The initial idea was to go through these questions with any AI model I could get my hands on.
Very early into my research, it became clear that this is pointless. Free and older models are just not good enough for this to be interesting or useful to anyone. I strongly recommend against using them for ad monetization advice.
In the end, I decided to focus on 3 models which are currently the best on the market (or close to it) according to the majority of the benchmarks:
- ChatGPT - o3
- Gemini - 2.5 Pro
- Claude - Opus 4
I went through all 40 questions with each model, trying to keep it as consistent as possible. The opening questions were always the same, and then the follow-up depended on the responses.
Here are the results. A pretty tight race as you can see, but ChatGPT-o3 takes the crown.
Grading was as follows:
- Full Marks - Complete answer, I could not give a better one myself, or if I could, the improvement would be marginal
- Partial - Either the answer wasn’t complete, or there was a mistake of some kind, but nothing too crucial
- Bad - Flat out wrong or problematic in some other major way
My first impression was that these results were quite good. More than half of the answers got top grades! However, after the initial awe wears off, it becomes clear that not even the best models are good enough yet.
Consider it this way: who wants to collaborate with someone who is wrong almost half of the time? In addition to that, I need to highlight something important - the bad answers were often catastrophically wrong.
To give an example, one recommendation was that the optimal banner refresh rate was 30-120 sec. When the industry standard is in the 5-15 sec range, implementing even the 30 sec suggestion would drop your ad revenue by 30%-60%!
We’ll go over the bad parts in detail later. Let’s begin by looking at what worked well.
The Good
AI is great at giving detailed overviews on various topics. It can serve as a refresher or a starting point when tackling a new challenge. It also helps a lot with bringing less technically savvy colleagues up to speed.
Massive improvement to data analytics. AI is able to manipulate data and organize it in a way that makes it easier to use. It can also dig for insights, and while it’s not perfect at those tasks, it’s a great starting point. The level of analysis you can do independently is now much higher. Just beware of hallucinations, and make sure to double-check everything.
Amazing for ideation and prototyping. I had great results with questions that were focused on brainstorming about potential implementations, from in-game design to finding solutions to technical issues, AI was exceptional at these tasks.
Here you can see one of the answers that was very impressive to me. This is Gemini on Ad Quests, which after just one question, gave 3 pages of detailed analysis, covering:
- In-game implementation details - game design, UX/UI look and feel
- Rewards suggestions and how they fit into the overall economy
- Analysis of benefits and potential challenges
In addition to all this, the suggested implementation is thematically fitting, which is something a lot of developers that we work with often miss. Often ads are seen as an afterthought or even worse, something disconnected from the game.
Also, just look at those puns (italicized). I don’t even like puns, but these are great. Implemented as is, they would beat most implementations on the market in terms of flavor.
The Bad
A significant number of mistakes come from AI using bad sources. It often seems that everything with relevant keywords is scraped from the internet, giving priority to websites that are high on Google's index. A lot of these sources are misleading or straight up incorrect. In addition to this, in some cases, the data is just old.
I’ve had multiple instances where a source from 2021 was quoted as relevant. Our industry moves at an incredible pace. The sheer volume of changes can make a single year feel like a decade, which makes older sources completely obsolete.
All models that I've interacted with sometimes exhibit a strange hyperfixation on pattern matching. There was a scenario where I mentioned Brazil as one of the relevant GEOs in my app. Not the most important GEO, not even top 3. Despite that, a disproportionately large number of suggestions for that scenario were in some way focused on Brazil.
This sort of thing happens often enough that it’s worth avoiding all details that are not highly relevant to the question at hand. Which brings me to the next point.
Avoid complex context. Giving too much information (even if it’s relevant) can make the AI confused. Especially if multiple different priorities need to be juggled, things will get messy fast.
Another AI peculiarity is making unsubstantiated claims. There were several cases where the model gives a recommendation that, while theoretically possible, isn’t supported by any kind of data or an existing solution on the market. And I know what you’re thinking, maybe the AI is making connections that we didn’t see!
Nope, all suggestions were quite bad.
The Hallucinated
Hallucinations, the bane of AI. I think everyone who uses AI has experienced these in some shape or form. No matter what you try, they tend to sneak in somehow.
My experience with this research was that the number of answers that included hallucinations of some sort was quite high, between 10% to 20%, depending on the model. However, some of the hallucinations were relatively minor and didn’t significantly downgrade the quality of the answers.
All models struggled significantly with questions in the format “Recommend me top X in a certain category.”
Worst examples were:
- Recommending a revive mechanic for a genre of games where there is no dying/fail state
- Claimed that Liftoff is a great network because they bought Chartboost. When asked for a source, fabricated a whole news link that doesn’t exist.
Lesser hallucinations often happen when the scope of the question becomes too large. If possible, try to break things down into smaller parts to avoid this.
Mixing up information and terminology from disciplines that are close to mobile ad monetization (user acquisition, search engine optimization, web-based ad monetization, etc.), and then using information from these disciplines to give suggestions regarding ad monetization.
Just making stuff up, with no obvious rhyme or reason.
Am I Going To Be Replaced By A Chatbot?
The outcome of this research was surprising, with the AI proving to be both better and worse than I had imagined.
Some answers were quite impressive, even better than what I could come up with myself. However, the good was spoiled by the bad. There were plenty of mistakes, even on basic questions.
Other issues also became visible during this research:
• For many topics, it’s challenging to properly utilize the AI if you, the user, lack knowledge on the subject. The quality of the answers would be much worse if I weren’t able to utilize my experience to ask better questions and, in some cases, guide the AI in the right direction.
• Beyond the knowledge of the subject, there is also a lot of skill in the prompting itself. Deeper knowledge on how AI operates and lots of trial and error will give much better results than talking to AI “naturally”.
• The hallucinations pose a significant problem. It’s hard to get proper data on this, but what I managed to find shows that even with the tightest possible setup, the limit on the hallucination-free responses seems to be around 90%. Most are below this number, which means that creating a consistent workflow involving AI is a challenge.
• The old adage, garbage in, garbage out is still as relevant as ever. If the source of AI’s knowledge is bad, you are going to get problematic answers, and there is no way to control that at the moment.
• The default settings of AI make it a dangerous yes-man. They are made to be pleasing and to go along with your ideas. Thread carefully, because current models are way too happy to lead you into an abyss.
So, what is the final verdict? Even with AI’s impressive capabilities, it seems that ad monetization is safe for now. My illustrious beekeeping career will have to wait.
AI is still a tool to augment us, instead of something that is able to replace us. However, when we look at how much AI has improved in just one year, they are definitely getting closer.
And that is quite scary.
