We’ve heard a lot over the last few weeks about how Generative AI is going to completely change the digital marketing landscape. Launching onto the scene in November 2022, ChatGPT exploded, reaching 1 million users within only 5 days. Since then, it’s been making its way into everyday use, for everything from proofreading emails to formatting code and even writing rap lyrics.
At fifty-five, where we consult with brands on the use of technology to power new and cutting-edge marketing use cases, it didn’t take us long to start testing the capabilities of the hottest new chatbot in town. In particular, having partnered with a significant number of well-known brands to migrate onto Google Analytics 4 (GA4) well before the sunset of Universal Analytics, we decided to put it to the test on something our consultants do regularly. Answering questions asked by our clients using GA4. Here were the results:
Out of the 42 questions we asked, ChatGPT only provided 12 acceptable answers we’d send to our clients, a success rate of just 29%. A further eight answers (19%) were deemed ‘semi-correct’. These either misinterpreted the question and provided a different answer to what was asked (although factually correct), or had a small amount of misinformation in an otherwise correct response. For example, ChatGPT told us that the ‘Other’ row you find in some GA4 reports is a grouping of many rows of low-volume data (correct) but that the instances when this occurs are defined by ‘Google machine learning algorithms’ (incorrect - there are standard rules in place).
The remaining 52% of answers were factually incorrect, and in some cases actively misleading. The most common reason for this is that ChatGPT does not use training data beyond 2021, so many of the recent updates are not factored into its answers. For example, Google only officially announced the deprecation of Universal Analytics in 2022, so ChatGPT couldn’t say when this would be. In this instance, the bot did at least caveat its answer with this context, leading with ‘As of my knowledge cut off in 2021…’.
However, some of the remaining answers were wrongly answered with a worrying amount of confidence, such as the bot telling us that ‘GA4 uses a machine learning-based approach to track events and can automatically identify purchase events based on the data it collects.’
While GA4 does have auto-tracked ‘Enhanced Measurement’ events, these are generally defined by listening to simple code within a web page’s meta-data, rather than through any machine learning or statistical model. Furthermore, purchase events are certainly not within the scope of Enhanced Measurement.
The explanation behind the second type of incorrect answer can be explained by considering another limitation in ChatGPT’s training data. There’s a lot of misinformation out there on the world wide web, which by and large was what was used to train the chatbot, and it’s trained using patterns. Many of Google’s support articles on GA4 are sprinkled with the term ‘machine learning’ and this has subsequently been echoed in external articles about the platform, so it’s fairly unsurprising that it was mentioned in the responses to 17 questions, including the incorrect answer above.
Finally, it’s worth mentioning that there’s an element of randomness to using ChatGPT - you can give the bot the same prompt twice and get slightly different answers.
In summary, there are plenty of reasons that ChatGPT isn’t quite at the same standard as fifty-five’s consultants in giving balanced answers to complex questions about marketing tech. However, it does provide a great starting point to discuss its responses on a given topic and follow up with more research. This is in line with where we see AI supporting the workplace today. Most jobs aren’t at risk of being taken over by AI completely, but tools like ChatGPT do a fantastic job at improving productivity, sparking creativity and kicking off a conversation. As the BCS writes, ‘AI isn’t going to replace your job - but someone using AI will’.
Discover all the latest news, articles, webinar replays and fifty-five events in our monthly newsletter, Tea O'Clock.
