A curated roundup of marketing analytics news, tools and ideas — with a focus on the Google analytics stack and the intersection of AI and analytics.
As we’ve moved beyond the gee-whiz phase of pasting data into an AI chatbot, the focus of the analytics community has shifted to developing trustworthy architectures for generating insights with AI. I’ve shared a number of articles on this topic in recent months, with a few more below. Some consistent themes are emerging. To do it right, you need:
- Deterministic data modeling – everyone worth listening to seems to have arrived at the same conclusion: an LLM + raw data = pandemonium.
- A robust context layer – six months ago it was all “semantic layer this” and “semantic layer that”. We have since realized that providing context regarding customers, business goals, domain best practices, etc. is just as important. Some say this is the most impactful piece of the puzzle.
- Governed calls to an LLM – telling it what not to do may be more important than telling it what to do.
- A verification loop – human, of course, but also automatic. This month, I share a really interesting article about an organization that developed an ‘evaluation harness’ to score accuracy.
For more on my perspective, check out this video:
All that is to say that LLMs need a lot of scaffolding to be useful and accurate when it comes to analytics. In a sense, I think we are battling the fact that an LLM is designed to return the most probable answer, based on text it has consumed. That is somewhat opposed to what we do as analysts – we earn our paychecks by discovering opportunities and insights that are surprising and far from obvious.
One article I share below discusses the limitation of basing AI on language. In it, the author describes how Google is putting a lot of resources towards building ‘world models’ at the cost of falling further behind in the LLM race. The article provides A LOT more detail, but think of it like this: what if you grounded a machine in fundamental principles of biochemistry instead of just showing it a lot of pictures of plants?
As marketing analysts, we are currently in the business of prediction. I wonder if we are headed towards being in the business of simulation.
Product Updates
- Google has added an AI-driven dashboarding feature to Google Ads and Analytics. I’m seeing it in some, but not all of my GA accounts. So far, it’s pretty underwhelming. You can drag-and-drop a rudimentary set of visualizations, with no styling/customization options. I’m also confused by the product strategy behind it, given that we already have customizable reports in GA, plus explorations, not to mention Data Studio.
- In June, Google Analytics quietly announced that the Measurement Protocol is being replaced by Data Manager, their “central infrastructure for future data ingestion innovations”. This video from Julius Fedorovicius provides a nice introduction: The new way Google wants your first-party data (Google Data Manager).
- Microsoft Clarity has added several new features to their AI Visibility reporting:
- Branded vs Non-Branded Grounding Queries
- Scrape-to-referral ratios (via Barry Schwartz) – this metric tells you how many referral visits you get from LLMs in relation to how often they visit pages on your site. If you monitor AI bot activity, this can help you quantify the business value, though I have observed that it is a fairly chaotic measure.
It’s cool to see how quickly Microsoft is adding features to this product – I continue to hope that it puts pressure on Google to improve gen AI reporting in Search Console.
- Power BI Reports Are Turning Into Code. I Don’t Think Your Team Is Ready for the One-Way Door, Gulab Chand Tejwani
This one caught me by surprise. I’ve been working in Power BI a lot recently, but kind of on the down low, because most of my friends favor Data Studio. The TL;DR is that in the near future, all Power BI reports will be stored as folders of JSON files. Last month I shared my thoughts about the rash of ‘build a dashboard in minutes with AI’ posts and articles. The JSON approach addresses one of my main concerns: governance. You can achieve similar build-efficiencies by creating and editing Power BI JSON files with a coding agent, but the data sources are still deterministic and the code can live in source control. That feels like the right middle ground.
AI-assisted analysis
- How we built an agentic loop to improve every skill in our company, instantly, Will Reynolds & Jordan Strauss, Seer Interactive
A loose discussion of how Seer uses a shared LLM skill repository to quickly iterate on and share innovations. Two of the key ingredients that make it work well:- The repository is managed in Github, and a human gatekeeper reviews changes before they are merged.
- Updates to the repository are automatically propagated to everyone in the org.
Interestingly, Reynolds describes how this approach relies on people recognizing when LLMs make mistakes with real client data and flagging them for an agent to fix. That seems like it would be ok for certain kinds of decisions, but risky for others.
- How do you know if the analytics agent is right?, Robert Sahlin
The author describes how his organization built an ‘evaluation harness’ to test the accuracy of their analytics agent. I really like the idea (and appreciate the detail he provides), and plan on incorporating something similar into my own agentic analytics workflow. It would also dovetail nicely with the loop described in the previous article. The thing I’m puzzling over is how not to run afoul of Goodhart’s Law. - Answers you can question: how we built an analytics agent you can verify, Alex Litvinov
If you are ready to start building your own analytics agent, this is a pretty good blueprint. I especially like the guidance on the trust ladder and the router that prevents unnecessary context from being sent to the LLM.
Attribution & measurement
- Why Attribution is More Art Than Science, Connor Phillips
If you need to convince someone that attribution ain’t all that, this is a pretty concise and complete list. The only thing I’d add is that the major ad platforms and GA use machine learning models to varying extents to infer attribution, so even if you think you are doing deterministic attribution, you probably aren’t. - Google silently routes Shopify purchases into GA4 – attribution gaps remain, Luis Rijo, PPC Land
This seems like kind of a big deal if you use Shopify and GA, but Shopify doesn’t seem to be talking about it much. It is notoriously difficult to reconcile GA and Shopify transaction and revenue numbers—hopefully this helps. I also noticed from Julius’ video that one of the Google Data Manager connectors is Shopify. Maybe that ends up being redundant with this, but I like that it gives a tiny bit more visibility and control.
Ideas
- Getting empathy right in the age of AI #2 Thick Customer Profiles (TCPs), Helge Tennø
A methodology for creating customer profiles that go deep into the ‘why’ factors that drive decision making. I can see how this would provide an extremely helpful context layer for doing LLM-assisted analysis, but it’s a very concept-dense article—I’m going to give it a few more read-throughs before testing it out. - The end of programming, Paul Dix
Devs with unlimited token budgets are demonstrating that frontier models can churn out production-ready code faster than it is possible to review. The author posits that we are a year or so out from this being affordable for everyone. He doesn’t say this exactly, but an implication is that organizations that do continue to do human code review will get left behind. It also gets me thinking about what happens when you can have working software to meet any need in barely more than the time it takes to describe it. - The Night the Earth Shook, Strangers Started to Draw, the data drop
A beautiful story about how volunteers of an organization called the ‘Humanitarian OpenStreetMap Team’ provide critical mapping data to relief and rescue workers when natural disasters happen. - The Actual Reason Why Google “Fell Out” of the AI Race Changes Everything, Alberto Romero
A look at how Google is pursuing ‘world model’ building instead of trying to catch up to OpenAI and Anthropic in the recursive-self-improvement (RSI) race. Underlying the business implications is a question about the nature of intelligence: does language provide the fundamental building blocks of knowledge, or is it instead a limiter of understanding? The article is really long, and requires a subscription, but it is fascinating. If you have a Medium subscription, you can read it there too.
Miscellaneous
- How colour drained from our furniture, Toon Vos
I really love clever and informative data visualization. This graphic and the accompanying narrative triggered a flashback to the seminal issue 145 of Design Quarterly, which I hadn’t thought about in years. They share a similar ethos, as well as pure brilliance. - How to Build an AI-Powered Data Pipeline, Madison Mae
This is a five-part series aimed at people who work in analytics, but are new to analytics engineering. The author writes in a clear, no-nonsense style, and provides guidance on when to consider different products and methodologies at each stage of the process. An $11/month subscription is required to access the series. If you are an aspiring analytics engineer, she is worth following, IMO. - The Quiet Politics of Dashboards: Visualizing Power, Governance, and Blind Spots in African Organizations, Yasmin Abdul Hamid
The act of creating a dashboard usually entails (over)simplifying complex truths and obfuscating flawed data. This article describes how that can have a real, human cost. The principles she lays out are relevant to any kind of dashboard or report building.
