AI metrics + analytics = value? (1/2) Domo's Chief AI Officer on what organizations really need to measure - diginomica

Companies may be deploying many AI tools, but measuring whether those tools are improving business outcomes is much harder to do as we’ve seen this year from the number of pauses being invoked in rollouts as value-derivation has become a major topic du jour rather than a simple pursuit of the hype cycle,

Put simply, enterprises, typically led the CFO, need to see demonstrable metrics that their AI spend is delivering tangible and quantifiable business benefits - and all-too-often that’s an ask that’s proving beyond many organizations.

But what’s clearly a challenge for so many might be an opportunity for a few, such as analytics maven Domo, fallen from grace after a long decline, but now  potentially looking at a new lease of life in the care of Progress Software. 

According to Domo’s Chief AI and Analytics Officer Ben Schein, the metrics challenge facing organizations is not simply being able to track productivity or model costs - although the tokenomics panic means that certainly does still matter - but rather lies in understanding the hidden costs of verifying AI output and providing the right business context for the tech to work most optimally.

Internal analytics

Schein started working at Domo over eight years ago, first as tech evangelist and then, a few years ago, taking over all the non-engineering parts of Product Management, tasked with helping build a platform that allows people to take data to the next level, Over the past year, his focus has turned to internal analytics and the impact of AI on them, specifically how to achieve better outcomes.

Domo staffers across the company use a plethora of AI tools, including Quad, Domo, Copilot, and Claude Code via Cursor, and as part of the day job, he has to review usage across this landscape to determine whether the outputs being produced are worth the cost of the tools.

He also wants to ensure consistency and continual learning. For example, one team might innovate and learn some things, and Schein needs to make sure those learnings are shared with other teams, even if the tools used aren’t the same:

I think a lot of times AI makes those kinds of silos, which we've always had with shadow teams or people going on their own. It supercharges them in a way that's crazy. And so, how do you honor that intent but keep it focused or make sure we're all learning together?

In fact, one the biggest challenges with implementing AI is its deployment in pockets without an overarching strategy. While Schein contends there’s no issue with using different tools for different use cases or personas, it is important to understand the costs of each tool. It’s equally important to understand who’s using what and how, and connect that usage to specific business outcomes because the cost may be worth it. 

That poses two questions that need to be addressed:

Number one is: how do I get visibility into all that is going on, and then number two is: how do I rationalize or at least create consistency across those, so that we're all getting the advantage of the best information.

The real cost of AI is more than inference

Most recent research around the benefits of AI focuses on productivity, with diappointing results all round in the main. For example, a Standard Social Media Lab and BetterUp Labs study found that 40% of what comes out of an AI is garbage, either work slop or AI slop. It also estimates that employees spend an average of one hour and 51 minutes dealing with just one instance of work slop, which ends up costing an organization around $186 per month per employee.

That rather begs the question of whether what’s been built with AI has become a big burden without a corresponding level of benefits. Schein suggests that vompanies need to think about three components of cost:

- Inference cost - this is the cost of the actual model used.

- Verification cost - testing whether the AI process works as expected.

- Context cost - does the AI have enough business context to work as expected.

He explains:

You don't see that if you're only measuring how many tokens you use or produce. I used a lot of tokens to produce something that not only didn't add value, it not only didn't save time; it actually negatively saved time because it helped make someone else spend time that they didn't have trying to weed through it and do that. And so I think that those are some of those hidden costs there that people are not acknowledging.

Productivity in itself is an incomplete measure, he warns, as AI tools may save time for the person generating the work, but can create downstream verification or clean-up work that impacts the bottom line.

Context matters before more data does

The scariest thing about AI is not the old cliche of ‘garbage in, garbage out,’ Schein explains, but rather the danger of “clean in, garbage out.’ As is again common AI folklore by now, most organizations don’t the necessary clean data foundations in place to support their AI ambitions.

Even if they have got the building blocks in place, AI’s probabilistic nature can still cause it to go off the rails and produce garbage. It’s important to have the verification and context pieces, along with some more deterministic processes.

But rather than succumb to a temptation to throw more data at it, it’s more important to provide more context for the data you do have, suggests Schein:

We've been doing some work to look at old dashboards and apps that we have, and then generate custom chat agents off of that. At some point, some human thought these were the metrics. This is what you should look at. All these things, whether everyone looked at them, whether some of them were buried two clicks down, all those things. And so, we bring those forward.

But as you do that, it's like, ‘Do I need unstructured?’. Yes, unstructured data is a huge piece of this, but sometimes I feel like there's a lot still in structured data that, with the right instructions, the right skills, the right guardrails, there's a lot that could be there.

Once an organization has that context, it can drill deeper and add more unstructured data to it. Schein says there is value in adding more data, but cautions that without verification, context, and processes, again there is the risk of going off the rails.

Of course, organizations have struggled to get their data foundations in order for a long, long time, despite handing over squillions to tech vendors to ‘fix’ the problem. Schein calls it “the forever question”, but after 25 years in the data business, he argues that cleanliness issuesare solvable. But there are common problems:

It's always this dis-connect in my mind between the business context, the business value, the people that actually understand what's going on, and what gets designed and implemented in the data.

And here’s another complication - AI makes the data problem both worse and better. On the one hand, it can help clean the data, but on the other hand, it can scale a problem really fast. Schein says:

Make one change in how the ecosystem works and AI multiplies that times 1,000, so the risks of a mistake get much greater with AI because it's not fully in human control and it doesn't take time off.

In part two of this interview, why data foundations may be vital, but there’s no need to wait for perfect data.