Every executive eventually asks the same question, “How do we know whether our AI investment is actually working?”

The responses usually seek to reassure leadership.

“We’ve deployed 20,000 Copilot licenses.”

“Eighty percent of employees are experimenting with AI daily.”

“We’ve trained 6,000 people.”

“We’ve launched 200 AI pilots.”

Those numbers sound impressive. They also tell you almost nothing about whether the organization is performing better.

For decades, we’ve managed organizations by measuring activity because activity is easy to count, measuring outcomes is hard. Our default sense of progress was projects completed, hours worked, meetings held, features shipped.

Now we’ve simply replaced those measures with prompts written, licenses activated, models deployed and agents created. Different technology. Same management mistake.

AI isn’t valuable because it’s deployed. Like every major technology shift before it, its value comes from changing how the organization operates, makes decisions and delivers better customer experiences and results.

So why are we counting all our activity again?

Table 1: Activity vs. Outcomes

Table 1: Activity vs. Outcomes

The Dashboard Is Giving Executives False Comfort

When a CEO asks, “How are we doing with AI?” and the answer usually arrives as a dashboard. Usages, adoption, training ran, and technology deployed. Those measures have a purpose, but it’s not performance. If you’re investing in a capability, you should know whether people can access and use it. But don’t confuse awareness, access and adoption measures with outcome measures.

Knowing that 80 percent of employees used Copilot tells you something about adoption. It tells you nothing about whether that employee, or team is more effective. Even less if decisions are better, cycle times have fallen, revenue has increased, costs have declined, risk has reduced, or if customers are happier with the quality of service.

That’s the distinction leaders need to make.

Every company can increasingly access the same foundation models from OpenAI, Microsoft, Google, Anthropic and others. Competitive advantage doesn’t come from having access, it comes from what your organization becomes capable of doing differently because of access.

Management Systems Create Behavior

One lesson has remained true throughout every transformation we’ve led, organizations behave the way they’re designed to behave.

And that matters enormously in the age of AI because success will not be about the technology. Your AI transformation is a behavior transformation.

Giving people access to ChatGPT, Copilot or Claude doesn’t automatically change how they think, make decisions, collaborate or create value for customers. Those changes require people to challenge established habits, rethink how work gets done and develop new ways of working with AI.

Yet most organizations are still measuring the technology rather than the behaviors they want to change.

Every scorecard tells people what success looks like. Every KPI directs attention. Every incentive reinforces what matters.

Measure utilization and people maximize utilization. Measure activity and people produce more activity. Reward AI adoption and people find more opportunities to use AI, whether or not it improves the work.

But when leaders measure customer outcomes, the decision quality of teams, cycle times and organizational learning, they encourage an entirely different set of behaviors.

Teams start questioning unnecessary work. They challenge existing processes. They rethink how decisions are made, where human judgment creates value and how AI can help them achieve better results.

That’s the leadership opportunity.

Because what leaders choose to measure shapes how people behave. How people behave determines whether AI creates business value.

Diagram 1: What You Measure Shapes What You Get

Diagram 1: What You Measure Shapes What You Get

A Lesson We Learned Before Generative AI Arrived

One conversation during COVID fundamentally changed how John thought about leadership and measurement.

As Vanguard’s workforce shifted home almost overnight, the Board understandably wanted reassurance that people were still productive. One idea surfaced that was appearing across many companies at the time: measure how long employees were online or connected to the network. It was measurable, it was available and it looked like productivity.

When the conversation reached the technology organization, John pushed back.

“I don’t want to measure activity. I want to measure outcomes.”

Instead of asking whether someone was connected for eight or ten hours, the leadership team could ask questions that actually mattered.

  • How quickly were new client-facing capabilities being deployed?
  • How frequently were teams getting real customer feedback?
  • How quickly was that evidence being turned into improvements?
  • And ultimately, were clients experiencing something better?

That changed the conversation. During that period, software deployments increased by more than 40 percent while client satisfaction with digital interactions improved as teams turned feedback into improvements faster.

The lesson wasn’t about remote work. It was about management. The easiest thing to measure is rarely the most important thing to manage. That lesson has become even more important with AI.

Case Study - Vanguard during COVID

Case Study – Vanguard during COVID

Start With the Outcome, Not the AI

Here’s the discipline we’d encourage every executive team to adopt. Before approving another AI pilot, write down the business outcome you expect to change.

Not:

“Use AI to improve customer service.”

Instead:

“Reduce average customer resolution time from 48 hours to 24 hours while maintaining or improving customer satisfaction.”

Not:

“Use AI to improve executive decision-making.”

Instead:

“Reduce the time required to assemble a decision brief from five days to one while increasing the number of credible alternatives considered before major investment decisions.”

Not:

“Use AI to accelerate product discovery.”

Instead:

“Double the number of customer assumptions we can test each month without increasing team size, and reduce the time required to invalidate a weak idea.”

Now you have something you can manage.

Diagram 2: Baseline → Intervention → Outcome → Learning

Diagram 2: Baseline → Intervention → Outcome → Learning


AI becomes the intervention, not the outcome. That’s a fundamentally different management conversation.

What Better Outcomes Look Like

We’ve seen this discipline emerging across very different organizations.

At American Airlines, Misty Shafer Sterne wasn’t interested simply in whether executives used AI. The question was whether leaders could arrive at important conversations better prepared, with richer context, stronger alternatives and greater confidence in the decisions they needed to make.

But “better prepared” alone isn’t yet a metric. To manage it, leaders need to ask what observable consequences should follow.

  • Does preparation time fall?
  • Do decisions move from question to commitment faster?
  • Are fewer decisions reopened because critical information was missed?
  • Are more credible alternatives considered?
  • Does execution begin sooner?

That is the difference between an aspiration and an outcome.

At Progyny, Cassandra Pratt and her team used their HR Bot for more than answering employee questions. Interactions became evidence. Recurring questions exposed confusing policies, weaknesses in onboarding and moments where people still needed human support.

Again, the opportunity is to follow the evidence.

  • Are repeated questions declining?
  • Are policies becoming easier to understand?
  • Are HR escalations changing?
  • Are employees reaching the right support faster?

At Nobody Studios, we don’t ask founders how much AI they used this week. We ask how they improved their decision making systems last week.

  • Did you discover any new from customers?
  • Test more assumptions?
  • Invalidate weak ideas earlier?
  • Reach evidence faster?

Venture creation is ultimately a race to learn.

AI isn’t the outcome. Learning is.

Table 2: From Aspiration to Observable Outcome

Table 2: From Aspiration to Observable Outcome

The Artificial Organizations Outcome Scorecard

So what should executives measure?

In Artificial Organizations, we argue that the goal isn’t to deploy more AI. It’s to build an organization that makes better decisions, learns faster, creates better customer outcomes, and gives people more capacity to apply human judgment where it matters most.

That gives leaders five areas worth examining.

  • Decision Quality Are we making better decisions because of AI?
  • Decision Velocity Are important decisions moving from insight to action more quickly?
  • Customer Outcomes Are customers experiencing measurable improvements?
  • Organizational Learning Are we improving how we work every time we use AI?
  • Leadership Capacity Are leaders spending more time where their judgment creates the greatest value?

These aren’t universal KPIs. They’re five outcome dimensions that help executives decide where to look. The next step is to make each one observable inside the work.

Diagram 3: Outcome Scorecard

Diagram 3: Outcome Scorecard


For example, don’t simply ask whether decision quality improved. Look at whether fewer decisions are reopened because critical information was missed, whether more credible alternatives are considered, or whether costly reversals decline.

Don’t simply ask whether decision velocity improved. Establish how long an important class of decision takes today, then measure whether the time from insight to commitment and action decreases.

Customer outcomes might already have established measures: resolution time, retention, satisfaction, conversion, complaints or another metric specific to the work.

Organizational learning might be visible in how quickly teams test assumptions, incorporate customer evidence, eliminate recurring problems or stop weak ideas.

Leadership capacity might be measured through reductions in preparation and administrative work, and whether that capacity is actually reinvested in customers, people, strategy and high-value decisions.

The distinction matters, the five dimensions tell you where to look. The outcome tells you what to measure.

Better AI outcomes start with understanding how your organization works today. Establish your AI baseline in 10 minutes, see where you stand against the market, and discover which behaviors could improve your decisions, performance and results.

Decision Advantage Is the New Competitive Advantage

AI is making information and content extraordinarily cheap to produce.

Reports, presentations, analysis and alternatives can now be generated at a volume executives have never experienced before. That doesn’t necessarily make organizations smarter. It can simply give them more work artifacts to process. As intelligence becomes abundant, judgment becomes more valuable.

  • Which ideas deserve attention?
  • Which opportunities should we pursue?
  • Which risks matter?
  • Which decisions should we make?

We call the ability to consistently make better decisions, faster than competitors Decision Advantage. And even Decision Advantage needs to be made observable.

  • How long does an important class of decision currently take?
  • How often are decisions escalated?
  • How frequently are they reopened?
  • How long is the gap between decision and action?
  • Where does missing context repeatedly slow the process?

Now AI can be applied against a management problem we understand rather than a technology opportunity we’re hoping will create value.

The Leadership Shift

For more than a century, organizations have measured what was easiest to count: hours, projects, utilization and output. We often fall back to it when faced with new uncertainty as John example for remote work initially at Vanguard.

AI gives executives an opportunity to become more demanding about what they manage especially when they manage outcomes.

Not, “How much AI are we using?”

But, “How are we getting better because we’re using it?”

And then, “What have we learned that changes what we should work on next?”

Organizations don’t become adaptive because they deploy AI. They become adaptive because leaders build systems that continually turn new capabilities into better outcomes and those outcomes into new learning.

Diagram 5: Adaptive Enterprise Loop

Diagram 5: Adaptive Enterprise Loop


Every turn gives the organization another opportunity to become smarter, faster and better.

The real advantage isn’t AI adoption.

It’s developing an organization capable of continuously discovering what works, changing what doesn’t and learning faster than the world changes around it.

Because ultimately, what leaders choose to measure tells an organization what matters. What leaders choose to learn determines how quickly it gets better.

– By John Marcante and Barry O’Reilly

Is AI Actually Making Your Organization Better?

You’ve invested in AI. Your teams are experimenting. But can you confidently say which behaviors are improving decisions, performance and business results?

That’s exactly what we’re exploring through the 2026 Artificial Organizations Research Program.

Building on insights from more than 5,000 executives, we’re investigating how leading organizations are redesigning work, strengthening human judgment and turning AI capabilities into measurable business outcomes.

Now it’s your opportunity to see where you stand.

In just 10 minutes, you can establish your individual AI baseline, compare your capabilities against the emerging global benchmark, and get early access to insights on the behaviors and management practices associated with better performance.

Because the next competitive advantage will come from understanding what works, learning faster and knowing where to focus next.

See Where You Stand. Establish Your AI Baseline in 10 Minutes →

FAQs

Q1: How should leaders measure whether AI is creating business value?

Leaders should measure changes in business performance, such as customer resolution time, decision quality, or speed of learning. Adoption and usage measures show whether people are using AI, but do not establish whether the organization is improving.

Q2: Why do AI metrics influence employee behavior?

Measures direct attention and signal what success looks like. Rewarding AI usage encourages more usage. Measuring customer outcomes, decision quality, and learning encourages teams to question existing processes and change how work gets done.

Q3: How can organizations establish a baseline for AI outcomes?

Record how the work performs today, then define the improvement expected from a specific change. For example, measure current customer resolution time and satisfaction before introducing AI. Compare the results with that baseline and use the evidence to guide the next change.

Q4: What is the Artificial Organizations Outcome Scorecard?

The scorecard identifies five outcome dimensions: decision quality, decision velocity, customer outcomes, organizational learning, and leadership capacity. These are areas to examine rather than universal KPIs. Leaders need to choose observable measures relevant to their organization’s work.

Q5: What does decision advantage mean in the article?

John and Barry describe decision advantage as consistently making better decisions faster than competitors. Leaders can make it observable by tracking decision time, escalations, reopened decisions, and the gap between deciding and acting, alongside evidence of decision quality.

References