The Real AI Opportunity Isn’t Productivity. It’s Reinvention.
Authored by Anna Cley, Founder of Stardust Immersive
The next stage of AI adoption isn't simply about what the technology can do. It's about deciding what we should allow it to do, and when it should know to stop.
I recently attended AI Isn’t the Transformation. Your Business Is., a Fast Company Custom Studio conversation presented by GS1 US.
The discussion covered many of the questions occupying business leaders right now: productivity, agentic AI, data, governance, business-process reinvention, talent, and trust.
But the most important insight came from Campbell Brown, Cofounder and CEO of Forum AI:
As AI becomes more capable, our standards need to become higher.
We spend an enormous amount of time asking what AI can do.
Can it automate this process? Can it make this decision?
Can it interact with this customer? Can it evaluate this candidate?
But capability alone is not a sufficient standard for deployment.
The more consequential question may be:
What should good AI look like inside our organization?
And increasingly, companies may have to answer that question for themselves.
We Cannot Simply Wait for Regulation
Campbell Brown raised one of the most important issues of the conversation: accountability.
Her argument was that industries don't necessarily have to wait for governments to establish every standard around AI. Companies and industries must begin developing standards, evaluation mechanisms, and accountability themselves.
Regulation has a role to play. But technology is evolving extraordinarily quickly, and regulation will not always move at the same speed.
Leadership therefore cannot mean simply asking, What are we legally required to do?
Organizations also need to ask:
What standard are we willing to hold ourselves to?
Brown described this as defining “what good looks like” for your organization.
Models will change. Platforms will change. The AI systems companies use three years from now may look very different from the systems they use today.
But an organization's standards should not depend entirely on the technology.
What level of accuracy is acceptable? What level of consistency?
Where is human review required? What decisions should never be delegated completely?
What happens when the system is uncertain? What constitutes unacceptable risk?
Those are leadership questions before they are technology questions.
Perhaps the Most Important AI Skill Is Knowing When to Stop
This was the part of the conversation that stayed with me most.
Brown explained that evaluating AI shouldn't only mean testing whether it produces the correct answer. We should also test whether it knows when not to answer.
AI can sound remarkably confident when it is wrong.
So a trustworthy system needs more than the ability to perform. It needs mechanisms for recognizing uncertainty.
Does it know when to pause? Does it know when to escalate?
Does it know when to ask for help?
Does it know when the decision should return to a human?
For years, technological progress has often been framed around expanding what machines can do independently.
But as AI becomes increasingly agentic, perhaps one measure of sophistication should be the opposite:
Can we design systems that understand the boundaries of their own authority?
Knowing when to stop may become as important as knowing how to proceed.
Human-in-the-Loop Needs to Be Designed, Not Added Later
This also changes how we think about the familiar concept of “human-in-the-loop.”
The human shouldn't simply be the emergency backup added after an automated process has already been designed. We need to intentionally determine where human judgment belongs.
Some decisions can be automated confidently. Others require context. Others require empathy. Some involve ambiguity or ethical judgment. And some may simply carry consequences significant enough that a human should remain accountable.
That means organizations need to understand the workflow before deciding what AI should do within it. Melanie Hilton of GS1 US captured this beautifully:
“You can't agentify something you haven't identified.”
Before deploying agents, organizations need to identify how work actually happens.
Where does data enter? Where are decisions made? Where does judgment matter?
Where could an agent act? And where should it stop?
That isn't simply automation. It is the deliberate redesign of the relationship between people and technology.
AI Transformation Is Also a Standards Transformation
Another important theme from the conversation was that AI transformation requires foundations.
Clean and connected data.
Clear workflows.
Governance.
Education.
Talent.
Trust.
Jaime Montemayor of General Mills described individual and team productivity through AI as increasingly becoming “table stakes.” The larger opportunity, in his view, is business-process and capability reinvention.
But reinvention also increases responsibility. If AI is simply helping someone draft an email faster, the consequences of failure may be relatively contained.
When AI begins participating in hiring, customer interactions, supply chains, operational decisions, or autonomous workflows, the stakes change.
The question is no longer simply:
How much productivity can we gain?
It becomes:
What kind of system are we building?
And therefore:
What standards should govern it?
Raising the Standard Also Means Raising Human Capability
If we expect AI systems to operate according to higher standards, the humans designing, deploying, supervising, and working alongside those systems also need stronger capabilities.
People need to recognize when an AI output should be questioned.
Managers need to understand when automation is appropriate and when judgment is required. Teams need enough understanding of AI to challenge its outputs rather than simply accepting them. Leaders need to establish principles that can guide decisions even when the technology changes.
This is why AI education cannot stop at teaching people how to prompt a model or use a new tool.
We need to develop judgment.
Critical thinking.
Discernment.
Creative problem-solving.
The ability to operate in ambiguity.
And the confidence to say: This is where the technology stops and human responsibility begins.
The Question for Leaders Is Changing
We are moving quickly from a period of AI experimentation into one in which AI can increasingly act. That makes the leadership questions different.
Not simply:
What can we automate?
But:
What should we automate?
Not simply:
How autonomous can this agent become?
But:
Where should its authority end?
Not simply:
Is the output accurate?
But:
Is the system trustworthy enough to know when it may not be?
And not simply:
Are we compliant?
But:
Are we meeting the standard we believe responsible organizations should meet?
Technology companies will continue to innovate. Models will continue to improve. But organizations cannot outsource all responsibility for what good AI looks like. We have an opportunity to define a higher standard ourselves. And perhaps one of the clearest signs that we're getting there will not be when AI knows how to do more.
It will be when AI knows when to stop, and when we know why it should.
Ready to Bring This to Your Team?
If you are responsible for the experience, performance, and retention of your people, this is a conversation worth having.
Contact us today at anna@stardustimmersive.io
Comments