ImplementationNewsFor: Business operations · Customer support · IT and service desk

A CIO re-tested his 25 AI agents' results; two of three numbers weakened

A CIO put three headline results from 25 AI agents through five checks and one held. Operations leads can run the same checks before funding phase two.

Naren Gangavarapu, a technology executive who says he spent 18 months putting 25 AI agents into production at a large travel and cruise group, [wrote in CIO on 18 November 2026](https://www.cio.com/article/4249739/your-agentic-ai-results-cant-be-checked-heres-how-to-fix-that.html) that two of the three results his team used internally to justify further investment came out weaker once he checked how they had been measured. He does not name the company; [an earlier column](https://www.cio.com/article/4219780/the-ai-employees-are-already-on-the-floor-is-anyone-watching.html) calls it "one of Australia's largest tourism and cruise operators".

The three numbers

  • Tier 1 support escalations fell 34%. "This largely survives," he writes: the six-month period was fixed to include peak season, contact volume was flat year over year, and the counts came from the system of record and were reviewed by finance, not the agent team. The caveat is that "Tier 1" is the company's own category, so the figure cannot be set beside another company's.
  • Operator onboarding fell from 23 days to 3. The 23 days included time waiting in a standing backlog, so part of the gain is a queue that was cleared once, not a shorter process. He does not say how many of the 23 days were backlog.
  • Booking conversion rose 24%. He calls this "the weakest of the three": a marketplace rebuild and opt-in dynamic pricing ran in the same period as the AI search feature credited with most of the rise.

The five checks

Gangavarapu calls the checks CLEAR and says they need only ordinary operational records:

  • Concurrent changes: what else changed in the same period, and could it have produced the result on its own?
  • Locked window: was the period fixed before the result was known, and does it include the busiest trading conditions?
  • Custody: which system produced the count, who defines the category and who reviewed the figure?
  • Baseline: how much of the gain came from changing the process, and how much from clearing a backlog?
  • Volume: did the underlying volume hold steady, and is it stated?

Fixing the period in advance "does most of the work", he writes, because it stops a team picking the stretch that flatters the result afterwards.

The override nobody logged

Any line manager, "from the wholesale desk to the cruise operations floor", could stop an agent within 90 seconds with no ticket and no approval. The risk he describes is the stop or correction that is never recorded: the hard cases a careful person pulled out disappear from the agent's statistics. "Escalation rates fall because escalations were intercepted first."

Of the 25 agents, seven could act without a person confirming; 13 needed confirmation or were barred from contacting customers on their own. He does not describe the other five. Managers with agents kept a five-minute morning review of overnight agent activity, a weekly scorecard (completion, escalation rate, the agent's confidence in its decisions) and a monthly session checking the agents' limits against incidents.

The catch

  • This is one executive's account of an unnamed company. "We withheld volumes," he writes, so the 34% has no ticket count behind it.
  • No cost is given: not licences, build effort or staff time for 25 agents over 18 months.
  • The columns say line managers became accountable for the agents. They do not say what happened to support staff, headcount or anyone's job.
  • CLEAR is his own checklist. It "isn't a validity standard", he writes: a result that passes all five is still one organisation's observation.

What to do before you fund the next phase

  • AI programme and business operations leads: before the next reporting period starts, write down the period, the volume each rate will be divided by and every other change scheduled inside it, and have finance review the counts.
  • Support leads: open the override log itself this week. If a supervisor can stop or correct the agent without leaving a record, treat your current escalation and resolution rates as overstated until that is fixed.
  • Anyone reading a vendor's customer story: put the five questions to the vendor, starting with the period and the volume.
Was this useful for your work?

Take this to work

Forward this

A CIO who ran 25 AI agents at an Australian travel and cruise group for 18 months writes in CIO (18 November 2026) that only one of his three headline results, support escalations down 34%, survived checks on period, volume and other changes. Before we report or accept an AI result, we should fix the measurement period in advance, state the volume and confirm that human overrides of the agent are logged.

Questions to ask

  1. Was the measurement period fixed before the result was known, and does it include our busiest month?
  2. What was the contact or transaction volume in the before period and the after period?
  3. What else changed in that period: pricing, a platform rebuild, a reorganisation, a backlog cleared once?
  4. When a person stops or corrects the agent, where is that recorded, and are those cases counted in the escalation rate?
  5. Which system produced the count, and who outside the AI team reviewed it?

The arithmetic

escalations in the period ÷ contacts in the period, before and after, over a period fixed in advance and with overridden cases counted

The source gives a 34% fall in Tier 1 escalations over six months with contact volume flat year over year, but withholds the counts, so escalations per contact cannot be worked out. Onboarding went from 23 days to 3, 20 days shorter (23 − 3); the source does not say how many of the 20 were backlog.

Questions this article answers

What is the CLEAR standard for AI agent results?

It is a five-question checklist proposed by Naren Gangavarapu in CIO on 18 November 2026: concurrent changes, a measurement period locked in advance, custody of the count, an attributable baseline and a stated reference volume. He says it is not a validity standard; a result that passes is still one organisation's observation.

How do I know whether an AI agent's escalation rate can be trusted?

Gangavarapu's test is whether the period was fixed before the result was known, whether contact volume held steady and is stated, and whether a system of record produced the count and someone outside the AI team reviewed it. He also says to inspect the log of human overrides, because unlogged interventions remove the hard cases from the agent's statistics.

Which of the travel group's three AI results held up?

By the author's own account, the 34% fall in Tier 1 support escalations over six months largely held. The onboarding cut from 23 days to 3 included a cleared backlog, and the 24% rise in booking conversion coincided with a marketplace rebuild and new dynamic pricing.

Send this to someone who runs a teamLinkedInXBlueskyHNEmail