A 120-person wealth management SaaS company committed to its board that AI would reduce customer onboarding time by 60%. The AI delivered. The infrastructure costs did not — and the "cost reduction" story became a net cost increase that the CEO had to explain at the worst possible moment.
The company ran an AI-assisted onboarding flow that had, by any model metric, worked. Document verification time dropped from an average of four days to under eighteen hours. Customer satisfaction scores improved. The onboarding team processed forty percent more applications with the same headcount. The demo to the board at the end of Q2 was well received. The CEO had delivered what had been promised.
The CFO's slide that followed it was less well received.
The original AI business case had modelled infrastructure costs at roughly $180,000 per year — an estimate built on training costs, with a rough multiplier applied for inference. The actual annualised infrastructure bill by month seven was $740,000. The system had been provisioned on GPU instances sized for peak load, and peak load had never been accurately forecasted. The instances ran at full capacity continuously, because no one had built autoscaling or set up scheduled scale-down for off-hours. The original model — a large general-purpose LLM called via API — was running full inference for every document, including the trivial ones.
"The AI worked. Nobody disputes that. What didn't work was the assumption that an AI system costs whatever you paid to train it. The inference bill is a different animal entirely — and we hadn't looked at it."
The core problem was architectural, not operational. The system had been built by a capable engineering team making reasonable decisions in the wrong order. Model selection came first. Infrastructure sizing came second — provisioned to handle the highest anticipated load, with no autoscaling, because autoscaling had been identified as a "phase two" item. Cost modelling came last — and when it arrived, it was based on training cost rather than inference serving cost, because the team lead who built the model had never operated an inference-heavy production system before.
The result was a serving architecture that used a $0.04-per-call model for documents that could have been handled by a $0.0008-per-call model, on infrastructure that ran at full cost twenty-four hours a day including weekends, processing a document mix where approximately 80% of submissions were straightforward enough that they didn't need LLM processing at all.
The first conversation was with the CTO, and it needed to happen before the board response. We needed to understand the document mix in production — not what had been assumed at design time, but what was actually flowing through the system. The logs told a clear story: 78% of documents were standard employment proofs and bank statements that matched known templates. 17% were edge cases requiring interpretation. 5% were genuinely complex — unusual document types, blended sources, foreign formats — that required full LLM reasoning.
That distribution meant the system had been built for the 5% and was running the 95% through the same expensive path. A model that is excellent at the hard cases is also processing all the easy cases at the same price, because nobody built a routing layer that decides which case needs which model.
We redesigned the serving architecture around a tiered inference approach. A lightweight classifier — not an LLM, a simple pattern matcher — ran first on every document submission. Documents matching known templates were handled by a fine-tuned, smaller model that cost a fraction of the original. Only documents the classifier flagged as non-standard were routed to the full model. We also added autoscaling with scheduled scale-down outside business hours — the company operated in one timezone, and the system had been running at full cost from midnight to 6am every night.
The engineering lift was eight weeks. The classifier required two weeks of calibration against the production document mix. The autoscaling configuration took three days. The smaller model was off-the-shelf; the fine-tuning took four weeks against labelled production examples.
The board presentation in the following quarter told a different story. Infrastructure costs were down 68%. The onboarding efficiency gains were intact — accuracy had shifted by less than two percentage points, within the tolerance that the underwriting team had pre-agreed. The net P&L position of the AI programme moved from negative to positive for the first time since launch.
The CEO asked me what the single highest-leverage change had been. My answer was the classifier — not because it was technically sophisticated, but because it was the thing that nobody had built in the original design, despite the fact that the business case had implicitly assumed its function. The business case said "AI will process documents faster." It did not say "AI will process every document the same way, regardless of complexity." Those two statements produce radically different infrastructure cost curves. The first is what got approved. The second is what got built.
Most AI business cases are built on training cost estimates. The cost that matters at scale is inference cost — and inference cost depends on document mix, model routing, provisioning discipline, and serving architecture decisions that are usually made after the business case has been signed. The gap between those two numbers is where board confidence goes to die.
The right time to model inference cost is before the model is selected. The right time to design the routing layer is before the infrastructure is provisioned. The right question to ask in the business case review — before anyone approves the budget — is not "does the model work?" but "have we modelled what it costs to run this at production volume, on the actual distribution of inputs we expect, with autoscaling, for twelve months?" That question takes an afternoon. The alternative takes two quarters and a difficult board conversation.