Article
Sometimes infrastructure problems are not solved by more optimisation or more scale
There is a point where another round of tuning will not help, and the architecture itself has to be rethought.
The usual response to an infrastructure problem is to tune it, then to scale it. Both work — until they stop working, and every additional round of effort buys less than the one before.
That plateau is a signal, not a failure. It usually means the constraint is in the design, not in the configuration.
How to recognise the plateau
- Cost grows faster than throughput.
- Each optimisation gains a few percent and adds a new operational edge case.
- Latency is dominated by waiting — for a queue, a lock, an external call — not by compute.
- Failures cluster in the same seam between components every time.
- The team spends more time nursing the system than improving it.
What rethinking the architecture looks like
It rarely means a rewrite. More often it means moving a boundary: making a synchronous path asynchronous, changing where state lives, precomputing what was being recomputed, or splitting one workload that had two conflicting profiles.
In AI workloads specifically, it often means separating the fast interactive path from the heavy batch path, and putting a governed gateway between the applications and the models so routing, caching and cost control stop being application concerns.
Deciding rationally instead of emotionally
Set the threshold before you need it: if the next optimisation cycle does not deliver a defined improvement, the work shifts to a design review.
That single rule prevents the most expensive pattern we see — a year of heroic tuning around a constraint that a two-week redesign would have removed.
Optimisation and scale are the right first answers. Knowing when they have become the wrong answer is what keeps systems — and budgets — healthy.
This article is the English version of a post originally published in Hebrew on our LinkedIn page.