I was brought in as a technical advisor to identify and analyze a reporting environment that was buckling under its own weight. Over several weeks we did deep, evidence-based analysis, evaluated eight solution paths, and converged on an architecture that was, by every technical measure, the right answer: a relational (star schema) dimensional model that preserved correctness while cutting the footprint.
The engagement still closed unsuccessfully. Not because the architecture was wrong, but because of two things that had nothing to do with data modelling: the organization was not ready to have a hard conversation with its own report consumers, and it expected the new model to be magic — testing it for raw speed before ever confirming it produced correct results.
This is the story of how a technically sound solution can fail, and the signals I will never ignore again.
Correct architecture is necessary, but not sufficient. What actually decided success was on the other side of the scale.
The Engagement
A large, mature reporting environment under its own weight
The client was a multi-brand consumer subscription business. Their analytics team ran a large, mature reporting environment built on a single, very wide reporting source — roughly 70 GB, several hundred columns, and over 200 million rows, feeding hundreds of connected dashboards and workbooks. It was assembled by a single, sprawling stored procedure on the order of 9,000 lines that rebuilt the whole thing on a schedule.
The presenting problem was familiar: the nightly data refresh kept failing, data arrived late in the day, and the environment was pushing against the size limits of their cloud analytics platform. I was asked to find a way to make it reliable and sustainable.
The Approach
Eight options, one hard constraint, one clear answer
I did the work properly. I reverse-engineered the procedure, profiled the volume and width, audited field-level usage across hundreds of fields, and reconciled every recommendation against the client’s own run logs and workbook internals rather than assumptions. I framed eight distinct solution options — from simply pre-aggregating history, through incremental refresh, to a full platform shift — and rated each on complexity, achievement, cost, and timeline.
One hard constraint shaped everything: because the same customer and order identifiers recurred across products, any approach that summarised above the transaction line broke the distinct counts the business depended on. That single fact ruled out the “just make it smaller” answers and pointed to a dimensional model that kept the grain intact while moving repeated width into small, reusable dimensions.
The proof-of-concept had one job above all others: demonstrate that the new model returned the exact same distinct counts as the current system while being smaller and lighter. That correctness check was designed to be the gate before anything else.
The finalised recommendation was a logical star schema built in the analytics tool’s relationship layer — most of the benefit of a full re-architecture, achievable in weeks rather than months, with correctness preserved. I built field-level model maps, mapping documents, diagrams, and a working proof-of-concept. By the numbers, the analysis was some of the most thorough work I have done. And that is precisely why the outcome was instructive.
Where It Unravelled
Two failure modes, neither of them the architecture
The real problem underneath the failing refresh was not a modelling problem. It was demand. Over years, every new report request had been answered by bolting another requirement onto the same monolith. The backend was not “broken” so much as it was carrying the weight of an ungoverned set of consumer requirements, no two of which had ever been rationalised against each other.
A durable fix required the client to go back to their report consumers and say, in effect: “The way we have been giving everyone everything is what is blowing up the source. We need to sit down, organize these dashboards around specific business problems, and define the specific data each one actually needs.” That is a political conversation, not a technical one — and the organization was not willing to have it. Without it, any backend fix was treating a symptom while the cause kept growing.
A backend fix treats the tip of the iceberg. The mass — years of ungoverned reporting demand — keeps growing until someone manages demand rather than symptoms.
The team expected the relational model to deliver dramatic results the moment it was connected. Instead of first running the correctness check the proof-of-concept was built for — does it reconcile, does it return the right numbers — they jumped straight to stress testing it for raw performance. When early, untuned performance did not look magical, they concluded the approach itself was unviable.
This inverted the entire validation logic. A logical model’s value is proven first by correctness and footprint, then tuned for speed. Leading with a stress test on an unproven, untuned model almost guarantees a disappointing first result — and a disappointing first result, framed as “the model does not work,” is very hard to recover from.
The sequence of testing determines whether a project earns stakeholder confidence or loses it entirely. Never let the first test of a new model be a stress test.
The Root Cause
Two forces. Neither was the architecture.
If I strip away the specifics, two forces closed this project, and neither was the architecture:
An organizational readiness gap. The solution required the client to manage demand and reset expectations with their own stakeholders. They preferred to keep the peace with consumers rather than renegotiate what those consumers received. A technically correct solution that depends on organizational change will not survive an organization unwilling to change.
A trust-and-expectations gap. “Magical results” set an impossible bar, and a premature stress test gave everyone a reason to walk away before the model was ever judged on its actual merits. Once trust erodes, the quality of the underlying work stops mattering.
What to Watch Out For
Seven signals and safeguards for advisory work
These are the signals and safeguards I took away — useful for anyone doing advisory or platform-modernisation work.
Seven watch-outs for advisory and platform-modernisation engagements — the reference checklist for anyone doing this kind of work.
What I’d Own, and Do Differently
It would be easy to file this under “the client was not ready”
That would be partly true and entirely useless. What I would do differently is mine to own:
I would have assessed organizational and political readiness as deliberately as I assessed the stored procedure — and surfaced the demand-governance conversation as a precondition, not a footnote.
I would have made the correctness gate a hard, agreed checkpoint with a named owner, so that “we stress-tested it and it was slow” could not become the first and final verdict.
I would have managed the expectation of “magic” more forcefully and earlier, in writing, with the client’s leadership in the room — not just their practitioners.
The hardest work was not the model. It was the conversation about who gets what, and why.
Closing Reflection
The first genuinely less-successful project of my career
This was the first genuinely less-successful project of my career, and I have come to value it more than several that went smoothly. The engineering was sound; the analysis was thorough; the model was right. And none of that was enough, because success in advisory work is decided as much by readiness, expectations, and trust as by architecture.
Solving the right problem the right way still fails if the organization is not ready to change and the expectations were never set.
I no longer treat those as soft, secondary concerns. They are the engagement.
If you are wrestling with a reporting environment that keeps buckling, the hardest and most important work may not be the model at all. It may be the conversation about who gets what, and why.