Build vs Buy: Why an LLM Alone Won’t Give You Accurate Customer Insights

‍

Every leadership team has heard the pitch by now: first, export your tickets, calls, reviews, and surveys - or just connect your favorite LLM / AI harness (Claude, ChatGPT, Gemini) to their MCP.  Then, ask it anything. Ship "customer intelligence" in a weekend, skip the vendor, keep the budget.

It is the most tempting idea in CX right now. Over the eleven posts of our Build vs Buy series, we took it apart one capability at a time. This wrap-up puts the whole argument in one place, with the receipts.

Here is the short version: accurate customer insights do not come from the model, they come from the customer context underneath it. When we tested a raw LLM against structured classification on 136,864 real customer reviews, the raw approach produced a different top issue on every run, scored 30 points lower on accuracy, and burned most of its compute redoing the same work. If a real decision rides on the output, as it often does, that is not a rounding error. In our study, the gap between two plausible readings of the same data was the difference between fixing a real problem and funding an expensive fix for a symptom.

That is the stakes. Now, let’s look at the evidence.

The test: 136,864 reviews, three approaches, three runs each

We wanted to know what actually happens when you point an LLM at raw customer signals at production scale. So we built the experiment most build-vs-buy debates skip.

We took 136,864 real marketplace reviews across six major marketplaces and asked the questions every CX and product leader asks: what are the top issues, how big is each one, and where should we act first. We ran three setups side by side:

1. A structured customer context layer. Signals classified up front against a governed taxonomy, with the LLM reasoning on top of that structure. This is how Birdie works.
2. An LLM handed the exact same taxonomy. No structure, but every advantage a smart internal team could give it, including our own category definitions.
3. A raw LLM on raw data. The weekend project. Connect the data, write a good prompt, ask.

Every setup answered every question three times. We scored accuracy against held-out, ground truth labels from human experts. Same data, same questions, same model family. The only variable was structure.

Five ways the raw LLM route gets customer insights wrong

1. Ask the same question twice and you get two different answers

Reproducibility is the first thing to die. Across three runs, the structured approach returned the same top-10 issue list every time: 100% overlap. The LLM with the taxonomy managed 67%. The raw LLM managed 0%. Not "a little unstable." Zero overlap. A different number one issue on every single run, even when you play with model temperature, seeds and so on.

Think about what that means operationally. Monday’s board slide and Thursday’s board slide disagree, and both came from "the data." Whichever run you happen to trust becomes your strategy.

2. Coverage is not accuracy

LLMs sample because they have to. Your signals may not fit in a context window, so the model reads a slice and tells a confident story about the whole. You can engineer around the limit with chunking and data-savvy pipelines, and teams do. But then a subtler problem shows up: the labels themselves are wrong more often.

‍

Measured against expert-labeled ground truth, the structured layer scored an F-score of 95. The LLM working with the very same taxonomy scored 65. The F-score matters because it balances precision (how many answers were actually right) and recall (how many it missed). One score can look fine while hiding both failures, which is exactly why "LLM-as-judge" evaluations that report neither should not comfort anyone.

This is the coverage paradox: labeling more volume with a weaker classifier does not give you more truth. It gives you more wrongly labeled volume, delivered faster.

3. Even with your taxonomy in hand, the counts do not hold

The most surprising result was not the raw LLM failing. It was the well-equipped LLM failing. We handed it Birdie’s exact taxonomy, the same category definitions our structured pipeline uses, and asked it to size the issues.

The same issue, "delivery delay," was counted 1,385 times in one setup and 7,481 times in the other. That is a 5.4x difference in the size of the same problem, from the same data, using the same definitions. Prioritize by one count and delivery delay is a niche complaint. Prioritize by the other and it is the house on fire.

A taxonomy is not a document you attach to a prompt. It is a discipline you apply identically to every signal, every day, and govern as categories evolve so that this quarter stays comparable with last quarter. That governance is the part you cannot prompt your way into.

4. Cross-source questions compound the error

Real decisions rarely live inside one dataset. The questions that move budgets cross systems: do the customers complaining about onboarding also show up in churn, does the issue spiking in reviews match what agents hear on calls. Every join between an unstructured source and another system loses accuracy, and the losses multiply. In our study, cross-source accuracy landed near two-thirds end to end. The weakest link is always the unstructured step, and no amount of prompt engineering fixes it, because the problem is not the reasoning. It is the ground the reasoning stands on.

5. You pay the model to redo the same work, forever

On raw data, an LLM spends 68-75% of its compute re-structuring the same signals on every single run, before it can even start answering your question. Structure once, and every question after that is cheap. In our measurements the structured approach came out up to 5.3x cheaper per analysis.

So the build route does not just cost you an engineering team to wire and babysit pipelines. It also rents the same understanding over and over, at inference prices.

Raw LLM vs. customer context layer, side by side

It is not a model problem. It is a context problem.

None of this is an argument that LLMs are bad. The models are remarkable, and they are only getting better. It is an argument about what they are standing on.

An LLM answering questions about your customers is doing two jobs at once: understanding what each signal means, and reasoning about the pattern. When you feed it raw exports, you are asking it to redo the first job from scratch on every question, invisibly, with no ground truth, no consistency guarantees, and no memory of how it did it last time. Everything downstream inherits that instability.

A customer context layer does the first job once, properly. It pulls every signal your customers send, across tickets, calls, reviews, surveys, CRM notes, and product analytics, into one place. It classifies 100% of them, not a sample, against a governed taxonomy that means the same thing to every team and stays comparable as it evolves. It connects those signals to each other and to the numbers leadership acts on, like the ARR at risk behind an issue. And it keeps an auditable score: F-scores measured against expert labels, not a model grading its own homework.

That structured foundation is what our customer intelligence product builds on, and it is why the answers hold still between Monday and Thursday. The loop we kept returning to throughout the series runs on it: Signal, Diagnose, Act, Prove, Learn. Signals come in from every source in your stack, get structured and quantified, drive an action, and the impact of that action gets measured against the same stable categories that surfaced it. That last step, proving a fix worked, is only possible when this quarter’s categories mean the same thing they meant last quarter.

And in an AI world this matters more, not less. Everyone can build fast now. Very few teams are building on coherent foundations. Structured context is becoming the scarce asset, because it is the one thing the model cannot generate for itself.

The answer is not build or buy. It is build on top of buy.

Here is where the series lands, and it is not "never build."

The teams getting the most out of AI are doing both. They buy the hard, specialized layer: ingestion at scale, full-coverage classification, taxonomy governance, accuracy measurement, the connection between signals and revenue. Then they build freely on top of it, because now their LLMs, agents, and internal tools have solid ground under them.

That is exactly what the MCP layer is for. Anyone on your team can ask a question in plain language and get an answer grounded in 100% of your structured signals, not a sample of raw text. Your analysts, your agents, and yes, your LLMs, all query the same governed context. The weekend project becomes viable, because the part that made it dangerous is handled underneath.

DIY the workflows. Do not DIY the context.

"Can’t we fix this with better prompts, RAG, or fine-tuning?"

This is the objection we hear most, and it deserves a real answer.

Prompting, retrieval, and fine-tuning all improve the reasoning job. None of them create ground truth. RAG retrieves chunks of the same unstructured text and inherits the same ambiguity. Fine-tuning teaches the model your tone and your domain, not a consistent, governed count of what happened last month. Better prompts reduce variance; they do not eliminate it, and you will not know when it bites, because there is no audited score telling you.

The second objection is quieter: "our data team can build the structured layer too." They can, and some should. But be clear-eyed about the scope. It is not a pipeline, it is a product: classification quality measured continuously against human labels, a taxonomy governed through every reorganization and product launch, joins maintained across every source as formats change, at millions of signals a month. Wiring it up takes a weekend. Making it return the same answer twice, at 95 accuracy, for years, is a company’s worth of work. We know because it is the company we built.

Stop betting strategy on a coin flip

Eleven posts, one conclusion. Any LLM can produce customer insights. Only structured context can make them accurate, reproducible, and worth acting on.

‍

If your top issue would not survive a second run, it is not an insight yet. See what your signals look like on a real foundation: book a demo.

‍