A support engineer on a fintech product I advised on once pasted me a transcript that looked, at first glance, completely fine. A user asked a RAG-powered assistant: "What was our revenue growth from Q1 to Q2?" The bot retrieved the right earnings snippet, quoted the right two numbers, and then confidently reported a growth percentage that was wrong by almost double.
Nobody had done anything unusual. The retrieval step worked exactly as designed. The failure was assuming that because the model found the right facts, it could also reason correctly about them.
What RAG actually promises
Retrieval-augmented generation solves one problem well: getting relevant, current, and specific text in front of a language model before it answers. That's it. It is a search-and-stuff pipeline — embed the query, pull the nearest chunks from a vector store, drop them into the context window, and let the model generate a response grounded in that text.
What it does not do is turn the underlying model into a calculator. The model is still predicting the next token based on patterns in its training data. Arithmetic, multi-step percentage calculations, unit conversions, and anything involving carrying digits across several steps are exactly the class of task large language models are structurally bad at, retrieval or not.
Why this surprises people
It's easy to conflate "has the right inputs" with "produces the right output," because for most RAG use cases — summarization, Q&A over docs, support ticket triage — having the right inputs is basically the whole job. The generation step is just rephrasing. So teams ship a RAG bot, watch it nail a dozen qualitative questions, and assume the quantitative ones will follow the same reliability curve. They don't. A model can retrieve "$4.2M" and "$3.1M" perfectly and still botch (4.2 - 3.1) / 3.1.
The fix is not a better retriever
I've watched teams respond to this by re-chunking documents, upgrading embedding models, or adding more context — none of which touch the actual bug, because the retrieval was never broken. The fix is to stop asking the language model to do the math at all.
# Instead of asking the LLM to compute this inline...
prompt = f"Revenue was {q1} in Q1 and {q2} in Q2. What was the growth rate?"
# ...extract the operands and hand the arithmetic to code.
def growth_rate(q1: float, q2: float) -> float:
return (q2 - q1) / q1
answer = growth_rate(q1, q2)
response = f"Revenue grew {answer:.1%} quarter over quarter."In practice this means treating numeric reasoning as a tool call, not a generation task. Give the model function-calling access to a calculator, a small Python sandbox, or — if the numbers live in a database — let it generate a SQL aggregation instead of eyeballing rows of retrieved text. The retrieved chunks become the source of truth for values; a deterministic function becomes the source of truth for computation. The LLM's job shrinks to the part it's actually good at: figuring out which values are relevant and phrasing the final answer in plain language.
A rule of thumb
If a question can be answered by finding a sentence, RAG alone is fine. If a question requires combining two or more retrieved numbers with an operation — a ratio, a delta, a sum across rows, a currency conversion — assume the model will get it wrong some nonzero percentage of the time, and route that step to code. It costs one extra function definition and saves you from shipping a bot that sounds authoritative while being quietly, confidently incorrect.
RAG is a retrieval architecture, not a reasoning engine. Once a team internalizes that distinction, the fix is usually a day of work, not a redesign.
I write about the systems behind shipping AI features — RAG pipelines, evals, and the gap between demo and production — in my newsletter, AI Shipped. New issue every week.