Most enterprise AI data platforms are evaluated the way a school evaluates a student who only ever takes multiple-choice tests. Ask it to find every transaction over a threshold, and it does that instantly. Ask it to summarize what a compliance document says, and it does that too. Both look like competence. Neither is the test that matters.
The test that matters is the one where the correct answer requires both at once, in the same response, reconciled against each other. This is not an edge case dressed up to sound hard. It is the shape of almost every consequential question a BFSI operations, finance, or compliance team actually asks.
The Two-Mode Problem Nobody Names
Ask an AI data platform "what is the interchange fee for a Mastercard debit transaction above a certain threshold on a government merchant," and watch what happens. A platform built around text-to-SQL will find every matching transaction in seconds and stop there, because it has no mechanism for knowing that government merchants sit in a different fee tier, a fact that exists only in a fee schedule document, not in any column of the transaction table. A platform built around document retrieval will find and quote the fee-tier language accurately and stop there too, because it was never given query access to the transaction table that the rule needs to be applied to.
Neither platform is malfunctioning. Both are doing exactly what they were architected to do. The problem is that the question needed both, joined, and nothing in either architecture does that joining.
This pattern is not specific to fee calculations. It shows up in delinquency analysis, where classifying a loan account correctly requires reading both the overdue-days data and the regulator's current classification thresholds, which have changed more than once in recent years. It shows up in covenant monitoring, in regulatory exposure calculations, in nearly every question where a number only means something once a rule has been applied to it. Once a team starts looking for this pattern deliberately, it becomes hard to find a genuinely consequential business question in BFSI that does not have this shape somewhere inside it.
Why This Stays Invisible Until It's Expensive
The reason this gap survives most vendor evaluations is that it is invisible in a demo. A sales call built around "watch it answer this question" will naturally showcase whichever retrieval mode the platform is strong at. Nobody demos the failure case, because the failure case requires the buyer to already know the correct answer well enough to notice it's wrong. A demo audience that doesn't already know the interchange fee for that specific transaction has no way of knowing the platform silently dropped the merchant-category exception.
This is compounded by the fact that both failure modes look identical from the outside. A platform that ignores the document produces a confident, well-formatted, wrong number. A platform that ignores the query produces a confident, well-formatted, wrong number. Neither throws an error. Neither flags uncertainty. The output format is the same whether the underlying reasoning was complete or half-missing, which is exactly why this gap can sit undetected in a production system for months, surfacing only when someone downstream, often in finance or compliance, notices a number that doesn't reconcile.
The DABstep benchmark, built from over 450 tasks drawn from a real financial analytics platform, was specifically constructed to surface this gap rather than let it hide. Its hard split, which makes up roughly 84% of the benchmark, requires exactly the kind of query-plus-document reasoning described above. The best baseline model scores 76% on questions needing only one retrieval mode. On the hard split, that same model drops to under 15%. The gap between those two numbers is not a rounding error in model capability. It is the size of the architectural problem that most platforms have not solved, measured directly rather than argued about.
Why Buyers Default to the Wrong Evaluation Criteria
Most procurement processes for AI data platforms are built around criteria that were correct for the last generation of tools and are incomplete for this one: query accuracy on a known schema, latency, uptime, and integration effort. All of these matter, and none of them test whether the platform can reconcile a query result against a document rule. A platform can score well on every one of these and still fail the interchange-fee test on day one of production use.
This is not a criticism of procurement teams. It's a reasonable consequence of evaluating a new category of failure using an old category's checklist. The fix is not a longer checklist. It's one additional, specific test: bring a real question the evaluator already knows the answer to, one that spans both a table and a document, and watch what the platform actually does with it, not what the vendor says it does.
What an Actual Fix Looks Like
Solving this is not a matter of bolting a document search tool onto a SQL engine, or bolting a SQL connector onto a RAG pipeline. That produces two separate half-answers sitting next to each other, and pushes the work of reconciling them back onto the person who asked the question in the first place, which defeats the point of asking an AI system at all.
A real fix requires an orchestration layer that classifies each component of a question before retrieving anything: which parts point to structured, row-level data, and which parts point to a rule that lives in unstructured text. It routes each component to the retrieval mechanism built for it, then merges the results with the document's conditions correctly applied to the query's rows. This has to happen before the answer is assembled, not after, because reconciling two already-generated half-answers after the fact tends to just pick whichever one sounds more confident, rather than whichever one is correct.
This is a genuinely different engineering problem from either text-to-SQL or document retrieval in isolation, which is part of why so few platforms have solved it well. It requires the system to reason about the question itself before reasoning about the data, a step most architectures skip entirely because it wasn't necessary for the single-mode questions they were originally built to answer.
The Auditability Requirement This Creates
BFSI has a requirement that most other industries evaluating AI data platforms don't have to weigh as heavily: every number that ends up in a regulatory filing, a provisioning calculation, or a customer-facing dispute needs to be explainable after the fact, not just correct in the moment. A platform that reconciles a query and a document internally but can't show which part of the final answer came from which source has solved half the problem and left the more consequential half unsolved.
This changes what "correct" needs to mean in practice. It's not enough for an orchestration layer to produce the right number. It needs to produce the right number along with a visible trail: this figure came from these rows, filtered by this rule, drawn from this version of this document. When a compliance team is later asked to defend a classification decision or a fee calculation, that trail is the difference between a five-minute explanation and a multi-week reconstruction effort involving whoever originally ran the analysis, if they're still reachable at all.
This is also why heterogeneous retrieval and compliance-grade auditability tend to be solved by the same underlying architecture rather than as two separate features bolted together later. A system that classifies question components and routes them deliberately is, by construction, in a position to log which component produced which part of the answer. A system that blends everything into a single opaque generation step, by contrast, has to retrofit explainability after the fact, which tends to produce plausible-sounding justifications rather than accurate ones.
What This Means for Anyone Evaluating an AI Data Platform
The most useful thing a buyer can do before signing anything is bring a real question, one they already know the correct answer to, that requires both a table lookup and a document rule. Ask the vendor to answer it live, and ask them to show which part of the answer came from which source. A platform that can't do this on demand, only in a controlled demo, has not solved the problem this article describes.
That gap tends to surface at the worst possible moment: during a fee dispute, a provisioning calculation, or a regulatory audit, exactly the situations where an approximately correct answer is not good enough. In BFSI specifically, the cost of an unreconciled answer is rarely just an inconvenience. It's a misstated provision, an incorrectly classified asset, or a fee dispute with a card network that could have been avoided with a single correctly joined query.
The organizations that get this right going forward will not be the ones with the best individual retrieval components. They will be the ones whose systems know, for every question, which components matter and how to bring them together, correctly, every time, without waiting for someone downstream to catch the gap.