Vector search retrieves what is similar. Enterprise questions ask what is connected. A long-form look at why the dominant RAG stack fails on relational, temporal, and evidentiary structure - and what the open research problems look like if we take that failure seriously.
Almost every enterprise AI system deployed in the last three years shares one architecture. Documents are split into chunks, chunks are embedded into a vector space, a question is embedded into the same space, the nearest neighbours are pasted into a prompt, and a language model writes an answer. The architecture is so standard that it has stopped being described as a design decision. It is simply what “doing RAG” means.
This post argues that the architecture encodes a specific and mostly unexamined assumption - that the information needed to answer a question is local to a contiguous span of text - and that this assumption is false for the majority of questions that enterprises actually care about. It is false in a way that no amount of better embeddings, larger context windows, or more aggressive reranking repairs, because the failure is representational rather than statistical.
The counter-proposal is not “use a graph database.” It is that the object we should be retrieving over is a typed, temporal, evidence-bearing graph induced from the corpus, and that once that object exists, the interesting research questions change character entirely: from “how do we rank passages” to “how do we reason under uncertainty over an incomplete, contradictory, continuously-changing symbolic structure whose nodes are grounded in pixels.”
I work on this in a regulated setting - pharmaceutical manufacturing, GxP documentation, regulatory submissions - where an unsupported claim is not an embarrassing demo, it is a deviation. That setting makes the failure modes unusually legible, so I will use it for examples. But nothing here is domain-specific. The same structure exists in legal discovery, clinical evidence synthesis, semiconductor process engineering, financial disclosure, and aerospace certification.
Start with the standard formulation. Lewis et al. (2020) define retrieval-augmented generation as a latent-variable model over retrieved documents:
\[p(y \mid x) \;=\; \sum_{z \in \mathcal{Z}_k(x)} p_\eta(z \mid x)\, p_\theta(y \mid x, z)\]where \(\mathcal{Z}_k(x)\) is the top-\(k\) set returned by a retriever, typically a dual encoder scoring by inner product (DPR, REALM):
\[s(x, z) \;=\; \langle E_q(x),\, E_d(z) \rangle, \qquad \mathcal{Z}_k(x) = \operatorname*{arg\,top-}k_{z \in \mathcal{C}} \; s(x,z)\]Two properties of this objective are worth staring at, because they are where the trouble lives.
Property 1: the scoring function is separable over passages. Each candidate \(z\) is scored in isolation against \(x\). There is no term of the form \(s(z_i, z_j \mid x)\) - nothing that rewards a set of passages for being jointly sufficient. In practice, “fusion-in-decoder” architectures (Izacard & Grave, 2020) let the reader attend across passages, which patches the generation side, but the selection side remains a \(k\)-way independent argmax. A retriever cannot prefer a pair of mediocre passages that together entail the answer over two excellent passages that redundantly say the same thing.
Property 2: relevance is estimated from the question alone. The query embedding \(E_q(x)\) must, in a single vector, encode everything needed to locate every piece of evidence. If part of the evidence is only findable after you know a fact you do not yet have, the retriever is structurally blind to it.
Property 2 is the multi-hop problem, and it is worth being precise about how badly it degrades. Suppose a question requires \(h\) hops, and suppose - generously - that each hop is retrieved independently with recall \(r\). End-to-end recall is \(r^h\).
{
"type": "line",
"data": {
"labels": ["1", "2", "3", "4", "5", "6"],
"datasets": [
{
"label": "per-hop recall r = 0.90",
"data": [0.9, 0.81, 0.729, 0.656, 0.59, 0.531],
"borderColor": "#86b6ef",
"backgroundColor": "#86b6ef",
"borderWidth": 2,
"pointRadius": 4,
"pointHoverRadius": 6,
"tension": 0.25,
"fill": false
},
{
"label": "per-hop recall r = 0.80",
"data": [0.8, 0.64, 0.512, 0.41, 0.328, 0.262],
"borderColor": "#3987e5",
"backgroundColor": "#3987e5",
"borderWidth": 2,
"pointRadius": 4,
"pointHoverRadius": 6,
"tension": 0.25,
"fill": false
},
{
"label": "per-hop recall r = 0.75",
"data": [0.75, 0.563, 0.422, 0.316, 0.237, 0.178],
"borderColor": "#184f95",
"backgroundColor": "#184f95",
"borderWidth": 2,
"pointRadius": 4,
"pointHoverRadius": 6,
"tension": 0.25,
"fill": false
}
]
},
"options": {
"responsive": true,
"maintainAspectRatio": true,
"aspectRatio": 2.1,
"interaction": { "mode": "index", "intersect": false },
"plugins": {
"legend": {
"position": "bottom",
"labels": { "usePointStyle": true, "pointStyle": "line", "boxWidth": 28, "color": "#7a7a7a" }
},
"tooltip": { "displayColors": true }
},
"scales": {
"y": {
"min": 0,
"max": 1,
"title": { "display": true, "text": "end-to-end evidence recall", "color": "#7a7a7a" },
"grid": { "color": "rgba(128,128,128,0.16)" },
"ticks": { "color": "#7a7a7a" }
},
"x": {
"title": { "display": true, "text": "hops required to assemble the answer", "color": "#7a7a7a" },
"grid": { "display": false },
"ticks": { "color": "#7a7a7a" }
}
}
}
}
Figure 1. Compounding recall loss under independent hops, $$r^h$$. At an unrealistically strong per-hop recall of 0.90, a four-hop question retrieves its full evidence set only 66% of the time; at a realistic 0.75 it is 32%. And this model is optimistic, because hop $$i{+}1$$'s query terms are usually not present in the original question - they are the answer to hop $$i$$. The retriever is not merely lossy; it is querying for the wrong thing.
Press et al. (2022) named the empirical consequence the compositionality gap: models answer the sub-questions correctly at high rates but fail to compose them, and the gap does not close with scale. Benchmarks built specifically to require composition - HotpotQA, MuSiQue - show the same shape.
Here is a question of the sort my systems get asked daily:
“Which validated batches were produced on equipment that was, at the time of production, operating under a temporary engineering change that has since been rolled back?”
Decompose what answering it requires:
rolled_back terminal state.validated at the time, not now.No chunk in the corpus contains this answer. No chunk contains even two of the five required facts. The evidence is spread across three document families that share no vocabulary, joined by identifiers that appear in different notational conventions, and gated by temporal predicates that are nowhere written down as text. A vector retriever asked this question will return the most topically similar passages - probably the engineering change procedure SOP, which describes how rollbacks are documented and contains not a single one of the required facts.
The model will then answer. It will produce a fluent, plausible, well-cited-looking list of batches. This is the part that should worry us: retrieval failure and retrieval success produce indistinguishable-looking outputs. The generator has no channel through which to learn that the join it needed was never performed.
Figure 2. The two pipelines differ in one structurally decisive way: path B can fail loudly. A top-$$k$$ retriever always returns $$k$$ results and has no score at which it says "nothing here satisfies your constraints." A constraint-based retriever can report which constraint went unsatisfied - which is the difference between a tool an auditor will accept and one they will not.
THE CORE CLAIM
Vector search optimises a similarity objective. Most high-value enterprise questions are join queries with temporal and provenance predicates. These are different computational problems, and no improvement to the first solves the second.
The obvious rebuttal is that long-context models make retrieval quality moot: put the whole corpus in the prompt. Three things push back.
First, positional degradation is real and persistent. Liu et al. (2023) documented the U-shaped accuracy curve over context position, and while frontier models have improved, synthetic stress tests like RULER show that effective context length trails advertised context length substantially. Single-needle retrieval is close to solved. Multi-needle aggregation with variable tracing is not.
Second, the corpus does not fit and never will. A mid-size pharmaceutical manufacturer’s controlled document set is tens of gigabytes of PDF, growing daily, with versioned history that must remain queryable. The relevant comparison is not “128k tokens vs. 8k” but “128k tokens vs. \(10^{10}\) tokens.”
Third - and this is the deeper point - putting text in a context window does not perform a join. If the model must scan 200 batch records against 40 change records under a temporal predicate, it is being asked to execute a nested loop in the forward pass, with no working memory, no ability to backtrack, and attention that must simultaneously serve as index, comparator, and accumulator. The empirical comparisons (Li et al., 2024) find long context competitive on lookup, but lookup was never the hard part.
Long context changes the constant factor on retrieval. It does not change the complexity class of relational reasoning.
It is worth making the distinction sharp, because “semantic search understands meaning” is an easy thing to nod at.
An embedding model computes a map \(E: \text{text} \to \mathbb{R}^d\) trained so that distance approximates topical or paraphrastic relatedness. What that objective provably does not preserve:
Direction. “Reagent A inhibits Enzyme B” and “Enzyme B inhibits Reagent A” are near-identical under any bi-encoder. Cosine similarity is symmetric; causation, containment, supersession, and authorisation are not. Knowledge graph embeddings exist precisely because this is hard: TransE models a relation as a translation \(\mathbf{h} + \mathbf{r} \approx \mathbf{t}\), ComplEx uses complex-valued bilinear forms to represent antisymmetry, RotatE as rotation in complex space. Every one of these is an architectural commitment to representing asymmetry, made because a symmetric dot product cannot.
Negation and scope. “Procedure P applies to Line 3” and “Procedure P does not apply to Line 3” have cosine similarity above 0.95 in most production embedding models. In a compliance context these are opposite answers.
Identity across surface forms. EQ-4471, Equipment 4471, “the Bosch filler on Line 3”, and “the unit referenced in Annex C” may denote one machine. Embeddings place the first two nearby and the last two nowhere near either. Entity resolution is a distinct problem with its own literature, and it does not fall out of semantic similarity.
Counting, aggregation, and negative results. “Which SOPs have no linked training records?” cannot be answered by retrieval at all. There is no passage to retrieve. The answer is a property of the absence of edges - a closed-world question posed to an open-world system.
Chunking is usually presented as an engineering detail - a window size and an overlap. It is better understood as a destructive projection applied before any learning happens.
Take a document as a structured object \(D\) with a hierarchy of sections, a set of internal references, tables with row/column semantics, and figures with captions bound to body text. Chunking applies \(\pi: D \to \{c_1, \ldots, c_n\}\), a partition into flat strings. Ask what \(\pi\) destroys.
Figure 3. Every labelled edge on the left - exception_to, refers_to, caption_binds, and the header-to-cell binding inside the table - is absent on the right. Chunk 18 now reads as an unconditional statement; chunk 41 is a row severed from its column headers; chunk 63 is a dangling pointer. The projection is applied at index time, before any query is known.
Hierarchical scope. A clause reading “This requirement does not apply to products manufactured under Annex 1” inherits its scope from the section heading three levels up. Chunk it, and the clause becomes an unconditional statement. Prepending breadcrumbs helps and is now common practice; it recovers ancestry but not sibling structure, and not the fact that §4.2.1 is an exception to §4.2 rather than an elaboration of it.
Tables. A table is a relational object. Row 14 means what it means because of the column headers, the units row, and a footnote marker attached to one cell. Linearising it to text and slicing at token 512 severs the cell from its header. The document-understanding community has treated table structure recognition as its own hard problem for a decade - PubTables-1M exists because getting a table’s logical structure out of a PDF is not solved by OCR.
References. “See Section 7.3” is an edge. As a chunk it is a dangling pointer - the retriever will happily return the chunk containing the pointer and not the chunk it points to.
There is a clean information-theoretic way to say this. Let \(A\) be the answer to a query \(q\) and \(Z\) the retrieved set. The generator’s achievable accuracy is bounded by \(I(A; Z \mid q)\). Chunking is a fixed, query-independent map applied before \(q\) is known. If the mutual information between the answer and any contiguous span is low - which is exactly the case when the answer is a join - then no retriever over that chunk set can recover it, regardless of embedding quality. The information was destroyed at index time, and retrieval is a data-processing step that cannot create it.
This is why the ongoing refinement of chunking strategies - semantic chunking, recursive chunking, late chunking, hierarchical summarisation as in RAPTOR - yields real but bounded gains. They are better projections. They are still projections. RAPTOR is instructive precisely because it is the strongest version of the idea: build a tree of recursive summaries so that retrieval can operate at multiple granularities. It works. And its structure is a tree, which tells you the authors ran into the same wall and reached for topology as the fix. Enterprise document structure is not a tree; it is a multigraph with cycles, cross-document edges, and time.
The most consequential misconception in enterprise document AI is that a PDF is a container for text. A PDF is a serialisation of a graph onto a two-dimensional surface, and the serialisation is lossy in a specific way: it renders edges as typography.
Consider what a regulatory submission or a manufacturing batch record actually contains:
Layout is semantics. Indentation encodes subordination. A box encodes grouping. Bold encodes definitional status. Column position in a form encodes the field-value relation. This is why document-understanding models became multimodal: LayoutLM added 2D positional embeddings because reading order alone loses the information; LayoutLMv3 unified text and image masking; Donut went OCR-free end-to-end; Nougat targeted scientific documents specifically because equations and tables defeat text extraction. DocVQA exists as a benchmark because answering questions over a document image is a different task from answering them over the document’s extracted text.
Text extraction, then, is not a preprocessing step. It is a graph-destroying operation that we perform first and then spend the rest of the pipeline trying to invert.
Figure 4. Every edge here is present in the source PDFs - as a string, a glyph, a position on a page, or a line in a signature block. All of it survives rendering to a human reader. Almost none of it survives pdftotext. Note that the opening question of this post is a four-edge path through this diagram, and that no two adjacent nodes live in the same document.
The research problem this raises is not “better OCR.” It is: what is the right target representation for document understanding, and how do we supervise it? If the output of document parsing is a graph rather than a string, then the loss function, the annotation format, the evaluation metric, and the error taxonomy all need rethinking. We currently evaluate document AI mostly on span-extraction F1 and VQA accuracy, both of which are string-valued. We have no widely-adopted benchmark for structural fidelity of the recovered graph, which means we are not optimising for the thing that matters.
“Knowledge graph” is used loosely enough to be nearly content-free. The classic surveys (Hogan et al., 2021; Nickel et al., 2015) span everything from RDF triple stores to property graphs to embedding spaces. For enterprise reasoning under audit, the requirements are specific and mostly not satisfied by the Freebase/Wikidata-style triple store that the KG literature is built around.
Formally, take
\[\mathcal{G} = (V, E, \tau, \alpha, \sigma)\]with \(V\) typed nodes, \(E \subseteq V \times R \times V\) typed directed edges, \(\tau\) a temporal annotation, \(\alpha\) attributes, and \(\sigma\) a provenance/evidence map. The last two carry most of the weight.
Not just entities. Documents, sections, tables, figures, clauses, and assertions are nodes. A claim like “fill weight limits are 9.8–10.2 g” is a first-class node that can be cited, contradicted, versioned, and superseded - not a string inside a chunk. This matters because contradiction is a relation between assertions, and you cannot express it if assertions are not addressable.
Typed relations with a controlled vocabulary: governed_by, supersedes, derived_from, approved_by, deviates_from, applies_to, evidences, contradicts. The typing is what makes the reasoning tractable - a query planner can prune by relation type, and an LLM asked to traverse supersedes cannot silently substitute references.
This is the requirement most often skipped and most often fatal. Edges are not true; they are true over an interval. Write an edge as \(e = (h, r, t, [\,t_{\text{valid}}^{-}, t_{\text{valid}}^{+}), [\,t_{\text{trans}}^{-}, t_{\text{trans}}^{+}))\) - bitemporal, distinguishing when the fact was true in the world from when the system knew it. Regulatory questions are almost always asked in the past tense: not “which SOP governs this process” but “which SOP governed this process on 14 March 2025, and what did we believe on that date?” A non-temporal graph answers the first and silently answers the second wrongly.
The temporal KG literature - Know-Evolve, Temporal Graph Networks - has good machinery for the modelling side; the reasoning side over bitemporal enterprise graphs is comparatively untouched.
Documents are revised. SOP-QA-114 Rev 7 --supersedes--> Rev 6 is an edge, and a reference written in 2024 points at Rev 6 even though Rev 8 is current. Resolving references to current versions is the single most common silent corruption I see in production RAG systems: the retriever returns the latest revision because that is what was indexed, and the answer is confidently wrong about what a historical record meant.
Every edge must carry \(\sigma(e)\): which document, which revision, which page, which bounding box, which extractor, which confidence. This is not a logging nicety. It is what makes the graph auditable, and it is what turns citation from a post-hoc formatting step into a structural property. The attribution literature (Rashkin et al., 2021 on Attributable to Identified Sources; Bohnet et al., 2022; GopherCite) established that models asked to cite will cite plausibly rather than correctly.
If the reasoning substrate carries provenance intrinsically, the citation is a projection of the evidence path rather than a generated claim about it.
Figure 5. Note what temporal scoping buys. Two facts that a vector store would surface as conflicting evidence - leaving the model to pick one, usually whichever appeared first in the prompt (Xie et al., 2023 on knowledge conflicts) - are simply consecutive states of the same quantity. The contradiction was an artefact of dropping the interval.
None of this is free. Building \(\mathcal{G}\) from a corpus of PDFs requires entity recognition, entity resolution across surface forms and document families, relation extraction, temporal normalisation, and schema induction - every one of which is an open problem with a decades-long literature and none of which is solved to enterprise precision. Classical open information extraction (Angeli et al., 2015) produces untyped, unresolved triples at recall that is too low and precision that is too variable. End-to-end neural relation extraction (REBEL, Huguet Cabot & Navigli, 2021) is better but schema-bound.
LLM-based extraction (Zhu et al., 2023) is the current pragmatic answer and has its own pathology: the extractor hallucinates edges. We have moved the hallucination from inference time to index time, where it is more dangerous because it is now persistent, reused across queries, and wearing the costume of structured data. An LLM-extracted graph that nobody audits is a hallucination cache.
THE CENTRAL TENSION
Graphs give you verifiable reasoning, but constructing the graph requires exactly the unreliable extraction step whose unreliability motivated the graph. Any honest treatment has to model extraction confidence as a first-class quantity rather than pretending the graph is ground truth.
That tension is not a reason to abandon the approach. It is the reason edges need confidence scores, provenance, and abstention - and it is the most interesting open problem in the area.
Edge et al. (2024) put a name on the family. The conceptual move is small and consequential: instead of retrieving passages ranked by similarity, retrieve a subgraph selected by structure, then linearise it for the model.
The pipeline separates into three concerns that vanilla RAG conflates.
Anchoring. Map the query to entry points in the graph. This is where embeddings remain genuinely useful - semantic similarity is a good way to find which nodes to start from. Vector search does not disappear; it is demoted from “the retrieval system” to “the entity linker.”
Traversal. Expand from anchors along typed edges under constraints derived from the query: relation types, temporal predicates, hop budget, confidence floor. This is a query-planning problem, and it is where the relational structure does the work that similarity cannot.
Selection and linearisation. The expanded neighbourhood is usually far too large. Choose a subgraph that is sufficient for the query and small enough to fit in context, then serialise it - with provenance attached - in a form the model can reason over.
Figure 6. The two gates are the whole architectural argument. Sufficiency replaces top-$$k$$ as the stopping rule, so retrieval terminates when the constraint pattern is covered rather than when a counter runs out. Groundedness is checked mechanically against evidence edges rather than asked of the model. Both are cheap; neither is expressible over a bag of chunks.
Two things deserve emphasis.
The global/local distinction. Some questions are local (“what changed in this batch?”) and answered by a small neighbourhood. Others are global (“what are the recurring root-cause themes across three years of deviations?”) and are not answerable by any bounded retrieval - there is no small set of chunks that contains the answer. GraphRAG’s community-detection-plus-summarisation approach addresses this by precomputing hierarchical summaries over graph communities, so global questions are answered from structure computed at index time. This is a genuinely different capability, not a better version of an existing one; sensemaking over a corpus was simply outside vanilla RAG’s reach.
Traversal is not free. Naive \(k\)-hop expansion has neighbourhood size growing as \(O(\bar{d}^{\,k})\) in average degree. On a dense enterprise graph with hub nodes - a widely-referenced SOP may have \(10^4\) inbound edges - two hops swallows the corpus. So traversal must be learned or planned, not exhaustive. Variants in the literature take different positions: HippoRAG uses personalised PageRank over a schemaless graph as a neurobiologically-motivated single-step multi-hop mechanism; LightRAG uses dual-level keyword retrieval to bound cost; Think-on-Graph has the LLM itself perform beam search over the graph, interleaving traversal with reasoning. The survey by Peng et al. (2024) maps the space. That these approaches differ so fundamentally in where they place the traversal policy - precomputed, heuristic, or model-driven - is a good indicator that the question is open.
A sketch of the control loop, deliberately abstract:
\begin{algorithm}
\caption{Evidence-constrained graph retrieval}
\begin{algorithmic}
\PROCEDURE{GraphRetrieve}{$$q, \mathcal{G}, B$$}
\STATE $$\Phi \gets$$ \CALL{ParseConstraints}{$$q$$}
\COMMENT{relation types, time window $$W$$, confidence floor $$\epsilon$$}
\STATE $$A \gets$$ \CALL{LinkEntities}{$$q, \mathcal{G}$$}
\COMMENT{dense retrieval used only here}
\STATE $$F \gets A$$; $$\; S \gets \emptyset$$
\WHILE{$$F \neq \emptyset$$ \AND $$|S| < B$$}
\STATE $$v \gets$$ \CALL{PopBest}{$$F$$}
\COMMENT{score: relevance $$\times$$ path confidence $$\times$$ novelty}
\FOR{$$e = (v, r, u) \in \mathcal{G}$$ \textbf{ with } $$r \in \Phi.R$$}
\IF{$$\tau(e) \cap W \neq \emptyset$$ \AND $$\mathrm{conf}(e) \geq \epsilon$$}
\STATE $$S \gets S \cup \{e\}$$; \; \CALL{Push}{$$F, u$$}
\ENDIF
\ENDFOR
\IF{\CALL{Sufficient}{$$S, \Phi$$}}
\STATE \textbf{break}
\COMMENT{stop on sufficiency, not on $$k$$}
\ENDIF
\ENDWHILE
\IF{\NOT \CALL{Sufficient}{$$S, \Phi$$}}
\RETURN \CALL{Abstain}{$$\Phi \setminus \mathrm{covered}(S)$$}
\COMMENT{report the missing constraint}
\ENDIF
\RETURN \CALL{Linearise}{$$S, \sigma$$}
\ENDPROCEDURE
\end{algorithmic}
\end{algorithm}
The line that matters is the abstention branch. A similarity retriever always returns \(k\) results; there is no value of the score at which it says “nothing here satisfies your constraints.” A constraint-based retriever can report which constraint went unsatisfied - which is diagnostic information the system did not previously have.
The interesting question is not whether graph retrieval improves benchmark scores. It is what changes about the inference procedure.
The reasoning trace becomes checkable. When a model reasons over prose, its chain of thought (Wei et al., 2022) is natural language, and natural language reasoning steps are not verifiable - they are plausible-sounding text whose relation to the model’s actual computation is unclear. When the model reasons over a graph, each step is a traversal, and a traversal either exists in \(\mathcal{G}\) or it does not. The trace becomes a path, and paths can be validated by a non-neural checker. This is the practical content of the neurosymbolic argument (Garcez & Lamb, 2020): put the symbols where they can be checked, and let the network do the pattern-matching it is good at.
Composition moves out of the forward pass. ReAct established interleaved reasoning and acting; IRCoT and FLARE apply the pattern to retrieval specifically, re-querying as reasoning proceeds; Self-RAG adds learned reflection over whether retrieval is needed and whether output is supported. With a graph, the action space is not “issue another search string” but “traverse this typed edge from this node” - a discrete, finite, and verifiable action space.
Planning over it is a search problem with a well-defined transition function, which is a far better setting than planning over free-text queries. Given the evidence that LLMs are weak autonomous planners (Kambhampati et al., 2024), constraining the action space to legal graph operations and letting an external component verify each step is not a limitation - it is the design.
Attribution becomes structural. If every node carries \(\sigma\), the citation for a claim is the evidence set of the path that produced it. The model is not asked to remember where it read something; the provenance rides along with the fact. This inverts the usual failure mode where citations are generated post-hoc and are correct only when the model happens to remember correctly.
Negative and aggregate answers become expressible. “No SOP governs this process” and “seven batches match” are closed-world statements over the graph. They are not retrievable from any corpus of text.
Uncertainty becomes localisable. Where extraction confidence is low, the system can say which edge it is unsure of. Contrast with a text pipeline, where uncertainty is a scalar attached to a whole answer, if it exists at all.
The honest caveat: none of this happens automatically. An LLM handed a linearised subgraph in a prompt can still ignore it, over-generalise from it, or invent edges that were not present. Making structural fidelity a trained property rather than a prompted hope is, I think, one of the more important near-term problems.
This is the section I actually care about. Everything above is context for these.
Enterprise graphs are not built once. A new revision of an SOP arrives and invalidates an unknown number of downstream edges. Two questions follow, neither well-answered:
What is the correct update semantics? When a document is superseded, edges derived from it are not deleted - they are closed at a transaction time, because historical queries must still return the old state. This is bitemporal versioning applied to a derived, machine-extracted structure, and the interaction between “the world changed” and “our extractor changed its mind” is genuinely subtle. If a better model re-extracts and now disagrees with last year’s edge, which is the fact and which is the belief?
How do we propagate invalidation efficiently? Recomputing global structure - community summaries, embeddings, centrality - on every ingest is intractable at enterprise write rates. Incremental graph algorithms exist; incremental semantic structure over an LLM-derived graph mostly does not. Zep/Graphiti is one of the few systems to take temporally-aware incremental KG maintenance seriously as an architecture problem, and it is early work.
Extraction produces \(\hat{e}\) with confidence \(c \in [0,1]\), and those confidences are (a) badly calibrated and (b) not independent, since errors correlate by document, template, and extractor version. Path confidence is therefore not \(\prod_i c_i\), though that is what everyone computes.
The pre-neural literature had good formalisms for this: Markov Logic Networks (Richardson & Domingos, 2006) and probabilistic soft logic both do weighted first-order inference over uncertain facts. What is missing is the bridge: how does an LLM reason correctly over an uncertain symbolic structure? If we hand a model a subgraph annotated with confidences, does its output confidence track them? Every piece of evidence I have says no - models treat provided facts as categorical regardless of hedging. That is a concrete, testable research question with a clean experimental design and, as far as I can tell, no good answer.
Related and equally open: calibrated abstention. Given a partially-satisfied constraint pattern, when should the system answer partially, when should it ask a clarifying question, and when should it refuse? In regulated domains the cost asymmetry between a wrong answer and a refusal is enormous, and we have no principled decision-theoretic treatment of it.
If a node is a figure, its content is pixels. A control chart in a batch record contains the fact “the process drifted out of specification in week 3” - a fact expressible as an edge, present nowhere in the text.
Scene graph research (Visual Genome) and vision-language pretraining (CLIP) give partial tooling, but document figures are a different distribution from natural images: charts, schematics, P&IDs, equipment photographs, chromatograms, gel images. The open problems compound:
contradicts edge - and detecting it requires a shared representation in which the two are comparable. This is arguably the most valuable capability in the entire space and among the least studied.Enterprise graphs are open-world and permanently incomplete. Classical KG completion assumes a fixed relation vocabulary and predicts missing links. Real corpora introduce new relation types as business processes change - a new regulatory framework brings genuinely new predicates.
So: continual schema induction. How do we discover that a new relation type is needed, distinguish it from a surface variant of an existing one, and extend the schema without invalidating everything downstream? This connects to continual learning (EWC) and to ontology evolution, and it is largely unaddressed in the LLM-KG literature, which almost universally assumes a schema handed down at design time.
If graph traversal is the action space, retrieval becomes sequential decision-making. That reframes it usefully: the agent has a state (current subgraph), actions (typed expansions, joins, temporal filters, aggregations), a budget, and a termination condition.
Open questions:
Long-running agents need persistent memory. The lineage runs from Memory Networks through MemGPT’s OS-inspired paging to graph-structured agent memory. A graph is an attractive substrate - it supports associative recall, temporal scoping, and consolidation. The open problems mirror human memory research uncomfortably closely:
A relevant subgraph frequently exceeds any context window. So we need lossy compression of a symbolic structure with a task-dependent loss function - compress the graph while preserving its answers to a query distribution.
This has a nice theoretical shape and connects to a known GNN pathology: over-squashing (Alon & Yahav, 2020; Topping et al., 2021), where information from an exponentially-growing receptive field is crushed into fixed-width vectors, with bottlenecks predicted by graph curvature. Linearising a subgraph into a prompt is over-squashing by another name: an exponentially large structure compressed into a linear token budget. The question of which structure survives that compression, and whether we can choose it adaptively, is open in both settings.
Two domains where the graph is not an implementation choice but the actual object of study.
Scientific KGs. Claims, evidence, methods, reagents, and contradictions across a literature. The hard part is not extraction - it is that scientific claims are conditional (on population, dose, assay, statistical model) and that reproducibility failures make the graph internally inconsistent by nature. A KG that cannot represent “this holds in mice but not in humans” is not modelling science.
Regulatory KGs. Regulations form a graph of obligations, exemptions, cross-references, and jurisdictional scope, and they are defeasible: a general rule holds unless an exception applies, and exceptions have exceptions. Defeasible reasoning is classical AI territory (default logic, argumentation frameworks) that the current LLM stack has almost entirely ignored. It is precisely what compliance requires. Legal NLP resources like CUAD show how far span extraction gets you and where it stops.
There is also a hard question of temporal regulatory reasoning: was this action compliant with the rules in force at the time? Answering it requires a versioned regulatory graph and a versioned procedural graph, joined at a timestamp. I know of no system that does this properly, and it is a daily question in pharmaceutical manufacturing.
Quietly, the biggest problem. We measure RAG systems with answer-level metrics: exact match, F1, LLM-as-judge. None of these tell you why a system failed, and all are gameable by fluency. What is missing:
Without these we cannot tell whether a GraphRAG system works because of the graph or in spite of it. That is an uncomfortable position for a subfield to be in, and it is where I would spend effort first.
A few directions I find most promising, stated as claims I would be willing to defend.
Extraction and reasoning should be jointly optimised, not pipelined. Today: parse → extract → build → retrieve → generate, each stage trained separately, errors compounding. The extraction stage does not know which edges matter downstream. If downstream answer correctness could supervise upstream extraction - differentiably, or through RL over a discrete graph - the extractor would learn to spend precision where it counts. This is hard because the graph is discrete and the signal is distant, but the analogous problem was solved for retrieval (REALM, RAG) and the payoff is larger here.
Neurosymbolic inference deserves another look with LLMs in the loop. Differentiable theorem proving (Rocktäschel & Riedel, 2017), Neural LP, and query embedding methods like Query2Box all attacked “soft logical inference over incomplete KGs” before LLMs were good. The techniques were limited by weak natural-language front ends - which is exactly what LLMs now provide. The combination is under-explored: LLM as parser and hypothesis generator, symbolic engine as verifier and executor.
Verification should be a first-class architectural component. Not a guardrail bolted on, but a checker that consumes the reasoning trace and validates each step against the graph. For a claimed path this is cheap and exact. Systems that can verify cheaply can afford to generate speculatively, which is a strictly better operating point.
Structure should be trained, not prompted. Handing a linearised graph to a general-purpose model in a prompt and hoping it respects the structure is the current state of practice, and it is fragile. Post-training specifically on structured evidence with process supervision over traversal decisions - rewarding correct paths rather than only correct answers - is the obvious missing piece.
Documents should be parsed to graphs directly. Rather than PDF → text → entities → graph, train models that emit structure from pixels end-to-end. Donut and Nougat show OCR-free extraction to structured targets is viable; the target should become a graph rather than a markup string, with the layout signal preserved instead of reconstructed.
The name “retrieval-augmented generation” describes the mechanism honestly: we augment generation with retrieval. Retrieval is a lookup. The pipeline’s implicit theory is that the model’s deficit is access to facts, and that supplying the facts fixes it.
For a large class of questions - “what is our policy on X,” “summarise this document” - that theory is right, and RAG works well. But the questions enterprises most want answered are not lookups. They are joins across document families that share no vocabulary; temporal queries about states of the world at particular moments; aggregations over sets defined by structural properties; consistency checks between what a procedure requires and what a record shows; negative queries about what is absent; and causal traces through chains of events.
Figure 7. The progression is not "more retrieval." Each step adds an operation the previous one could not express - and the last step adds something rarer than accuracy: the ability of the system to know, and say, that it could not answer.
Every one of those question types is a computation over relations. Supplying more text does not perform the computation; it relocates it into the forward pass of a model with no working memory and no ability to backtrack. The deficit is not access to facts. It is the absence of a structure over which the computation can be expressed at all.
Reasoning-Augmented Generation is the position that the substrate should be a typed, temporal, evidence-bearing graph; that the retrieval step should be constrained traversal rather than similarity ranking; that intermediate reasoning should be a verifiable path rather than unverifiable prose; and that the system should be able to say this cannot be answered from the available evidence, and here is the specific constraint that failed.
None of the pieces are finished. Extraction is unreliable, uncertainty is uncalibrated, traversal policies are heuristic, multimodal grounding is early, and evaluation is inadequate to the point of being misleading. That is the argument for treating this as a research problem rather than an integration problem.
What makes me confident about the direction is the asymmetry in verification cost. In a text pipeline, checking an answer costs about as much as producing it - you must re-read the sources and re-do the reasoning. In a graph pipeline, checking a claimed path is a set of lookups. Cheap verification is the precondition for systems we can trust with consequential decisions, and it is not something you can add to an architecture that has thrown away its structure at index time.
The last decade of enterprise AI has been about getting models to see the right text. The next one is about giving them something to reason over.
Retrieval-augmented generation and retrieval
Graph-based retrieval
Knowledge graphs and graph representation learning
Temporal and dynamic graphs
Neurosymbolic and probabilistic reasoning
Reasoning, agents, and tool use
Long context
Document intelligence and multimodality
Attribution, hallucination, and evaluation
Extraction and continual learning