When a knowledge graph beats a vector index

"When would you use a knowledge graph instead of, or alongside, a vector index?"

What it is

A vector index stores each chunk of text as an embedding and retrieves by similarity to the query embedding. It answers "what text is semantically near this question?"

A knowledge graph stores entities as nodes and relationships as typed edges, and retrieves by traversal. It answers "what is connected to this entity, and how?"

The distinction that matters for retrieval:

	Vector index	Knowledge graph
Unit stored	A chunk of text	An entity and a typed relationship
Query	Nearest neighbours of an embedding	A traversal from a seed node
Good at	"What does the doc say about X?"	"How is X connected to Y?"
Bad at	Anything requiring two hops	Anything requiring paraphrase matching
Build cost	Embed once, cheap and mechanical	Extract entities and relations, expensive and error-prone
Failure mode	Retrieves plausible but unrelated text	Misses what was never extracted

Commonly confused with "graph database versus vector database", which is a storage question. The real question is whether the answer requires composing facts that appear in different places, and that is a property of your questions, not of your infrastructure.

Also commonly confused with GraphRAG specifically. GraphRAG is Microsoft's particular implementation (LLM extraction, community detection, hierarchical summaries). It is one design point, not the category.

The problem it solves

Vector retrieval fails on a specific, identifiable class of question, and it fails confidently, which is worse than failing loudly.

Multi-hop questions. "Which of our suppliers are affected by the port closure in Ningbo?" The answer requires: port closure affects region → which suppliers are in that region → which of those supply us. No single chunk contains that chain. Vector search retrieves chunks about port closures and chunks about suppliers, and the model writes something plausible from the pieces.

Aggregate and global questions. "What are the main themes across these 400 incident reports?" Top-k retrieval returns k chunks. The answer requires all 400. No value of k fixes this, because the question is about the corpus rather than about a passage in it.

Questions about relationships rather than content. "Which services would be affected if we deprecate the auth-v1 endpoint?" The dependency structure is the answer; the prose describing each service is not.

Explainability requirements. A regulator asking why the system produced an answer gets a traversal path from a graph and a similarity score from a vector index. One is auditable and one is not.

The counter-case is equally important: for "what is our refund policy for digital goods?" a vector index is better, cheaper and more robust, because the answer lives in one passage and the hard part is matching a paraphrase.

Mechanics

What a graph retrieval actually looks like

// "Which suppliers are affected by the Ningbo port closure?"
MATCH (e:Event {name: 'Ningbo port closure'})-[:AFFECTS]->(r:Region)
MATCH (s:Supplier)-[:LOCATED_IN]->(r)
MATCH (s)-[:SUPPLIES]->(c:Component)<-[:USES]-(p:Product)
RETURN DISTINCT s.name, collect(DISTINCT p.name) AS products_at_risk

Three hops, an exact answer, and a path you can show someone. The equivalent vector query returns chunks that mention Ningbo and chunks that mention suppliers, and the join happens inside the language model's head, where it is unverifiable.

Building the graph: the part that is genuinely hard

For structured sources (a service catalogue, a CMDB, an org chart, a product taxonomy), the graph already exists and you are just loading it. That is the easy and usually correct case, and it is the one people skip past.

For unstructured text, you extract:

EXTRACTION_PROMPT = """Extract entities and relationships from the text.

Entity types: Person, Organization, Product, Component, Region, Event
Relationship types: SUPPLIES, LOCATED_IN, USES, AFFECTS, REPORTS_TO, ACQUIRED

Return JSON:
{"entities": [{"name": ..., "type": ..., "description": ...}],
 "relations": [{"source": ..., "target": ..., "type": ..., "evidence": ...}]}

Include only relationships stated or directly implied in the text.
Use the exact surface form of the entity name as it appears.

Text:
{chunk}
"""

Three problems appear immediately, and naming them is what shows you have built one:

Entity resolution. "Acme Corp", "Acme Corporation", "ACME" and "Acme" must become one node. Without resolution the graph fragments and traversals return nothing. Embedding-based clustering plus a normalisation pass handles the bulk; the tail needs rules or human review.

Extraction cost. One LLM call per chunk. A 100,000-chunk corpus is 100,000 calls, and at typical prices that is thousands of dollars before you have answered a single question. It is also the cost you pay again when the schema changes.

Schema drift. Ask an LLM for relationship types without constraining them and you get SUPPLIES, PROVIDES, DELIVERS_TO and IS_SUPPLIER_OF as four distinct edge types describing one relationship. Constrain the type vocabulary in the prompt, and validate against it.

GraphRAG's specific contribution: community summarisation

Microsoft's GraphRAG addresses the global question, and the mechanism is worth knowing precisely because it is the part that has no vector-index equivalent.

1. Extract entities and relations from every chunk        (expensive, one-time)
2. Build the graph
3. Detect communities with the Leiden algorithm, hierarchically:
   level 0 = fine-grained clusters, level 2 = broad themes
4. For each community at each level, generate a summary with an LLM
5. Query time:
   LOCAL search  -> seed from entities in the question, traverse,
                    retrieve connected entities + their source chunks
   GLOBAL search -> map over community summaries at the chosen level,
                    then reduce to a single answer

Global search is the capability that does not exist in a vector index. "What are the main themes in this corpus?" is answered by mapping over pre-computed community summaries, not by retrieving top-k chunks. The cost is that the summaries are computed in advance, so the corpus must be reasonably static.

Microsoft's own reported figures, from their GraphRAG paper and blog: on multi-hop and global sensemaking questions, human and LLM judges preferred GraphRAG's answers on comprehensiveness and diversity by a substantial margin over naive vector RAG. They also report the indexing cost as the main barrier, which is the honest framing and the one to repeat.

The hybrid, which is what you actually build

Almost nobody runs a graph alone. The production shape is:

Query
  |
  +-- 1. Vector search over chunks           -> candidate passages
  +-- 2. Entity linking on the query          -> seed nodes
  |
  v
  3. Graph traversal from seed nodes (1-2 hops)
       -> connected entities, and the chunks they came from
  |
  v
  4. Merge and rerank: vector hits + graph-expanded hits
  |
  v
  5. Generate, citing both passages and traversal paths

Vector search finds the semantically relevant starting material; the graph supplies the structure the embeddings threw away. Neo4j's vector index, and the equivalents in most graph databases, let you store both in one system, which removes a consistency problem you would otherwise own.

The pragmatic middle ground worth naming: you often do not need a graph database at all. Extracting entities into a relational table and adding a metadata filter to your vector search covers a large fraction of "graph" use cases at a tiny fraction of the cost. If the questions are one hop ("documents about supplier X"), that is a WHERE clause, not a traversal.

A worked example

An internal support assistant over 40,000 documents: runbooks, incident reports, service documentation, architecture decision records. Vector RAG is deployed and answers 78 percent of questions acceptably. The failures cluster.

The failing questions, categorised:

Category A (61% of failures): multi-hop dependency
  "If we take the ranking service down, what breaks?"
  "Which teams are affected by the auth-v1 deprecation?"

Category B (24%): aggregate / global
  "What are the recurring causes across our Q3 incidents?"
  "Which services have never had a load test?"

Category C (15%): genuine retrieval misses
  Chunking and reranking problems. Not a graph problem.

Category C first, because it is cheaper: fix chunking and add a reranker. That is the discipline, and skipping it to build a graph is the classic error.

For A and B, the crucial observation: most of this graph already exists. The service dependency data is in the service catalogue and in the distributed tracing system. Team ownership is in the repo metadata. Incident-to-service mapping is in the incident tool. None of it requires LLM extraction.

Nodes from existing systems (no extraction):
  Service    (2,400)   <- service catalogue
  Team       (180)     <- org directory
  Incident   (3,100)   <- incident tool
  Runbook    (900)     <- docs repo front-matter

Edges from existing systems:
  Service -[:DEPENDS_ON]-> Service   <- distributed tracing, last 30 days
  Team    -[:OWNS]-> Service         <- catalogue
  Incident-[:AFFECTED]-> Service     <- incident tool
  Runbook -[:COVERS]-> Service       <- front-matter

Edges requiring extraction (the expensive 5%):
  Incident -[:CAUSED_BY]-> Cause     <- LLM over postmortem text

Now the earlier question is a traversal:

MATCH (s:Service {name: 'ranking'})<-[:DEPENDS_ON*1..3]-(dependent:Service)
MATCH (t:Team)-[:OWNS]->(dependent)
RETURN dependent.name, t.name, length(path) AS hops
ORDER BY hops

And the global question uses the extracted causes:

MATCH (i:Incident)-[:CAUSED_BY]->(c:Cause)
WHERE i.date >= date('2024-07-01') AND i.date < date('2024-10-01')
RETURN c.category, count(*) AS n ORDER BY n DESC

Costs, computed:

Extraction over 3,100 postmortems only (not 40,000 docs):
  3,100 x ~4k tokens in, ~600 out
  ~12.4M input + 1.9M output tokens -> roughly $100 one-time at current
  mid-tier pricing. Recomputed monthly for new incidents: ~$5/month.

If we had extracted over all 40,000 documents instead:
  ~160M input tokens -> roughly 13x the cost, for edges that the
  service catalogue already contained, more accurately.

Query-time cost: graph traversal is single-digit milliseconds and
free of LLM calls. Only the final generation costs tokens.

Result: answerable rate on category A questions goes from near zero to high, because the questions are now exact queries rather than retrieval gambles; and the answer includes the traversal path, so a reader can check it. Category B becomes possible at all.

The lesson to state out loud: the expensive part of a knowledge graph is extraction, and the highest-leverage move is noticing how much of the graph you already have in structured systems. Building it all with an LLM because "GraphRAG uses an LLM" is paying thirteen times as much for worse edges.

Production evidence

Microsoft GraphRAG (Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization", 2024) is the reference work. It reports that on global sensemaking questions over a corpus, graph-based community summarisation was preferred over naive RAG on comprehensiveness and diversity, and it is explicit that indexing cost is the principal barrier. The code is open source.

Neo4j ships a native vector index alongside the graph, and its GraphRAG documentation describes the hybrid pattern (vector for entry points, traversal for expansion) as the recommended production shape rather than graph-only retrieval.

LinkedIn's customer service application of graph-based RAG, reported in "Retrieval Augmented Generation with Knowledge Graphs for Customer Service Question Answering" (2024), structures historical tickets as a graph and reports a reduction in median per-issue resolution time. It is a good example because the graph structure came from the ticket system rather than from extraction.

Amazon Neptune Analytics and Google's Vertex AI both added graph-plus-vector retrieval paths, which is weak evidence about efficacy but strong evidence that the hybrid shape is the one the market converged on.

The debate

The case for graphs: multi-hop and global questions are not answerable by top-k retrieval at any k, the traversal path is auditable in a way a similarity score is not, and for domains that are inherently relational (dependencies, supply chains, org structures, financial ownership) the graph is the data model rather than an overlay.

The case against: extraction is expensive, brittle and needs maintaining; entity resolution is a real engineering problem that people underestimate; the graph is stale the moment the corpus changes; and a very large fraction of questions that sound multi-hop are answered fine by a good chunking strategy, a reranker, and a metadata filter. Teams routinely build a graph to avoid fixing their chunking.

My position: do not build a knowledge graph until you have categorised your retrieval failures and shown that multi-hop or global questions are a large share of them. When they are, build the graph from structured sources first, because most of it already exists in your service catalogue, ticket system, org directory or product taxonomy, and those edges are both free and more accurate than extracted ones. Use LLM extraction only for the relationships that exist solely in prose, and constrain the type vocabulary when you do.

And run it as a hybrid, not a replacement: vector search for entry points and for the single-passage questions that are the majority, graph traversal for expansion and for the questions embeddings cannot answer. A graph-only retrieval system is worse than plain RAG at the most common question type, which is "what does the document say about X".

The one case where I would build the graph first, before measuring: when explainability is a hard requirement. If a regulator or an auditor needs to see why the system said what it said, "cosine similarity was 0.83" is not an answer and no amount of retrieval tuning makes it one.

Follow-up Q&A

"When does a knowledge graph beat a vector index?" Four cases. Multi-hop questions, where the answer requires composing facts from different documents and no single chunk contains the chain. Global questions about the corpus rather than a passage, where no value of k works because the answer needs everything. Questions where the relationship structure is the answer, like dependency impact analysis. And anywhere the reasoning path has to be auditable. For the most common case, "what does the doc say about X", a vector index is better and cheaper, so the graph is an addition rather than a replacement.

"How expensive is building one, really?" For unstructured text, one LLM call per chunk, so a 100,000-chunk corpus is 100,000 calls before you answer a single question, and you pay it again when the schema changes. That is the main reason projects stall. The move that changes the arithmetic is recognising how much of the graph already exists in structured systems: service catalogues, tracing data, org directories, ticket systems, product taxonomies. Extract with an LLM only for relationships that exist purely in prose. In the case I described, that was 3,100 postmortems instead of 40,000 documents, roughly a thirteenth of the cost, and the structured edges were more accurate than extracted ones would have been.

"What breaks in practice?" Entity resolution, first: "Acme Corp", "Acme Corporation" and "ACME" become three nodes and traversals return nothing. Then schema drift, where an unconstrained extraction prompt produces SUPPLIES, PROVIDES and IS_SUPPLIER_OF as three edge types for one relationship, so you constrain the vocabulary in the prompt and validate against it. Then staleness, because the graph is a point-in-time projection and the corpus keeps changing. And the quiet one: recall failures are invisible, because a traversal that finds nothing looks identical to a traversal over a relationship that was never extracted.

"What is GraphRAG's global search actually doing?" It detects communities in the graph with the Leiden algorithm, hierarchically, then generates an LLM summary of each community at each level, in advance. At query time a global question maps over those community summaries and reduces to an answer, rather than retrieving chunks. That is the capability with no vector-index equivalent, because "what are the themes across this corpus" is a question about all of it. The trade is that the summaries are precomputed, so the corpus needs to be reasonably static or you are re-summarising constantly.

"How do you combine the two at query time?" Vector search over chunks for candidate passages, entity linking on the query to get seed nodes, one or two hops of traversal from those seeds to pull in connected entities and their source chunks, then merge and rerank both sets before generation. The graph supplies the structure the embeddings discarded; the vectors supply the paraphrase matching the graph cannot do. And keeping both in one system, as Neo4j's vector index allows, removes a consistency problem you would otherwise have to solve yourself.

"Is there a cheaper thing to try first?" Usually yes, and I would try it. Extract entities into a relational table and add metadata filtering to the vector search. If the questions are one hop, "documents about supplier X", that is a WHERE clause rather than a traversal, and it costs a fraction of a graph build. Also fix chunking and add a reranker first, because a meaningful share of what looks like multi-hop failure is ordinary retrieval failure wearing a costume.

Common misconceptions

"A knowledge graph replaces the vector index." It complements it. Graph-only retrieval is worse than plain RAG on the most common question type.

"GraphRAG means LLM extraction." GraphRAG is one implementation. If your graph comes from a service catalogue, you have a knowledge graph with no extraction cost and better edges.

"More hops is better." Traversal depth past two or three hops returns most of the graph and the precision collapses. Bound it.

"The graph is a one-time build." It is a pipeline with the same staleness problems as any derived store, and the maintenance is the part teams do not budget for.

"Multi-hop questions need a graph." Some do. Many are solved by better chunking, a reranker, or query decomposition into two retrieval calls, all of which are far cheaper. Categorise the failures before building.

Interview delivery note

Answer the "when" with the four cases, crisply: "Four cases. Multi-hop, where the answer needs facts from different documents and no chunk has the chain. Global questions about the corpus rather than a passage, where no value of k works. Relationship questions like dependency impact, where the structure is the answer. And anywhere the reasoning path has to be auditable, because 'cosine similarity was 0.83' isn't an explanation. For 'what does the doc say about X', which is most questions, the vector index is better and cheaper."

Then the cost framing, which is the staff signal: "the expensive part is extraction, one LLM call per chunk, and that's what stalls these projects. So the first thing I'd check is how much of the graph already exists in structured systems: service catalogue, tracing data, org directory, ticket system. In one case that was the difference between extracting over 3,100 postmortems and over 40,000 documents, about a thirteenth of the cost, and the structured edges were more accurate than extracted ones."

Close with the discipline, because it distinguishes you from someone who has read the GraphRAG blog post: "and I wouldn't build one until I'd categorised the retrieval failures. A good share of what looks multi-hop is ordinary retrieval failure that a reranker and better chunking fix for a fraction of the cost."

The Staff & Team Lead Interview Reference