Vector RAG vs. Graph RAG: What Does the Research Say?

If you are building a question-answering system over your own documents, you will eventually face this choice: store text as embeddings and retrieve by similarity (vector RAG), or extract entities and relationships into a graph and retrieve by following links (Graph RAG). Blog posts tend to declare a winner. The research does not.

This article summarizes what three recent papers actually found about vector RAG vs Graph RAG, explains the trade-offs in plain terms, and then lets you test the core idea yourself with a small Python experiment that runs on a laptop in seconds. The experiment is deliberately simple, and I spell out exactly what it can and cannot tell you.

What you will learn:

• How vector RAG and Graph RAG differ, from indexing to retrieval
• Which question types favor each approach, according to controlled studies
• Why multi-hop questions break plain similarity search
• How to reproduce a small retrieval experiment with scikit-learn and networkx
• A decision guide for choosing between them

Prerequisites: basic Python, and a rough idea of what an LLM and an embedding are.

What Are Vector RAG and Graph RAG?

Retrieval-augmented generation (RAG) gives an LLM relevant outside information at question time instead of relying only on what it memorized. The first step is retrieval: finding the right pieces of your documents. The two approaches differ in how that retrieval index is built.

Vector RAG splits documents into chunks, converts each to an embedding (a list of numbers capturing meaning), and at query time returns the chunks closest to the question. Microsoft’s GraphRAG paper describes this conventional approach as retrieving records closest to the query in vector space, where closeness reflects semantic similarity.

Graph RAG adds a structured layer. An LLM or other extractor reads the chunks and builds a graph of entities (people, companies, places) and their relationships. Retrieval then uses that structure. The studies below use two main flavors:

• Knowledge-graph-based Graph RAG extracts triples (head, relation, tail) and retrieves from the graph.
• Community-based Graph RAG, introduced by Edge et al. (Microsoft), groups closely related entities into communities and pre-writes summaries of each, which are later combined to answer broad questions.

	Vector RAG	Graph RAG
What is indexed	Chunk embeddings	Entities, relations, and often community summaries
Retrieval idea	Nearest neighbours to the question	Traverse links or use pre-built summaries
Extra indexing work	Embedding only	Entity and relation extraction, often LLM-based
Natural strength	Specific facts in a passage	Connecting facts across passages; whole-corpus questions

What the Research Says

Graph RAG was built for “global” questions

Edge et al. argue that vector RAG fails on global questions about an entire corpus, such as “What are the main themes in the dataset?”, because that is really a summarization task, not a lookup. Their method builds an entity graph, then pre-generates summaries for communities of related entities. On two datasets of roughly one million tokens each, they report that Graph RAG improved the comprehensiveness and diversity of answers over a vector RAG baseline, judged by an LLM. They also note that for specific, precise facts a conventional vector approach is more likely to give a short, direct answer.

A controlled head-to-head: the approaches are complementary

Han et al., “RAG vs. GraphRAG: A Systematic Evaluation and Key Insights” is the most direct comparison I found. They run both approaches under a unified protocol (same LLMs, embeddings, and retrieval settings) on question answering and query-based summarization. Their conclusions:

• RAG tends to do better on single-hop questions and those needing detailed information, while GraphRAG is more effective on multi-hop questions.
• For summarization, RAG captures fine-grained details, while GraphRAG produces more diverse, multi-faceted summaries.
• Results depend strongly on the design of the GraphRAG system, for example local versus global search.
• GraphRAG systems have a much higher upfront construction cost.
• Combining them helps. The authors propose a Selection strategy (route each query to RAG or GraphRAG) and an Integration strategy (merge both retrieval results), and report consistent improvements.

They also found strong position effects in LLM-as-a-judge summarization evaluation, which is a reminder to treat win rates from LLM judges with caution.

GraphRAG-Bench: when do graphs actually help?

Xiang et al., “When to use Graphs in RAG” (ICLR 2026) start from an uncomfortable observation: GraphRAG frequently underperforms vanilla RAG on many real-world tasks, despite its conceptual promise. They build GraphRAG-Bench, with tasks of increasing difficulty (fact retrieval, complex reasoning, contextual summarization, creative generation) and corpora of different information density, to find out where graph structure measurably helps. The paper also cites earlier studies reporting lower GraphRAG accuracy on simple lookup questions and higher latency, which is consistent with the Han et al. findings above.

What the three papers agree on

Question type	Better fit, according to the studies
Specific fact in one passage	Vector RAG (often as good or better, and cheaper)
Multi-hop reasoning across documents	Graph-based retrieval tends to help
Broad “what are the themes?” questions	Community-based Graph RAG
Mixed workloads	A hybrid or routing strategy

Caveats. These are benchmark results on particular datasets, LLMs, and graph-construction pipelines. They do not guarantee the same outcome on your data, and the field changes quickly. Treat them as a map of where to look first, not a verdict.

Why Multi-Hop Questions Break Similarity Search

Consider this question: “In which city is the company founded by Mara Quillon headquartered?” The answer needs two facts stored in two different chunks.

Chunk A shares words with the question, so similarity search finds it. Chunk B, which holds the actual answer, shares almost nothing with the question except “headquartered”, and your corpus likely has many chunks like that. The connection exists only through the bridge entity, Veltrix Labs, which the question never mentions. A graph can follow that link. Plain top-k similarity cannot, no matter how good the embedding model is, because the question itself does not contain the bridge.

This is the mechanism the multi-hop results in the papers point to. The experiment below isolates it.

A Small Reproducible Experiment

What it tests

The script builds a synthetic corpus of 244 short chunks about 60 invented companies (who founded each, where each is headquartered, what each builds, which country each city is in), plus distractor chunks. It asks three kinds of questions:

• 1-hop: “Who founded Veltrix Labs?” (one chunk needed)
• 2-hop: “In which city is the company founded by Mara Quillon headquartered?” (two chunks)
• 3-hop: “In which country is the company founded by Mara Quillon headquartered?” (three chunks)

It compares four retrieval strategies. Retrieval only, no LLM is involved, so the metric is simply whether the needed chunks come back:

Method	How it works
Vector	TF-IDF cosine similarity (a classic lexical vector method), top-k
Iterative vector	A non-graph multi-hop baseline: retrieve, build a follow-up query from the question’s relation words plus the best chunk, repeat
Graph	Detect entities in the question, walk the entity graph up to 3 hops, rank chunks by hop distance, then by similarity
Hybrid (RRF)	Reciprocal rank fusion of the vector and graph rankings, untuned

Metric: the percentage of questions where all supporting chunks appear in the top-k. Results are averaged over 5 random corpus seeds.

Setup

pip install numpy scikit-learn networkx

Tested with Python 3.12.3, NumPy 2.4.4, scikit-learn 1.8.0, and NetworkX 3.6.1.

The code

Save this as rag_experiment.py. It is deterministic: running it twice gives identical output.

"""Vector vs graph retrieval on a tiny synthetic multi-hop corpus.
Retrieval only (no LLM): we measure whether the supporting chunks are retrieved.
Run:  python rag_experiment.py
Deps: numpy, scikit-learn, networkx
"""
import random, re, statistics, itertools
from collections import defaultdict, deque
import numpy as np
import networkx as nx
from sklearn.feature_extraction.text import TfidfVectorizer

SYL = ["vel","tri","qua","mor","zen","bra","kor","lum","dex","nov","sil","pry","hax","ond","fur","jex"]
FIRST = ["Mara","Odin","Lena","Tariq","Ines","Piotr","Aiko","Nadia","Soren","Elif","Kofi","Rhea","Lars","Maya","Idris","Yuki"]
LAST = ["Quillon","Aldermoor","Voss","Hartwell","Okafor","Lindqvist","Marchetti","Tanaka","Brandt","Castellan","Ivanova","Reyes"]
CITIES = ["Osterby","Kaldmere","Pellston","Varnholt","Brightmoor","Tallowick","Rivenport","Halvard","Dunmarrow","Esk Harbour","Fenwick Bay","Corvale"]
COUNTRIES = ["Norvia","Alderland","Santoria","Beskia"]
PRODUCTS = ["solar routers","tidal sensors","smart ledgers","drone cameras","water filters","battery packs"]

def make_corpus(seed, n_companies=60):
    rnd = random.Random(seed)
    names, founders = set(), set()
    while len(names) < n_companies:
        names.add((rnd.choice(SYL)+rnd.choice(SYL)).capitalize()+" "+rnd.choice(["Labs","Systems","Works","Dynamics"]))
    names = sorted(names)
    SUR = ["ald","berg","chen","dorn","essen","fall","gard","holm","ivar","jorn","kess","lund","marsh","nord","orm","pell"]
    while len(founders) < n_companies:   # mostly unique surnames, so name collisions don't unfairly hurt lexical retrieval
        founders.add(rnd.choice(FIRST)+" "+(rnd.choice(SYL)+rnd.choice(SUR)).capitalize())
    founders = sorted(founders); rnd.shuffle(founders)
    city_country = {c: rnd.choice(COUNTRIES) for c in CITIES}
    chunks, meta = [], []
    def add(text, kind): chunks.append(text); meta.append(kind)
    companies = []
    for name, founder in zip(names, founders):
        city = rnd.choice(CITIES); prod = rnd.choice(PRODUCTS)
        a = len(chunks); add(f"{founder} founded {name} in {rnd.randint(1995,2020)}.", "founded")
        b = len(chunks); add(f"{name} is headquartered in {city}.", "hq")
        add(f"{name} builds {prod}.", "product")
        companies.append(dict(name=name, founder=founder, city=city, a=a, b=b))
    city_chunk = {}
    for city, country in city_country.items():
        city_chunk[city] = len(chunks); add(f"{city} is a city in the country of {country}.", "city")
        add(f"A trade fair about company headquarters was held in {city}.", "filler")
    for _ in range(40):   # lexical distractors about founding / headquarters
        f = rnd.choice(founders)
        add(f"{f} gave a talk about how founders choose where to place a headquarters.", "filler")
    return chunks, companies, city_chunk, city_country

def make_questions(companies, city_chunk, seed):
    rnd = random.Random(seed+1); qs = []
    pool = companies[:]; rnd.shuffle(pool)
    for c in pool[:30]:
        qs.append(("1-hop", f"Who founded {c['name']}?", {c["a"]}))
    for c in pool[30:60]:
        qs.append(("1-hop", f"Where is {c['name']} headquartered?", {c["b"]}))
    for c in pool[:30]:
        qs.append(("2-hop", f"In which city is the company founded by {c['founder']} headquartered?", {c["a"], c["b"]}))
    for c in pool[30:60]:
        qs.append(("3-hop", f"In which country is the company founded by {c['founder']} headquartered?", {c["a"], c["b"], city_chunk[c["city"]]}))
    return qs

STOP = {"The","A","Many","In","Who","Where","Which","What"}
ENT = re.compile(r"\b[A-Z][a-z]+(?: [A-Z][a-z]+)*\b")
def entities(text):
    out = []
    for m in ENT.findall(text):
        words = [w for w in m.split() if w not in STOP]
        if words: out.append(" ".join(words))
    return out

class VectorIndex:
    def __init__(self, chunks):
        self.vec = TfidfVectorizer(stop_words="english"); self.m = self.vec.fit_transform(chunks)
    def scores(self, q): return (self.m @ self.vec.transform([q]).T).toarray().ravel()
    def rank(self, q): return list(np.argsort(-self.scores(q), kind="stable"))

class GraphIndex:
    def __init__(self, chunks, vec_index, max_hops=3):
        self.g = nx.Graph(); self.chunk_ents = [entities(c) for c in chunks]
        self.ent_chunks = defaultdict(set); self.vi = vec_index; self.max_hops = max_hops
        for i, ents in enumerate(self.chunk_ents):
            for e in ents: self.ent_chunks[e].add(i)
            for a, b in itertools.combinations(set(ents), 2): self.g.add_edge(a, b)
    def rank(self, q):
        seeds = [e for e in entities(q) if e in self.ent_chunks]
        dist = {e: 0 for e in seeds}; dq = deque(seeds)
        while dq:
            e = dq.popleft()
            if dist[e] >= self.max_hops or e not in self.g: continue
            for n in self.g[e]:
                if n not in dist: dist[n] = dist[e]+1; dq.append(n)
        cdist = {}
        for e, d in dist.items():
            for c in self.ent_chunks[e]: cdist[c] = min(cdist.get(c, 99), d)
        sim = self.vi.scores(q)
        ranked = sorted(cdist, key=lambda c: (cdist[c], -sim[c], c))
        rest = [c for c in np.argsort(-sim, kind="stable") if c not in cdist]
        return ranked + rest

def iterative_vector(vi, chunks, q, rounds=3, per_round=2):
    """Non-graph multi-hop baseline. Each round adds the top `per_round` new chunks to the context;
    the follow-up query = the question's relation words (names removed) + the best chunk so far."""
    relation_words = ENT.sub(" ", q)
    ctx, qq = [], q
    for _ in range(rounds):
        r = [c for c in vi.rank(qq) if c not in ctx]
        ctx.extend(r[:per_round]); qq = relation_words + " " + chunks[r[0]]
    return ctx

def rrf(rank_lists, k=60):
    s = defaultdict(float)
    for rl in rank_lists:
        for r, c in enumerate(rl): s[c] += 1/(k+r+1)
    return sorted(s, key=lambda c: -s[c])

def evaluate(seed, ks=(3,5,8)):
    chunks, companies, city_chunk, _ = make_corpus(seed)
    qs = make_questions(companies, city_chunk, seed)
    vi = VectorIndex(chunks); gi = GraphIndex(chunks, vi)
    res = defaultdict(lambda: defaultdict(list))
    for hop, q, gold in qs:
        v, g = vi.rank(q), gi.rank(q)
        runs = [("vector", v), ("iterative vector", iterative_vector(vi, chunks, q)),
                ("graph", g), ("hybrid (RRF)", rrf([v, g]))]
        for name, r in runs:
            for k in ks:
                res[(name, k)][hop].append(gold <= set(r[:k]))
    return res

METHODS = ["vector", "iterative vector", "graph", "hybrid (RRF)"]
if __name__ == "__main__":
    ks = (3, 5, 8); seeds = [0, 1, 2, 3, 4]
    agg = defaultdict(lambda: defaultdict(list))
    for s in seeds:
        for key, by_hop in evaluate(s, ks).items():
            for hop, vals in by_hop.items(): agg[key][hop].append(100*np.mean(vals))
    chunks, *_ = make_corpus(0)
    print(f"corpus: {len(chunks)} chunks | seeds={seeds} | per seed: 60 one-hop, 30 two-hop, 30 three-hop questions")
    print("metric: % of questions where ALL supporting chunks are in the top-k (mean ± std over seeds)")
    for k in ks:
        print(f"\nk = {k}")
        print(f"{'method':<18}{'1-hop':>14}{'2-hop':>14}{'3-hop':>14}")
        for name in METHODS:
            row = "".join(f"{statistics.mean(agg[(name,k)][h]):>8.1f} ±{statistics.pstdev(agg[(name,k)][h]):>4.1f}" for h in ["1-hop","2-hop","3-hop"])
            print(f"{name:<18}{row}")

Run it

python rag_experiment.py

Output

corpus: 244 chunks | seeds=[0, 1, 2, 3, 4] | per seed: 60 one-hop, 30 two-hop, 30 three-hop questions
metric: % of questions where ALL supporting chunks are in the top-k (mean ± std over seeds)

k = 3
method                     1-hop         2-hop         3-hop
vector               100.0 ± 0.0     0.0 ± 0.0     0.0 ± 0.0
iterative vector      61.0 ± 3.1    81.3 ± 6.9     0.0 ± 0.0
graph                100.0 ± 0.0    84.7 ± 3.4     0.0 ± 0.0
hybrid (RRF)         100.0 ± 0.0     0.0 ± 0.0     0.0 ± 0.0

k = 5
method                     1-hop         2-hop         3-hop
vector               100.0 ± 0.0     0.0 ± 0.0     0.0 ± 0.0
iterative vector      73.3 ± 4.3    91.3 ± 4.5     7.3 ± 5.7
graph                100.0 ± 0.0    99.3 ± 1.3    89.3 ± 4.4
hybrid (RRF)         100.0 ± 0.0     0.7 ± 1.3     0.7 ± 1.3

k = 8
method                     1-hop         2-hop         3-hop
vector               100.0 ± 0.0     0.0 ± 0.0     0.0 ± 0.0
iterative vector      73.3 ± 4.3    91.3 ± 4.5    11.3 ± 9.1
graph                100.0 ± 0.0   100.0 ± 0.0   100.0 ± 0.0
hybrid (RRF)         100.0 ± 0.0     4.7 ± 5.0     5.3 ± 4.5

How to read the results

• Single-hop: everything works with plain vector retrieval (100%), matching the papers’ finding that vector RAG is strong on direct lookups. The iterative baseline is lower here because it spreads its small budget across several rounds instead of stopping when it already has the answer, a real trade-off of iterative retrieval.
• Two-hop: plain vector retrieval scores 0% at every k, because the bridging chunk shares no words with the question. Graph traversal reaches 84.7% at k = 3 and about 99% at k = 5. The iterative baseline gets partway there (81% at k = 3, 91% at k = 5), which shows that a graph is not the only way to do multi-hop retrieval.
• Three-hop: graph retrieval needs a larger budget (0% at k = 3, 89% at k = 5, 100% at k = 8) because it returns neighbours in order of distance and several nearby chunks compete for the first slots. The iterative baseline struggles with three hops.
• Hybrid with untuned fusion did worse than graph alone. Equal-weight reciprocal rank fusion let the vector ranking’s distractors push bridging chunks out of the top results. The papers’ Selection and Integration strategies are more careful, so do not read this as evidence against hybrids in general.

What this experiment does not show

Read this section before quoting any number above.

• It is a toy. The corpus is synthetic and templated, with clean, unambiguous entity names. Real text is messy.
• Graph construction is idealized. Entities come from a simple capitalization rule. Real Graph RAG uses LLM extraction, which costs money and makes mistakes, and the papers identify construction cost and quality as major factors.
• The vector baseline is lexical. TF-IDF is not a neural embedding model. I expect a dense embedding model to behave differently in places, although the core problem remains: a question that does not mention the bridge entity gives any similarity search nothing to match.
• No answer generation, latency, or cost was measured. Retrieving the right chunks is necessary but not sufficient for a correct answer.
• Baselines were revised after seeing early results. My first iterative baseline was unfairly weak (it kept re-boosting the original name and performed poorly), so I rewrote it, and I also reduced surname collisions in the data that were hurting lexical retrieval. The numbers above come from the final code. Results like these are sensitive to such choices, which is exactly why you should run and modify the script yourself.

Ideas to extend it

1. Swap TF-IDF for a sentence-embedding model and re-run (you will need a model download).
2. Add noise: misspell some entity names or add aliases, and watch graph retrieval degrade.
3. Add a global-style question and see why neither method here answers it, which is the problem community summaries were designed for.
4. Replace RRF with a rule-based router that uses the graph only when the question mentions a known entity.

Trade-Offs at a Glance

Factor	Vector RAG	Graph RAG
Build cost	Low	Much higher upfront (per Han et al. and Edge et al.)
Single-fact lookup	Strong	Often no better
Multi-hop questions	Weak without extra machinery	Designed for it
Whole-corpus questions	Weak	Strong with community summaries
Complexity	Simple, well-understood	More components to build and tune
Explainability	Chunks and similarity scores	Paths through entities (an advantage, in my judgment)
Updating documents	Re-embed new chunks	Re-extract and update the graph (in my judgment, more work)

The last two rows are my own practical assessment, not findings from the papers.

Which Should You Choose?

1. Start with vector RAG. It is cheap, mature, and good at the most common question type.
2. Look at your questions, not your documents. Do users ask for single facts, or for connections across documents? Log failures.
3. If failures are multi-hop, try cheaper options first, such as iterative or query-rewriting retrieval, before building a full graph.
4. If users ask broad, corpus-level questions, community-based Graph RAG is the approach designed for that.
5. If you have both kinds of questions, consider routing: send simple queries to vector RAG and relational ones to the graph, as Han et al.’s Selection idea suggests.
6. Measure before committing. Build a small test set of your real questions and compare, as the experiment above does in miniature.

Common Mistakes and Troubleshooting

Problem	Likely cause	Fix
Vector RAG finds only half of a multi-hop answer	Bridge entity absent from the question	Iterative retrieval, query rewriting, or a graph
Graph retrieval returns too much context	Traversal depth too large, no ranking	Limit hops; rank by distance and similarity
Graph answers are wrong despite good retrieval	Extraction errors in the graph	Sample and audit extracted triples
Great demo, poor production results	Tested only on questions the graph suits	Evaluate on a realistic mix of questions
Hybrid scores below either method	Untuned fusion	Tune weights, or route by query type
LLM-judged win rates look too good	Judge position bias	Randomize answer order; add automatic metrics

Implementation Checklist

• Collect at least 50 real or realistic questions, labeled by type
• Measure retrieval success (are the needed chunks returned?) separately from answer quality
• Baseline with vector RAG before adding a graph
• Budget for graph construction cost and re-indexing
• Audit a sample of extracted entities and relations
• Compare answer latency and cost, not just accuracy
• Consider routing or hybrid strategies only after the baselines are measured

Frequently Asked Questions

What is the main difference between vector RAG and Graph RAG? Vector RAG retrieves text chunks that are semantically similar to the question. Graph RAG builds a graph of entities and relationships and retrieves by following those connections or by using pre-written community summaries.

Is Graph RAG better than vector RAG? Not across the board. Controlled studies find RAG stronger on single-hop and detail-oriented questions and Graph RAG stronger on multi-hop reasoning and broad summarization, with results depending heavily on design.

Why is Graph RAG more expensive? It needs an extra indexing stage that extracts entities and relationships, often with an LLM, and sometimes generates community summaries. The papers report much higher upfront construction costs than for plain vector indexing.

Can I combine both? Yes. Han et al. propose a Selection strategy (route queries to one approach) and an Integration strategy (merge both retrievals), and report consistent improvements over either alone.

Do I need Graph RAG for multi-hop questions? Not necessarily. Iterative retrieval can handle some multi-hop cases, as the experiment above shows, though it has its own limits and depends on how follow-up queries are built.

Where to Go Next

Read Han et al. for the clearest controlled comparison, then Edge et al. to understand community summaries, and try extending the script with real embeddings. A natural follow-up for your own project is building a small evaluation set from your actual user questions.

References

• Edge et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization (Microsoft, 2024).
• Han et al. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv, latest revision March 2026).
• Xiang et al. When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation (ICLR 2026).
• Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020), the original RAG paper.
• Experiment code: rag_experiment.py (included above; depends on NumPy, scikit-learn, and NetworkX).