Skip to main content
Last updated: August 2026. All figures are point-in-time measurements for GraphRAG-SDK 1.3.0 with gpt-4o-mini, and for competitor entries as published on the GraphRAG-Bench leaderboard in August 2026.

​
Summary: how much more accurate is GraphRAG than vector RAG?

On GraphRAG-Bench (Xiang et al., ICLR 2026), the FalkorDB GraphRAG SDK scores 71.48 overall against 55.39 for the benchmark’s vector-RAG baseline, RAG (w/ rerank) — a +16.09 point absolute gap, or ~29% relative improvement. Both systems use the same generation backbone, gpt-4o-mini, and the same judge, so the difference is attributable to retrieval, not to the model. Overall = (Novel ACC + Medical ACC) / 2. Overall is our own cross-dataset summary; the GraphRAG-Bench leaderboard ranks each dataset separately.
Competitor figures are taken unchanged from the published GraphRAG-Bench leaderboard as of August 2026. We did not re-run them. See Limitations.
For the full 15-system leaderboard comparison, ontology integrity notes and per-run counts, see the deep methodology reference at Benchmark.

​
What was measured, and on which datasets?

GraphRAG-Bench has two subsets, each with four task categories. The four task categories, scored identically on both subsets, are Fact Retrieval, Complex Reasoning, Contextual Summarize and Creative Generation.

​
Per-category results — GraphRAG-Bench (Medical), ACC average 76.87

​
Per-category results — GraphRAG-Bench (Novel), ACC average 66.09

Cov is coverage and FS is factual score, reported by the benchmark’s evaluator for the categories where they apply.

​
How is accuracy (ACC) computed?

Per dataset, ACC is the unweighted mean of the four task-category scores, matching the GraphRAG-Bench leaderboard convention:
Scoring is done by the benchmark’s own Evaluation/generation_eval.py, unmodified, with:

​
What exact model and embedding configuration produced 71.48?

Per the benchmark’s Appendix H.2, GraphRAG-SDK’s own defaults are preserved rather than tuned to the benchmark; the protocol is matched on the backbone model and the judge.

​
Declared deviations from SDK defaults

Two settings were raised above the SDK’s own defaults. Both are single global values, chosen before any results were scored, and applied identically to both subsets.
  • enable_cypher = True (SDK default False) — ontology-guided text-to-Cypher retrieval is a core GraphRAG-SDK capability, enabled so the measurement reflects the framework as deployed. It contributed to 22% of Medical and 29% of Novel answers.
  • embedding_dimension = 1024 (SDK default 256) — the same vector width as bge-large-en-v1.5, the embedding model Appendix H.2 specifies for evaluated systems.
Declaring these is the point: a benchmark number is only checkable if every deviation from the shipped defaults is stated up front.

​
How do I reproduce these results?

Point the SDK at Azure OpenAI gpt-4o-mini for both extraction and generation, and at text-embedding-3-large with dimensions=1024 for embeddings.
Download the Novel and Medical subsets and the Evaluation/ directory from the GraphRAG-Bench project. Use them as published; do not filter or re-split the question sets.
Both ontologies are published so the run is checkable without the runner:They were hand-authored from the corpus only. The question sets and the evidence, evidence_relations and evidence_triple fields were never read during ontology design or indexing, and no question, answer or evidence text appears in either file.
Per subset, the pipeline runs in three stages:
  1. Index — each corpus document is chunked, and entities and relationships are extracted against that subset’s ontology into its own FalkorDB graph (one graph per document, so retrieval is scoped to the document a question was written against).
  2. Answer — every question in the subset is answered through MultiPathRetrieval, with the retrieved context recorded alongside the answer.
  3. Judge — answers are scored by the benchmark’s own Evaluation/generation_eval.py, unmodified, with gpt-4o-mini as judge and bge-large-en-v1.5 embeddings.
The run produces, per subset: the leaderboard row, the fully resolved configuration, per-question predictions with their retrieved context, and each answer’s individual judge scores.

​
How long does indexing take?

Measured on a 24-core machine, and dominated by extraction API calls rather than local compute:
The full reproduction package — the runner, the ontology builders and pinned dependencies — is available on request. Open an issue or ask on Discord. The ontologies and the full configuration are already published above.

​
Limitations: what this benchmark does not prove

This section matters more than the headline number. Read it before citing.
  • Competitor numbers were not re-run by us. They are taken from the published GraphRAG-Bench leaderboard as of August 2026. Hardware, run dates and prompt versions differ between entries.
  • Novel produced 2,009 predictions rather than 2,010. The dataset contains a duplicate question id (Novel-55f0c0e2) carrying identical question text under two different source values with contradictory ground truths — one answers the question asked, the other is unrelated. Our runner answers each id once. This has been reported to the benchmark authors.
  • The evaluator skips samples whose judge call raises. Averages are over 2,053 of 2,062 Medical and 1,995 of 2,009 Novel predictions. Every prediction itself succeeded: there were no generation failures and no empty answers in either subset.
  • LLM-as-judge scoring carries its own variance and bias. The judge is a language model; repeated runs will not be bit-identical, and judge preferences can favour particular answer styles.
  • Two datasets in two domains do not generalize to every workload. Fiction and medical text are not your corpus. Results are point-in-time for a specific model and SDK version pairing.
  • Overall (71.48 / 55.39) is our own cross-dataset average, not a figure published by the leaderboard, which ranks each dataset separately.

​
What a benchmark does and does not tell you

An accuracy benchmark measures task performance on a fixed dataset with fixed questions and fixed ground truths. That is a genuine signal — it is the cleanest apples-to-apples comparison of retrieval quality available — but it is not the same thing as reliability in production. Reducing hallucinations in a deployed system depends on four things the benchmark holds constant or does not exercise at all:
  • Source-data quality and coverage. The graph can only ground what was ingested. Missing, stale or contradictory source documents put a ceiling on any retrieval method.
  • Retrieval rules and scoping. Traversal depth, per-tenant graph isolation and ontology-guided Cypher determine whether the right evidence is even reachable. See Retrieval and Strategies.
  • Prompting and context construction. How retrieved entities, relations and chunks are assembled into the generation prompt changes what the model attends to.
  • Abstention logic. Returning an evidence-insufficient response — rather than guessing — when retrieval comes back empty or below a confidence threshold. A system that answers everything will eventually answer wrongly.
No retrieval architecture eliminates hallucinations. Graph-grounded retrieval reduces them, grounds answers in retrieved evidence and makes claims verifiable by exposing the traversal path behind each answer. Treat a higher benchmark score as raising the ceiling; the surrounding system — data, scoping, prompting and abstention — determines the floor.

​
How to cite

Plain text:
BibTeX:

​
Source code and further reading

Leaderboard figures as published at graphrag-bench.github.io. Last updated: August 2026.