Skip to main content
Results for GraphRAG-SDK on GraphRAG-Bench (Xiang et al., ICLR 2026), covering both subsets and all four task categories.
For the headline comparison against vector RAG (71.48 vs 55.39), the citation block and what a benchmark does and does not tell you about production hallucinations, see GraphRAG Accuracy Benchmark: FalkorDB vs Vector RAG. This page is the deep methodology reference behind those numbers.
The reproduction package — the runner, the ontology builders and pinned dependencies — is available on request; open an issue or ask on Discord. The ontologies themselves are published below, along with the full configuration. This score is a ceiling on task performance against a fixed dataset, not a guarantee about your data. Grounded retrieval and an explicit abstention path are what set the floor for production reliability — see Reducing LLM Hallucinations.

​
Results

Scores are ACC unless noted. Average is the unweighted mean of the four ACC values, the convention the GraphRAG-Bench leaderboard uses.

​
GraphRAG-Bench (Medical) — Average 76.87

​
GraphRAG-Bench (Novel) — Average 66.09

​
Against the published leaderboard

Competitor figures are taken from the GraphRAG-Bench leaderboard as published; we did not re-run them. All entries use gpt-4o-mini as the generation backbone.

​
Configuration

Per the benchmark’s Appendix H.2, GraphRAG-SDK’s own defaults are preserved rather than tuned to the benchmark; the protocol is matched on the backbone model and the judge.

​
Declared deviations from SDK defaults

Two settings were raised above the SDK’s own defaults. Both are single global values, chosen before any results were scored, and applied identically to both subsets.
  • enable_cypher = True (SDK default False) — ontology-guided text-to-Cypher retrieval is a core GraphRAG-SDK capability, enabled so the measurement reflects the framework as deployed. It contributed to 22% of Medical and 29% of Novel answers.
  • embedding_dimension = 1024 (SDK default 256) — the same vector width as bge-large-en-v1.5, the embedding model Appendix H.2 specifies for evaluated systems.

​
Benchmark integrity

The ontologies were hand-authored from the corpus only. The question sets and the evidence, evidence_relations and evidence_triple fields were never read during ontology design or indexing. Both are published here so this is checkable without needing the runner: They contain only domain type definitions — entity labels, their descriptions and properties, and the relation patterns between them. No question, answer or evidence text appears in either file.

​
How the run works

Per subset, the pipeline runs in three stages:
  1. Index — each corpus document is chunked, and entities and relationships are extracted against that subset’s ontology into its own FalkorDB graph (one graph per document, so retrieval is scoped to the document a question was written against).
  2. Answer — every question in the subset is answered through MultiPathRetrieval, with the retrieved context recorded alongside the answer.
  3. Judge — answers are scored by the benchmark’s own Evaluation/generation_eval.py, unmodified, with gpt-4o-mini as judge and bge-large-en-v1.5 embeddings.
The run produces, per subset: the leaderboard row, the fully resolved configuration, per-question predictions with their retrieved context, and each answer’s individual judge scores. Indexing cost, measured on a 24-core machine and dominated by extraction API calls rather than local compute: Medical (1 corpus, 510 chunks) ~1.5 h end to end; Novel (20 documents, ~2,500 chunks) ~4 h.

​
Counts and caveats

Two things worth stating plainly: Novel is 2009 rather than 2010. The dataset contains a duplicate question id (Novel-55f0c0e2) carrying identical question text under two different source values with contradictory ground truths — one answers the question asked, the other is unrelated. Our runner answers each id once. This has been reported to the benchmark authors. The evaluator skips samples whose judge call raises, so the averages are over 2053 of 2062 Medical and 1995 of 2009 Novel predictions. Every prediction itself succeeded; there were no generation failures or empty answers in either subset.
A benchmark score measures task performance on a fixed, offline dataset. It does not measure hallucination rate on your own data — that also depends on your source-data quality, how narrowly retrieval is scoped, prompting, and the abstention logic your application defines. See Reducing LLM Hallucinations for how to ground and verify answers in your own deployment.
Leaderboard figures as published at graphrag-bench.github.io. Last updated: August 2026.