Skip to main content
Results for GraphRAG-SDK on GraphRAG-Bench (Xiang et al., ICLR 2026), covering both subsets and all four task categories. The reproduction package — the runner, the ontology builders and pinned dependencies — is available on request; open an issue or ask on Discord. The ontologies themselves are published below, along with the full configuration.

​
Results

Scores are ACC unless noted. Average is the unweighted mean of the four ACC values, the convention the GraphRAG-Bench leaderboard uses.

​
GraphRAG-Bench (Medical) — Average 76.87

​
GraphRAG-Bench (Novel) — Average 66.09

​
Against the published leaderboard

Competitor figures are taken from the GraphRAG-Bench leaderboard as published; we did not re-run them. All entries use gpt-4o-mini as the generation backbone.

​
Configuration

Per the benchmark’s Appendix H.2, GraphRAG-SDK’s own defaults are preserved rather than tuned to the benchmark; the protocol is matched on the backbone model and the judge.

​
Declared deviations from SDK defaults

Two settings were raised above the SDK’s own defaults. Both are single global values, chosen before any results were scored, and applied identically to both subsets.
  • enable_cypher = True (SDK default False) — ontology-guided text-to-Cypher retrieval is a core GraphRAG-SDK capability, enabled so the measurement reflects the framework as deployed. It contributed to 22% of Medical and 29% of Novel answers.
  • embedding_dimension = 1024 (SDK default 256) — the same vector width as bge-large-en-v1.5, the embedding model Appendix H.2 specifies for evaluated systems.

​
Benchmark integrity

The ontologies were hand-authored from the corpus only. The question sets and the evidence, evidence_relations and evidence_triple fields were never read during ontology design or indexing. Both are published here so this is checkable without needing the runner: They contain only domain type definitions — entity labels, their descriptions and properties, and the relation patterns between them. No question, answer or evidence text appears in either file.

​
How the run works

Per subset, the pipeline runs in three stages:
  1. Index — each corpus document is chunked, and entities and relationships are extracted against that subset’s ontology into its own FalkorDB graph (one graph per document, so retrieval is scoped to the document a question was written against).
  2. Answer — every question in the subset is answered through MultiPathRetrieval, with the retrieved context recorded alongside the answer.
  3. Judge — answers are scored by the benchmark’s own Evaluation/generation_eval.py, unmodified, with gpt-4o-mini as judge and bge-large-en-v1.5 embeddings.
The run produces, per subset: the leaderboard row, the fully resolved configuration, per-question predictions with their retrieved context, and each answer’s individual judge scores. Indexing cost, measured on a 24-core machine and dominated by extraction API calls rather than local compute: Medical (1 corpus, 510 chunks) ~1.5 h end to end; Novel (20 documents, ~2,500 chunks) ~4 h.

​
Counts and caveats

Two things worth stating plainly: Novel is 2009 rather than 2010. The dataset contains a duplicate question id (Novel-55f0c0e2) carrying identical question text under two different source values with contradictory ground truths — one answers the question asked, the other is unrelated. Our runner answers each id once. This has been reported to the benchmark authors. The evaluator skips samples whose judge call raises, so the averages are over 2053 of 2062 Medical and 1995 of 2009 Novel predictions. Every prediction itself succeeded; there were no generation failures or empty answers in either subset.
Leaderboard figures as published at graphrag-bench.github.io. Last updated: August 2026.