Reducing LLM Hallucinations

​

Why RAG systems produce unsupported answers

• Retrieval misses the connecting fact. The answer requires combining two facts from different documents (or different parts of the same document), but no single chunk contains both, so similarity search never surfaces the link between them. Multi-hop questions — “who manages the office that Acme opened in 2026?” — are the usual casualty.
• Chunks are similar but not relevant. Cosine similarity rewards topical overlap, not answer-bearing content. A chunk can rank highly because it shares vocabulary with the question while containing none of the facts needed to answer it.
• The model fills the gap. When the retrieved context is incomplete, an LLM will usually still produce a fluent, confident-sounding answer rather than say “I don’t know” — because nothing in the pipeline checks whether the context actually supports the claim before generation.

​

How each mechanism grounds an answer

​

Vector search

​

Full-text / keyword search

​

Graph traversal and relationship expansion

​

Ontology-guided text-to-Cypher

from graphrag_sdk import ConnectionConfig, FalkorDBConnection, GraphStore, VectorStore
from graphrag_sdk.retrieval.strategies.multi_path import MultiPathRetrieval

connection = FalkorDBConnection(ConnectionConfig(host="localhost", graph_name="my_graph"))

strategy = MultiPathRetrieval(
    graph_store=GraphStore(connection),
    vector_store=VectorStore(connection, embedder=embedder, embedding_dimension=256),
    embedder=embedder,
    llm=llm,
    enable_cypher=True,  # opt-in, off by default
    # Required: GraphRAG injects the ontology only into its own default
    # strategy, never into one you pass as `strategy=`. Without this, the
    # text-to-Cypher prompt is built against an empty schema.
    ontology=await rag.get_ontology(),
)
result = await rag.completion("How many organizations are mentioned?", strategy=strategy)

​

Cosine reranking

​

Inspecting the evidence

result = await rag.completion("What did Professor Harmon discover?", return_context=True)

print(result.answer)

for item in result.retriever_result.items:
    if item.metadata.get("section") == "passages":
        # The section content opens with a "## Source Document Passages"
        # header glued to the first passage — strip it before splitting.
        body = item.content.removeprefix("## Source Document Passages\n")
        for passage in body.split("\n---\n"):
            print(passage)

context = await rag.retrieve("What did Professor Harmon discover?")

print(len(context.items), "context items")
for item in context.items:
    print(item.metadata.get("section"), item.content[:120])

​

Defining an abstention path

INSUFFICIENT_EVIDENCE = (
    "I don't have enough evidence in the knowledge graph to answer that question."
)

#: MultiPathRetrieval emits an answer-format hint section carrying no
#: document evidence. Ignore it when deciding whether to abstain.
NON_EVIDENCE_SECTIONS = frozenset({"hint"})


async def answer_or_abstain(rag, question: str, *, min_items: int = 1, min_score=None):
    """Answer from graph context, or abstain when the evidence is too thin."""
    retrieved = await rag.retrieve(question)

    supporting = [
        item
        for item in retrieved.items
        if (item.content or "").strip()
        and item.metadata.get("section") not in NON_EVIDENCE_SECTIONS
        and (min_score is None or (item.score is not None and item.score >= min_score))
    ]

    if len(supporting) < min_items:
        return INSUFFICIENT_EVIDENCE, []

    result = await rag.completion(question, return_context=True)
    return result.answer, result.retriever_result.items

• Filter non-evidence sections. The default MultiPathRetrieval strategy emits an answer-format hint section that carries no document evidence; counting it would defeat the gate.
• Only threshold on scores when the pipeline produces them. MultiPathRetrieval returns section items with score = None — scoring happens internally during fact filtering and passage reranking and is not surfaced per section. Attach a CosineReranker (or use LocalRetrieval) to populate item.score before setting min_score. See Strategies.
• A count-only gate barely gates. No retrieval path applies a similarity floor: chunk vector search takes its top 15 hits and reranking takes the top k of those, neither with a threshold. A populated graph therefore returns something for almost any question, however unrelated, so min_items alone fires only on an empty graph, a failed ingestion or a retrieval error. Scores are what make the gate discriminating.
• Return a fixed refusal string, rather than prose the model wrote, so abstentions stay machine-detectable in logs and evaluations.
• Don’t share one Context across both calls. Context starts its clock when it is constructed and remaining_budget_ms never resets, so a ctx passed to retrieve() and then reused for completion() charges the gate’s latency against the generation’s budget — and the gate is the cheap half. Pass ctx.child() to the second call: it inherits the tenant, trace and remaining budget, but starts its own clock.

​

Schema and ontology as a guardrail

from graphrag_sdk import Entity, Ontology, Relation

ontology = Ontology(
    entities=[
        Entity(label="Person", description="A human being"),
        Entity(label="Organization", description="A company or institution"),
    ],
    relations=[
        Relation(
            label="WORKS_AT",
            description="Is employed by",
            patterns=[("Person", "Organization")],  # source -> target
        ),
    ],
)

​

Verifying grounding

1. Check the trail. Run completion(question, return_context=True) and confirm the passages and facts sections contain the specific fact the answer depends on — not just topically related text.
2. Cross-reference the source. Compare [Source: <document>] tags against the actual source document to confirm the passage says what the answer claims.
3. Prefer structured evidence for high-stakes claims. Questions answerable from the facts section (RELATES edges) or a cypher_results section beat free-text passages alone — structured graph results are direct query output, not LLM-summarized prose.
4. Log the trail. Store retriever_result items next to each served answer so any claim can be re-checked later.
5. Test the refusal path. Assert that a question with no supporting context returns your evidence-insufficient response.
6. Track abstention rate. A rate near zero usually means the gate is too loose; a rate that spikes usually means ingestion or finalize() coverage regressed.

​

Related

• Reliability and Grounding — the API-level reference behind every mechanism on this page
• Architecture — how chunks, entities and MENTIONED_IN edges fit together
• Retrieval — the retrieval paths that gather evidence
• Strategies — retrieval and reranking strategies, including scored ones
• Benchmark — measured accuracy and reproduction instructions