Document Ingestion Pipeline in GraphRAG SDK

When you call rag.ingest("document.txt"), the SDK transforms your raw text into a structured knowledge graph through a 9-step sequential pipeline. Think of it as an assembly line: each step takes the output of the previous one, refines it, and passes it forward. This document explains what each step does, why it exists, and how to tune it.

This pipeline is built for prose. For CSVs and other tabular sources, see Structured Ingestion: you declare what the columns mean and the graph is written deterministically, with no model in the loop, into the same graph your documents live in.

The Big Picture

                         Your Document
                              |
                    ┌─────────┴─────────┐
                    v                    v
              1. Load Text         (or pass text= directly)
                    |
              2. Chunk Text
                    |
              3. Build Lexical Graph
              (Document + Chunk nodes, provenance edges)
                    |
              4. Extract Entities & Relationships
                    |
              4b. Quality Filter
              (remove bad nodes/dangling edges)
                    |
              5. Prune Against Schema
              (keep only schema-conforming data)
                    |
              6. Resolve Duplicates
              (merge same-entity mentions)
                    |
              7. Write to Graph
              (batched MERGE into FalkorDB)
                    |
         ┌─────────┴─────────┐
         v                    v
   8. Write Mentions    9. Index Chunks
   (MENTIONED_IN edges)  (embed + store vectors)
         └─────────┬─────────┘
                   v
            Done — IngestionResult

Steps 1-7 run sequentially (each depends on the previous). Steps 8-9 run in parallel since they’re independent.

Step-by-Step Explanation

Step 1 — Load

What it does: Reads raw text from a file, URL, or string. How: The LoaderStrategy ABC handles this. The SDK auto-detects the loader based on file extension:

.pdf files use PdfLoader
Everything else uses TextLoader
If you pass text= directly, the loader step is skipped entirely

Output: DocumentOutput containing the raw text and a DocumentInfo with a unique ID and file path. Code: LoaderStrategy.load() in ingestion/loaders/base.py

Step 2 — Chunk

What it does: Splits the document text into smaller overlapping windows called chunks. Each chunk is small enough for the LLM to process, but large enough to contain meaningful context. How: The default FixedSizeChunking uses a sliding window:

Window size: 1000 characters (configurable)
Overlap: 100 characters between consecutive chunks
Step size: chunk_size - chunk_overlap = 900 characters

Why overlap? Without overlap, an entity mentioned right at the boundary between two chunks might be split across them and lost. Overlap ensures entities near boundaries appear in at least one complete chunk. Output: TextChunks — a list of TextChunk objects, each with a unique ID (uid), the text content, and an index number. Code: ChunkingStrategy.chunk() in ingestion/chunking_strategies/base.py

Step 3 — Build Lexical Graph (Mandatory)

What it does: Creates the provenance backbone of the knowledge graph — this is how every answer traces back to its source document. Creates:

1 Document node (with the file path and metadata)
N Chunk nodes (one per text chunk, storing the chunk text and index)
N PART_OF edges (Document → each Chunk)
N-1 NEXT_CHUNK edges (Chunk → next Chunk, preserving reading order)

The result looks like:

Document
├── PART_OF → Chunk 0
├── PART_OF → Chunk 1
└── PART_OF → Chunk 2

Chunk 0 ──NEXT_CHUNK──> Chunk 1 ──NEXT_CHUNK──> Chunk 2

Why mandatory? This is the Zero-Loss Data principle — every piece of source material is traceable in the graph. When the retrieval system finds a chunk, it can always trace back to the source document. Code: IngestionPipeline._build_lexical_graph() in ingestion/pipeline.py

Step 4 — Extract Entities & Relationships

What it does: The most important step — an LLM reads each chunk and extracts structured knowledge: entities (people, places, organizations, etc.) and the relationships between them. How: The default GraphExtraction strategy uses a 2-step process:

Step 1 (NER): A pluggable entity extractor identifies entities in the text. Default: GLiNER (a local transformer model, no API calls needed). Alternative: LLMExtractor (uses the LLM for NER).
Step 2 (Verify + Relationships): The LLM receives the pre-extracted entities and the original text. It verifies the entities (fixing errors, adding missed ones) and extracts all relationships between them.

For a detailed explanation of the extraction process, see Extraction. Output: GraphData containing nodes (entities), relationships, and mention records. Code: ExtractionStrategy.extract() in ingestion/extraction_strategies/base.py

Step 4b — Quality Filter

What it does: Removes bad data that slipped through extraction — nodes with empty or None IDs, and relationships whose endpoints don’t exist. Why: LLMs sometimes produce malformed output (empty entity names, references to entities that weren’t extracted). This step catches those before they reach the graph. Code: IngestionPipeline._filter_quality() in ingestion/pipeline.py

Step 5 — Prune Against Schema

What it does: Filters extracted data to only keep entities and relationships that match your schema definition. How it works:

If your schema defines entity types (e.g., Person, Organization, Location), only entities with those labels pass through
If your schema defines relationship types, only those relationship types pass through
Relationships whose endpoints were pruned are also removed
Special cases: "Unknown" entities (low-confidence NER) and "RELATES" edges (the unified relationship type) always pass through

Open schema mode: If you define no entity or relationship types (empty GraphSchema()), this step is skipped entirely — everything passes through. Code: IngestionPipeline._prune() in ingestion/pipeline.py

Step 6 — Resolve Duplicates

What it does: Merges entities that refer to the same real-world thing. When the LLM extracts “Alice” from chunk 1 and “Alice” from chunk 5, this step recognizes they’re the same entity and merges them. Default at ingest: ExactMatchResolution

Groups by (normalized name, label) only — same name but different labels stay separate (e.g., Person “Paris” vs Location “Paris”); the judge decides those at finalize()
Survivor keeps every member’s description joined with " | " (no LLM summary); all relationship endpoints are re-pointed to the survivor before the write, so no edge is lost
No embeddings, no LLM calls; zero added cost
Ingest is per document, so a resolver here can only compare one file’s entities with each other — the cross-document duplicates that matter are handled once, at finalize()

Alternative: LLMVerifiedResolution

Runs the same exact-match pass first, then compares the surviving names by embedding similarity within each label
Pairs at or above hard_threshold (0.95) merge outright; pairs down to soft_threshold (0.65) go to the LLM for a YES/NO verdict
Useful when the same entity is named inconsistently across documents (e.g., “Acme Corp” vs “Acme Corporation”)

At finalize(): the LLM-judged cross-document phase

Runs after the exact-name dedup and the resolver pass, over every entity of every document
Name and description embeddings plus “A’s name appears in B’s description” nominate candidate pairs (no LLM); candidates form dense sets of ≤ 8
The judge LLM partitions each set into same-referent groups; with judge_vote=True (default) a second pass over shuffled sets must agree, with judge_vote=False the single pass decides (a pair that reached across two sets is still linked rather than merged, with agreement=1)
Both passes agree → merge, by the same rules as every finalize merge: a table’s row survives a mention, every member’s description is kept in descriptions (joined with " | " in description), other names kept in aliases, edges moved first, then the loser deleted; the survivor also gains every member’s label
Only one pass agrees → link: a SAME_AS edge (source='llm_judge'), both nodes stay
Pairs a resolver remembered as DISTINCT_FROM, two keyed rows of a table, and a mention two such rows could own are never asked, grouped or linked
finalize(judge=False) skips it (finalize(resolve=False, judge=False) runs no model at all); deduplicate_entities() on its own runs it only with judge=True; judge_llm= picks the model (gpt-4.1-class recommended); pass resolver=LLMVerifiedResolution(...) to ingest() to also resolve per document

For details on resolution strategies, see Strategies. Code: ResolutionStrategy.resolve() in ingestion/resolution_strategies/base.py

Step 7 — Write to Graph

What it does: Persists all the extracted and resolved data into FalkorDB using batched Cypher queries. How:

Nodes are written via UNWIND $batch AS item MERGE (n:Label {id: item.id}) SET n += item.properties
Relationships are written similarly, using label hints for efficient MATCH operations
Batch size: 500 items per query
Entity nodes automatically get the __Entity__ secondary label (structural nodes like Chunk and Document do not)

For details on the storage layer, see Storage. Code: GraphStore.upsert_nodes() and GraphStore.upsert_relationships() in storage/graph_store.py

Steps 8 & 9 — Mentions + Index Chunks (Parallel)

These two steps run simultaneously since they’re independent:

Step 8 — Write Mentions

What it does: Creates MENTIONED_IN edges linking every entity to every chunk it was extracted from. These edges are critical for retrieval — they let the system find text passages for any entity. Details: Uncapped — every entity-chunk pair gets an edge. Duplicates are deduplicated by (entity_id, chunk_id).

Step 9 — Index Chunks

What it does: Embeds each chunk’s text into a vector and stores it on the Chunk node. These embeddings power the vector similarity search during retrieval. How:

Batch-embed all chunk texts in one API call (aembed_documents)
Write vectors to Chunk nodes via SET c.embedding = vecf32(vector)
Falls back to sequential embedding if the batch call fails

Code:

Mentions: IngestionPipeline._write_mentions() in ingestion/pipeline.py
Chunk indexing: VectorStore.index_chunks() in storage/vector_store.py

Post-Ingestion: finalize()

After all documents are ingested, call finalize() to prepare the graph for querying. This is a separate step because some operations (like deduplication) work best when run globally across all documents, not per-document. finalize() runs 5 steps in order:

Step	What it does	Why
1. NULL cleanup	Removes entities with `NULL` names	Legacy edge cases from MERGE operations
2. Deduplicate	Global exact-name entity dedup	”Alice” from doc 1 and doc 5 become one node
3. Entity embeddings	Embeds entity names into vectors	Powers entity vector search during retrieval
4. Relationship embeddings	Embeds fact text on RELATES edges	Powers RELATES edge vector search during retrieval
5. Ensure indexes	Creates all 5 standard indexes	Vector + fulltext indexes for search

# After ingesting ALL documents:
stats = await rag.finalize()
# Returns: {
#     "null_stubs_removed": 0,
#     "entities_deduplicated": 142,
#     "entities_embedded": 3200,
#     "relationships_embedded": 8500,
#     "indexes": { "vector_Chunk": True, "vector___Entity__": True, ... }
# }

Important: Do not call finalize() after each document — call it once after all ingestion is complete. Entity backfill re-scans all entities and is slow when called repeatedly.

Configuration Quick Reference

Chunking

Parameter	Default	Benchmark Value	Effect
`chunk_size`	1000	1500	Larger = richer extraction context, more LLM tokens
`chunk_overlap`	100	200	Larger = fewer boundary losses, more redundancy

Extraction

Parameter	Default	Effect
`entity_extractor`	`GLiNERExtractor()`	Local NER model (fast, no API calls)
`coref_resolver`	`None`	Optional coreference resolution (resolves pronouns)
`entity_types`	11 defaults	Custom ontology for your domain
`max_concurrency`	LLM default (12)	Parallel LLM calls during extraction

Resolution

Strategy	When to use
`ExactMatchResolution` (default)	Fast, deterministic. Good when extraction is consistent
`LLMVerifiedResolution`	Multi-document ingestion where names vary. Costs embedding calls, plus LLM calls for ambiguous pairs. Cross-document dedup is `finalize()`’s resolver and judge phases

Performance Notes

Slowest step: Extraction (Step 4) — involves LLM calls for every chunk. Expect ~2-5 seconds per chunk.
Fastest step: Quality filter, prune, and resolve — all in-memory, sub-second.
Parallelism: Steps 8-9 run in parallel. Step 1 NER uses a semaphore (default 12 concurrent calls).
Batch size: The benchmark uses 1500-character chunks. 20 documents (~4.7 MB total) take ~47 minutes to ingest.

File Reference

File	What it contains
`pipeline.py`	The 9-step pipeline orchestrator
`api/main.py`	`GraphRAG.ingest()` and `finalize()` — user-facing API
`loaders/text_loader.py`	TextLoader implementation
`loaders/pdf_loader.py`	PdfLoader implementation
`chunking_strategies/fixed_size.py`	FixedSizeChunking implementation
`extraction_strategies/graph_extraction.py`	2-step extraction (NER + LLM verify/rels)
`resolution_strategies/exact_match.py`	ExactMatchResolution
`resolution_strategies/llm_verified_resolution.py`	LLMVerifiedResolution
`storage/judge_dedup.py`	LLMJudgeDeduplicator — cross-document dedup at `finalize()`
`storage/graph_store.py`	Batched node/relationship writes
`storage/vector_store.py`	Chunk embedding + indexing

Home

Pipeline

Ontology

Reference

Document Ingestion Pipeline in GraphRAG SDK

The Big Picture

Step-by-Step Explanation

Step 1 — Load

Step 2 — Chunk

Step 3 — Build Lexical Graph (Mandatory)

Step 4 — Extract Entities & Relationships

Step 4b — Quality Filter

Step 5 — Prune Against Schema

Step 6 — Resolve Duplicates

Step 7 — Write to Graph

Steps 8 & 9 — Mentions + Index Chunks (Parallel)

Step 8 — Write Mentions

Step 9 — Index Chunks

Post-Ingestion: finalize()

Configuration Quick Reference

Chunking

Extraction

Resolution

Performance Notes

File Reference

​The Big Picture

​Step-by-Step Explanation

​Step 1 — Load

​Step 2 — Chunk

​Step 3 — Build Lexical Graph (Mandatory)

​Step 4 — Extract Entities & Relationships

​Step 4b — Quality Filter

​Step 5 — Prune Against Schema

​Step 6 — Resolve Duplicates

​Step 7 — Write to Graph

​Steps 8 & 9 — Mentions + Index Chunks (Parallel)

​Step 8 — Write Mentions

​Step 9 — Index Chunks

​Post-Ingestion: finalize()

​Configuration Quick Reference

​Chunking

​Extraction

​Resolution

​Performance Notes

​File Reference

The Big Picture

Step-by-Step Explanation

Step 1 — Load

Step 2 — Chunk

Step 3 — Build Lexical Graph (Mandatory)

Step 4 — Extract Entities & Relationships

Step 4b — Quality Filter

Step 5 — Prune Against Schema

Step 6 — Resolve Duplicates

Step 7 — Write to Graph

Steps 8 & 9 — Mentions + Index Chunks (Parallel)

Step 8 — Write Mentions

Step 9 — Index Chunks

Post-Ingestion: finalize()

Configuration Quick Reference

Chunking

Extraction

Resolution

Performance Notes

File Reference