rag.ingest("document.txt"), the SDK transforms your raw text into a structured knowledge graph through a 9-step sequential pipeline. Think of it as an assembly line: each step takes the output of the previous one, refines it, and passes it forward.
This document explains what each step does, why it exists, and how to tune it.
This pipeline is built for prose. For CSVs and other tabular sources, see
Structured Ingestion: you declare what the columns mean
and the graph is written deterministically, with no model in the loop, into the
same graph your documents live in.
The Big Picture
Step-by-Step Explanation
Step 1 — Load
What it does: Reads raw text from a file, URL, or string. How: TheLoaderStrategy ABC handles this. The SDK auto-detects the loader based on file extension:
.pdffiles usePdfLoader- Everything else uses
TextLoader - If you pass
text=directly, the loader step is skipped entirely
DocumentOutput containing the raw text and a DocumentInfo with a unique ID and file path.
Code: LoaderStrategy.load() in ingestion/loaders/base.py
Step 2 — Chunk
What it does: Splits the document text into smaller overlapping windows called chunks. Each chunk is small enough for the LLM to process, but large enough to contain meaningful context. How: The defaultFixedSizeChunking uses a sliding window:
- Window size: 1000 characters (configurable)
- Overlap: 100 characters between consecutive chunks
- Step size:
chunk_size - chunk_overlap= 900 characters
TextChunks — a list of TextChunk objects, each with a unique ID (uid), the text content, and an index number.
Code: ChunkingStrategy.chunk() in ingestion/chunking_strategies/base.py
Step 3 — Build Lexical Graph (Mandatory)
What it does: Creates the provenance backbone of the knowledge graph — this is how every answer traces back to its source document. Creates:- 1 Document node (with the file path and metadata)
- N Chunk nodes (one per text chunk, storing the chunk text and index)
- N PART_OF edges (Document → each Chunk)
- N-1 NEXT_CHUNK edges (Chunk → next Chunk, preserving reading order)
IngestionPipeline._build_lexical_graph() in ingestion/pipeline.py
Step 4 — Extract Entities & Relationships
What it does: The most important step — an LLM reads each chunk and extracts structured knowledge: entities (people, places, organizations, etc.) and the relationships between them. How: The defaultGraphExtraction strategy uses a 2-step process:
-
Step 1 (NER): A pluggable entity extractor identifies entities in the text. Default: GLiNER (a local transformer model, no API calls needed). Alternative:
LLMExtractor(uses the LLM for NER). - Step 2 (Verify + Relationships): The LLM receives the pre-extracted entities and the original text. It verifies the entities (fixing errors, adding missed ones) and extracts all relationships between them.
GraphData containing nodes (entities), relationships, and mention records.
Code: ExtractionStrategy.extract() in ingestion/extraction_strategies/base.py
Step 4b — Quality Filter
What it does: Removes bad data that slipped through extraction — nodes with empty orNone IDs, and relationships whose endpoints don’t exist.
Why: LLMs sometimes produce malformed output (empty entity names, references to entities that weren’t extracted). This step catches those before they reach the graph.
Code: IngestionPipeline._filter_quality() in ingestion/pipeline.py
Step 5 — Prune Against Schema
What it does: Filters extracted data to only keep entities and relationships that match your schema definition. How it works:- If your schema defines entity types (e.g., Person, Organization, Location), only entities with those labels pass through
- If your schema defines relationship types, only those relationship types pass through
- Relationships whose endpoints were pruned are also removed
- Special cases:
"Unknown"entities (low-confidence NER) and"RELATES"edges (the unified relationship type) always pass through
GraphSchema()), this step is skipped entirely — everything passes through.
Code: IngestionPipeline._prune() in ingestion/pipeline.py
What was pruned.
IngestionResult.metadata["pruned"] says what this step removed, so you can see what the ontology misses: entities (undeclared label → count), relations (undeclared type → count), pattern_mismatches (type → "Source->Target" → count), dangling (relations whose endpoint was dropped) and unknown (entities kept as Unknown, which is how extraction stores a type outside the declared ones). Only non-empty keys appear; {} means nothing was pruned.
Step 6 — Resolve Duplicates
What it does: Merges entities that refer to the same real-world thing. When the LLM extracts “Alice” from chunk 1 and “Alice” from chunk 5, this step recognizes they’re the same entity and merges them. Default at ingest: ExactMatchResolution- Groups by
(normalized name, label)only — same name but different labels stay separate (e.g., Person “Paris” vs Location “Paris”); the judge decides those atfinalize() - Survivor keeps every member’s description joined with
" | "(no LLM summary); all relationship endpoints are re-pointed to the survivor before the write, so no edge is lost - No embeddings, no LLM calls; zero added cost
- Ingest is per document, so a resolver here can only compare one file’s entities with each other — the cross-document duplicates that matter are handled once, at
finalize()
- Runs the same exact-match pass first, then compares the surviving names by embedding similarity within each label
- Pairs at or above
hard_threshold(0.95) merge outright; pairs down tosoft_threshold(0.65) go to the LLM for a YES/NO verdict - Useful when the same entity is named inconsistently within a document (e.g., “Acme Corp” vs “Acme Corporation”)
finalize(): incremental cross-document dedup
- Ingest writes a new entity without a name vector and marks nothing on it: having no vector is what makes it pending. A table row a re-sync renames or folds into another node loses its vector, so it is checked under its new name
finalize()checks only the pending entities, against the whole graph through FalkorDB indexes — it never reads every entity. It lists each one in a system record, then embeds its name once (stored ase.embedding, the vector search also uses), and the__Entity__vector index returns its nearest names (no LLM)- Same spelling once normalised, or a unique acronym of it, as a same-label table row → merged into the row by rule, no LLM (an extracted entity under a guessed label can also be adopted by a declared label of the same name). Same spelling between extracted mentions only, an acronym that spells several long forms, and the 5 nearest names at cosine ≥ 0.5, become candidate pairs; candidates form dense sets of ≤ 8
- The judge —
LLMJudgeby default, or anyDedupJudge— partitions each set once into same-referent groups, reading names, labels, descriptions, existing aliases and declared attribute values, and may name each merged group - Same → merge, by the same rules as every finalize merge: a table’s row survives a mention, every member’s description is kept in
descriptions(joined with" | "indescription; after dedup,finalize()summarises any entity holding more than 5 into one), other names kept inaliases, every edge moved in the same query that deletes the loser; the survivor gains every member’s label and takes the judge’s name for the group (a table row keeps its declared name). Groups chain: A = B and B = C become one node - Different → nothing is written; the pair is judged again only if a later new entity brings it into a set
- Every merge is decided before any is written, sealed in a merge plan, and applied one atomic query per merge — an interrupted
finalize()is finished by the next one - An entity’s check is cleared once it completes — except one that did not (a failed lookup or judge call, an answer that was not a clean partition, a group held for one, a merge an
update()changed), which stays pending; after 3 judged runs without a usable verdict it is reported indedup_unresolved— sofinalize()with nothing pending does no dedup work. Runfinalize(dedup_all=True)once to check every entity of a graph whose entity vectors were written outsidefinalize() - Two keyed rows of a table, and a mention two such rows could own, are never asked or grouped
- Every run has a judge:
finalize(judge=LLMJudge(other_llm))picks the model (gpt-4.1-class recommended),finalize(judge=MyJudge())plugs in your own; passresolver=LLMVerifiedResolution(...)toingest()to also resolve per document
ResolutionStrategy.resolve() in ingestion/resolution_strategies/base.py
Step 7 — Write to Graph
What it does: Persists all the extracted and resolved data into FalkorDB using batched Cypher queries. How:- Nodes are written via
UNWIND $batch AS item MERGE (n:Label {id: item.id}) SET n += item.properties - Relationships are written similarly, using label hints for efficient MATCH operations
- Batch size: 500 items per query
- Entity nodes automatically get the
__Entity__secondary label (structural nodes like Chunk and Document do not)
GraphStore.upsert_nodes() and GraphStore.upsert_relationships() in storage/graph_store.py
Steps 8 & 9 — Mentions + Index Chunks (Parallel)
These two steps run simultaneously since they’re independent:Step 8 — Write Mentions
What it does: CreatesMENTIONED_IN edges linking every entity to every chunk it was extracted from. These edges are critical for retrieval — they let the system find text passages for any entity.
Details: Uncapped — every entity-chunk pair gets an edge. Duplicates are deduplicated by (entity_id, chunk_id).
Step 9 — Index Chunks
What it does: Embeds each chunk’s text into a vector and stores it on the Chunk node. These embeddings power the vector similarity search during retrieval. How:- Batch-embed all chunk texts in one API call (
aembed_documents) - Write vectors to Chunk nodes via
SET c.embedding = vecf32(vector) - Falls back to sequential embedding if the batch call fails
- Mentions:
IngestionPipeline._write_mentions()iningestion/pipeline.py - Chunk indexing:
VectorStore.index_chunks()instorage/vector_store.py
Post-Ingestion: finalize()
After all documents are ingested, callfinalize() to prepare the graph for querying. This is a separate step because some operations (like deduplication) work best when run globally across all documents, not per-document.
finalize() runs 4 steps in order:
finalize() after each document — call it once after all ingestion is complete: one run judges the whole batch’s new entities together, in fewer judge calls than a run per document.
Agentic Strategy Selection (auto=True)
By default ingest() uses fixed strategies (see Configuration Quick Reference). Pass auto=True to let a planner inspect a sample of each document and pick the chunker, entity-extraction backend, and resolver per document — the ingestion-side analogue of retrieval-side routing.
With
auto=True and no plan_limits, the default planner may pick only the cheap options, CHEAP_PLAN_LIMITS: sentence, fixed or structural chunking, gliner extraction and exact resolution, with at most 1,000 chunks per document. Pass plan_limits (the LLMIngestionPlanner limits: chunkers, extractors, resolvers, max_llm_pairs, tune_params, max_chunks) to allow more. It is merged over CHEAP_PLAN_LIMITS: a key you leave out keeps its cheap value, so max_chunks stays 1,000 and resolution stays exact unless you set them. A resolvers that allows llm_verified without a max_llm_pairs gets the resolver’s default budget (500), not the cheap 0. plan_limits and planner cannot be passed together: a planner you build carries its own limits.
The planner may also tune parameters inside the chosen strategy (e.g. max_tokens, the llm extractor’s threshold; GLiNER’s threshold is model-specific and is not tuned); all values are coerced and clamped to safe ranges, so the model can never produce an unsafe config. The ranges also bound cost: every chunk costs extraction LLM calls, so chunk sizes have floors (256 tokens for the sentence-based chunkers, with at most 4 sentences of overlap; 500 characters for fixed, with overlap at most a quarter of the size; a tuned sentence-based window repeats at most half its tokens, so it advances by at least half), and llm_verified’s hard_threshold stays at 0.8 or above, so candidate pairs between unified_threshold (0.65 by default) and hard_threshold are still checked by the LLM (pairs below unified_threshold are not candidates at all).
structural needs the document’s structure: .md files carry it, and Markdown-looking text (headings and lists in a .txt, or text mode) is parsed for it when the planner picks structural. Anything else falls back to sentence, and a note says so.
Safety guarantees:
- Explicit wins — any
chunker/extractor/resolveryou pass is kept as-is; the planner only fills the ones left unset. - Never silently broken — an empty/invalid plan or any planner error falls back to the default strategies (
sentence/gliner/exact), reproducing today’s behavior. The exceptions: aplannerthat is not an instance with an asyncplan()method raisesTypeErrorbefore anything is ingested, and an exhausted latency budget (LatencyBudgetExceededError, also when the planner call itself outlivesctx’s budget) is raised like everywhere else in the SDK. - Opt-in —
auto=Falseis the default; nothing changes unless you enable it. - Bounded blast radius on malicious content — the
LLMIngestionPlannerprompt includes a sample of the document itself, so adversarial document content could in principle try to steer the planner’s choice. The sample is quoted in a<document>block the prompt marks as data, with instructions inside it to be ignored, but the worst case is still one of the options you allow (cheap by default, below) with clamped parameters (seePARAM_SPECS) — never an arbitrary strategy or an out-of-range value.
LLMIngestionPlanner you build yourself may pick any option above unless you limit it, and the llm_verified pair budget it sets is capped at 500 (the resolver’s default; RuleBasedIngestionPlanner caps it at 500 too). To keep document text from choosing the strategies that add LLM calls, allow only the cheap options, pass tune_params=False so chunk sizes stay at their defaults, and set max_chunks to cap the chunks per document, which bounds its extraction calls (ingest(auto=True) does the first and last of these for you):
- The planner call.
LLMIngestionPlannermakes one LLM call per document;HeuristicIngestionPlannerandRuleBasedIngestionPlannermake none. - Extraction. Each chunk costs one extraction LLM call with
glinerand two with thellmbackend (entities, then verification and relations), before retries, so the chunk count sets the cost.max_chunksis a chunk limit, not a request ceiling:max_chunks=Nmeans at mostNextraction requests withglinerand about2Nwithllm, before retries. It refuses a document that splits into more thanNchunks before anything of it is written or extracted (ChunkBudgetExceededError, exported fromgraphrag_sdk, aChunkingError; in a batch, that file fails and the others go on). It wraps whatever chunker is used — the planner’s, yours or the default — andIngestionResult.metadata["plan"]["max_chunks"]records it. Forcontextualchunking the base chunks are counted before the per-chunk context calls, so a refused document costs none of them either. The default,None, sets no limit (ingest(auto=True)sets 1,000). - Resolution. A choice outside the allowed options falls back to the default (or the first allowed option) and is never built; the prompt only lists the allowed options.
max_llm_pairs=caps thellm_verifiedpair budget, also withtune_params=False, andmax_llm_pairs=0disallowsllm_verified. The cap applies per document: a plan’sLLMVerifiedResolutionruns its default unified stage, which compares every entity in one group and verifies up to that many candidate pairs, whatever their types. On top of it come a second ask for each confirmed match between two entity types, one check per name shared by entities of different types, and the description-summary calls. The separate 200-pair cross-label pass (cross_label_max_pairs) runs only withunified_stage=False, which plans do not set. Withresolvers=("exact",)resolution makes no LLM calls.
finalize() (embedding, and the LLM judge it uses for deduplication) or the questions you ask afterwards.
Three planners ship in the SDK:
RuleBasedIngestionPlanner is what the agent tools use for ingest(files=..., plan=[...]) (see Agentic retrieval). It takes the same chunkers / extractors / resolvers / max_llm_pairs / max_chunks limits; CHEAP_PLAN_LIMITS (exported from graphrag_sdk, read-only) holds the cheap ones, with max_chunks=1000 (about 380,000 tokens at the default chunk size).
What was chosen. With auto=True, each IngestionResult records the decision in metadata["plan"]:
slots says where each strategy came from: "caller" (you passed it), "planner" (built from the returned plan) or "default" (the planner raised or returned no plan, or that strategy could not be built — the other slots still follow the plan). LLMIngestionPlanner does not raise when its LLM call fails or its reply is not a plan: it returns the default plan, so the slots read "planner" and plan.reason starts with default:. Re-ingesting unchanged content skips the planner as well (a note says so). notes explains any fallback or adjustment, e.g. structural becoming sentence when the document has no structural elements (text mode). Planner failures are also logged at WARNING. Structured sources (.csv, .tsv, …) never run the planner: no chunker, extractor or resolver applies to them. A document that cannot be loaded is not planned either; its load error is raised as a LoaderError (a loader’s own IngestionError as is), so nothing was written.
Limitation: update() does not re-plan. The chosen strategies are recorded only in the ingest result’s metadata["plan"]; they are not stored on the Document. update() has no auto and re-ingests changed content with the strategies you pass (or the defaults), so a document ingested as structural + llm is re-chunked with sentence + gliner unless you pass those strategies to update() yourself. The update result’s metadata["plan_note"] says which strategies were defaulted, and after a planned ingest on the same GraphRAG it is also logged at WARNING.
Code: ingestion_planner.py · wired into GraphRAG.ingest() in api/main.py.
Configuration Quick Reference
Chunking
Extraction
Resolution
Performance Notes
- Slowest step: Extraction (Step 4) — involves LLM calls for every chunk. Expect ~2-5 seconds per chunk.
- Fastest step: Quality filter, prune, and resolve — all in-memory, sub-second.
- Parallelism: Steps 8-9 run in parallel. Step 1 NER uses a semaphore (default 12 concurrent calls).
- Batch size: The benchmark uses 1500-character chunks. 20 documents (~4.7 MB total) take ~47 minutes to ingest.