Ingestion

When you call rag.ingest("document.txt"), the SDK transforms your raw text into a structured knowledge graph through a 9-step sequential pipeline. Think of it as an assembly line: each step takes the output of the previous one, refines it, and passes it forward. This document explains what each step does, why it exists, and how to tune it.

The Big Picture

                         Your Document
                              |
                    ┌─────────┴─────────┐
                    v                    v
              1. Load Text         (or pass text= directly)
                    |
              2. Chunk Text
                    |
              3. Build Lexical Graph
              (Document + Chunk nodes, provenance edges)
                    |
              4. Extract Entities & Relationships
                    |
              4b. Quality Filter
              (remove bad nodes/dangling edges)
                    |
              5. Prune Against Schema
              (keep only schema-conforming data)
                    |
              6. Resolve Duplicates
              (merge same-entity mentions)
                    |
              7. Write to Graph
              (batched MERGE into FalkorDB)
                    |
         ┌─────────┴─────────┐
         v                    v
   8. Write Mentions    9. Index Chunks
   (MENTIONED_IN edges)  (embed + store vectors)
         └─────────┬─────────┘
                   v
            Done — IngestionResult

Steps 1-7 run sequentially (each depends on the previous). Steps 8-9 run in parallel since they’re independent.

Step-by-Step Explanation

Step 1 — Load

What it does: Reads raw text from a file, URL, or string. How: The LoaderStrategy ABC handles this. The SDK auto-detects the loader based on file extension:

.pdf files use PdfLoader
Everything else uses TextLoader
If you pass text= directly, the loader step is skipped entirely

Output: DocumentOutput containing the raw text and a DocumentInfo with a unique ID and file path. Code: LoaderStrategy.load() in ingestion/loaders/base.py

Step 2 — Chunk

What it does: Splits the document text into smaller overlapping windows called chunks. Each chunk is small enough for the LLM to process, but large enough to contain meaningful context. How: The default FixedSizeChunking uses a sliding window:

Window size: 1000 characters (configurable)
Overlap: 100 characters between consecutive chunks
Step size: chunk_size - chunk_overlap = 900 characters

Why overlap? Without overlap, an entity mentioned right at the boundary between two chunks might be split across them and lost. Overlap ensures entities near boundaries appear in at least one complete chunk. Output: TextChunks — a list of TextChunk objects, each with a unique ID (uid), the text content, and an index number. Code: ChunkingStrategy.chunk() in ingestion/chunking_strategies/base.py

Step 3 — Build Lexical Graph (Mandatory)

What it does: Creates the provenance backbone of the knowledge graph — this is how every answer traces back to its source document. Creates:

1 Document node (with the file path and metadata)
N Chunk nodes (one per text chunk, storing the chunk text and index)
N PART_OF edges (Document → each Chunk)
N-1 NEXT_CHUNK edges (Chunk → next Chunk, preserving reading order)

The result looks like:

Document
├── PART_OF → Chunk 0
├── PART_OF → Chunk 1
└── PART_OF → Chunk 2

Chunk 0 ──NEXT_CHUNK──> Chunk 1 ──NEXT_CHUNK──> Chunk 2

Why mandatory? This is the Zero-Loss Data principle — every piece of source material is traceable in the graph. When the retrieval system finds a chunk, it can always trace back to the source document. Code: IngestionPipeline._build_lexical_graph() in ingestion/pipeline.py

Step 4 — Extract Entities & Relationships

What it does: The most important step — an LLM reads each chunk and extracts structured knowledge: entities (people, places, organizations, etc.) and the relationships between them. How: The default GraphExtraction strategy uses a 2-step process:

Step 1 (NER): A pluggable entity extractor identifies entities in the text. Default: GLiNER (a local transformer model, no API calls needed). Alternative: LLMExtractor (uses the LLM for NER).
Step 2 (Verify + Relationships): The LLM receives the pre-extracted entities and the original text. It verifies the entities (fixing errors, adding missed ones) and extracts all relationships between them.

For a detailed explanation of the extraction process, see Extraction. Output: GraphData containing nodes (entities), relationships, and mention records. Code: ExtractionStrategy.extract() in ingestion/extraction_strategies/base.py

Step 4b — Quality Filter

What it does: Removes bad data that slipped through extraction — nodes with empty or None IDs, and relationships whose endpoints don’t exist. Why: LLMs sometimes produce malformed output (empty entity names, references to entities that weren’t extracted). This step catches those before they reach the graph. Code: IngestionPipeline._filter_quality() in ingestion/pipeline.py

Step 5 — Prune Against Schema

What it does: Filters extracted data to only keep entities and relationships that match your schema definition. How it works:

If your schema defines entity types (e.g., Person, Organization, Location), only entities with those labels pass through
If your schema defines relationship types, only those relationship types pass through
Relationships whose endpoints were pruned are also removed
Special cases: "Unknown" entities (low-confidence NER) and "RELATES" edges (the unified relationship type) always pass through

Open schema mode: If you define no entity or relationship types (empty GraphSchema()), this step is skipped entirely — everything passes through. Code: IngestionPipeline._prune() in ingestion/pipeline.py

Step 6 — Resolve Duplicates

What it does: Merges entities that refer to the same real-world thing. When the LLM extracts “Alice” from chunk 1 and “Alice” from chunk 5, this step recognizes they’re the same entity and merges them. Default: ExactMatchResolution

Groups entities by ID
Keeps the first occurrence as the survivor
Merges properties from duplicates into the survivor
Remaps all relationship endpoints to the survivor
Deduplicates relationships by (start_id, type, end_id)

Alternative: DescriptionMergeResolution

Groups by (normalized name, label) — same name but different labels stay separate (e.g., Person “Paris” vs Location “Paris”)
Merges descriptions (concatenation or LLM summarization)
Useful when the same entity is described differently across documents

For details on resolution strategies, see Strategies. Code: ResolutionStrategy.resolve() in ingestion/resolution_strategies/base.py

Step 7 — Write to Graph

What it does: Persists all the extracted and resolved data into FalkorDB using batched Cypher queries. How:

Nodes are written via UNWIND $batch AS item MERGE (n:Label {id: item.id}) SET n += item.properties
Relationships are written similarly, using label hints for efficient MATCH operations
Batch size: 500 items per query
Entity nodes automatically get the __Entity__ secondary label (structural nodes like Chunk and Document do not)

For details on the storage layer, see Storage. Code: GraphStore.upsert_nodes() and GraphStore.upsert_relationships() in storage/graph_store.py

Steps 8 & 9 — Mentions + Index Chunks (Parallel)

These two steps run simultaneously since they’re independent:

Step 8 — Write Mentions

What it does: Creates MENTIONED_IN edges linking every entity to every chunk it was extracted from. These edges are critical for retrieval — they let the system find text passages for any entity. Details: Uncapped — every entity-chunk pair gets an edge. Duplicates are deduplicated by (entity_id, chunk_id).

Step 9 — Index Chunks

What it does: Embeds each chunk’s text into a vector and stores it on the Chunk node. These embeddings power the vector similarity search during retrieval. How:

Batch-embed all chunk texts in one API call (aembed_documents)
Write vectors to Chunk nodes via SET c.embedding = vecf32(vector)
Falls back to sequential embedding if the batch call fails

Code:

Mentions: IngestionPipeline._write_mentions() in ingestion/pipeline.py
Chunk indexing: VectorStore.index_chunks() in storage/vector_store.py

Post-Ingestion: finalize()

After all documents are ingested, call finalize() to prepare the graph for querying. This is a separate step because some operations (like deduplication) work best when run globally across all documents, not per-document. finalize() runs 5 steps in order:

Step	What it does	Why
1. NULL cleanup	Removes entities with `NULL` names	Legacy edge cases from MERGE operations
2. Deduplicate	Global exact-name entity dedup	”Alice” from doc 1 and doc 5 become one node
3. Entity embeddings	Embeds entity names into vectors	Powers entity vector search during retrieval
4. Relationship embeddings	Embeds fact text on RELATES edges	Powers RELATES edge vector search during retrieval
5. Ensure indexes	Creates all 5 standard indexes	Vector + fulltext indexes for search

# After ingesting ALL documents:
stats = await rag.finalize()
# Returns: {
#     "null_stubs_removed": 0,
#     "entities_deduplicated": 142,
#     "entities_embedded": 3200,
#     "relationships_embedded": 8500,
#     "indexes": { "vector_Chunk": True, "vector___Entity__": True, ... }
# }

Important: Do not call finalize() after each document — call it once after all ingestion is complete. Entity backfill re-scans all entities and is slow when called repeatedly.

Configuration Quick Reference

Chunking

Parameter	Default	Benchmark Value	Effect
`chunk_size`	1000	1500	Larger = richer extraction context, more LLM tokens
`chunk_overlap`	100	200	Larger = fewer boundary losses, more redundancy

Extraction

Parameter	Default	Effect
`entity_extractor`	`GLiNERExtractor()`	Local NER model (fast, no API calls)
`coref_resolver`	`None`	Optional coreference resolution (resolves pronouns)
`entity_types`	11 defaults	Custom ontology for your domain
`max_concurrency`	LLM default (12)	Parallel LLM calls during extraction

Resolution

Strategy	When to use
`ExactMatchResolution` (default)	Fast, deterministic. Good when extraction is consistent
`DescriptionMergeResolution`	Multi-document ingestion. Used in the benchmark pipeline

Performance Notes

Slowest step: Extraction (Step 4) — involves LLM calls for every chunk. Expect ~2-5 seconds per chunk.
Fastest step: Quality filter, prune, and resolve — all in-memory, sub-second.
Parallelism: Steps 8-9 run in parallel. Step 1 NER uses a semaphore (default 12 concurrent calls).
Batch size: The benchmark uses 1500-character chunks. 20 documents (~4.7 MB total) take ~47 minutes to ingest.

File Reference

File	What it contains
`pipeline.py`	The 9-step pipeline orchestrator
`api/main.py`	`GraphRAG.ingest()` and `finalize()` — user-facing API
`loaders/text_loader.py`	TextLoader implementation
`loaders/pdf_loader.py`	PdfLoader implementation
`chunking_strategies/fixed_size.py`	FixedSizeChunking implementation
`extraction_strategies/graph_extraction.py`	2-step extraction (NER + LLM verify/rels)
`resolution_strategies/exact_match.py`	ExactMatchResolution
`resolution_strategies/description_merge.py`	DescriptionMergeResolution
`storage/graph_store.py`	Batched node/relationship writes
`storage/vector_store.py`	Chunk embedding + indexing

Home

Pipeline

Ontology

Reference

The Big Picture

Step-by-Step Explanation

Step 1 — Load

Step 2 — Chunk

Step 3 — Build Lexical Graph (Mandatory)

Step 4 — Extract Entities & Relationships

Step 4b — Quality Filter

Step 5 — Prune Against Schema

Step 6 — Resolve Duplicates

Step 7 — Write to Graph

Steps 8 & 9 — Mentions + Index Chunks (Parallel)

Step 8 — Write Mentions

Step 9 — Index Chunks

Post-Ingestion: finalize()

Configuration Quick Reference

Chunking

Extraction

Resolution

Performance Notes

File Reference

​The Big Picture

​Step-by-Step Explanation

​Step 1 — Load

​Step 2 — Chunk

​Step 3 — Build Lexical Graph (Mandatory)

​Step 4 — Extract Entities & Relationships

​Step 4b — Quality Filter

​Step 5 — Prune Against Schema

​Step 6 — Resolve Duplicates

​Step 7 — Write to Graph

​Steps 8 & 9 — Mentions + Index Chunks (Parallel)

​Step 8 — Write Mentions

​Step 9 — Index Chunks

​Post-Ingestion: finalize()

​Configuration Quick Reference

​Chunking

​Extraction

​Resolution

​Performance Notes

​File Reference

The Big Picture

Step-by-Step Explanation

Step 1 — Load

Step 2 — Chunk

Step 3 — Build Lexical Graph (Mandatory)

Step 4 — Extract Entities & Relationships

Step 4b — Quality Filter

Step 5 — Prune Against Schema

Step 6 — Resolve Duplicates

Step 7 — Write to Graph

Steps 8 & 9 — Mentions + Index Chunks (Parallel)

Step 8 — Write Mentions

Step 9 — Index Chunks

Post-Ingestion: finalize()

Configuration Quick Reference

Chunking

Extraction

Resolution

Performance Notes

File Reference