Document ingestion lets you process files -- PDFs, Markdown, and plaintext -- into a knowledge graph. PlanOpticon extracts text from documents, chunks it into manageable pieces, runs LLM-powered entity and relationship extraction, and stores the results in a FalkorDB knowledge graph. This is the same knowledge graph format produced by video analysis, so you can combine video and document insights in a single graph.
4
5
## Supported formats
6
7
| Extension | Processor | Description |
8
|-----------|-----------|-------------|
9
| `.pdf` | `PdfProcessor` | Extracts text page by page using pymupdf or pdfplumber |
10
| `.md`, `.markdown` | `MarkdownProcessor` | Splits on headings into sections |
Additional formats can be added by implementing the `DocumentProcessor` base class and registering it (see [Extending with custom processors](#extending-with-custom-processors) below).
14
15
## CLI usage
16
17
### `planopticon ingest`
18
19
```
20
planopticon ingest INPUT_PATH [OPTIONS]
21
```
22
23
**Arguments:**
24
25
| Argument | Description |
26
|----------|-------------|
27
| `INPUT_PATH` | Path to a file or directory to ingest (must exist) |
28
29
**Options:**
30
31
| Option | Short | Default | Description |
32
|--------|-------|---------|-------------|
33
| `--output` | `-o` | Current directory | Output directory for the knowledge graph |
34
| `--db-path` | | None | Path to an existing `knowledge_graph.db` to merge into |
The ingested entities and relationships are merged with the existing graph. Duplicate entities are consolidated automatically by the knowledge graph engine.
86
87
### Choosing an LLM provider
88
89
Entity and relationship extraction requires an LLM. By default, PlanOpticon auto-detects available providers based on your environment variables. You can override this:
Both `.db` (SQLite/FalkorDB) and `.json` formats are saved automatically.
121
122
## How each processor works
123
124
### PDF processor
125
126
The `PdfProcessor` extracts text from PDF files on a per-page basis. It tries two extraction libraries in order:
127
128
1. **pymupdf** (preferred) -- Fast, reliable text extraction. Install with `pip install pymupdf`.
129
2. **pdfplumber** (fallback) -- Alternative extractor. Install with `pip install pdfplumber`.
130
131
If neither library is installed, the processor raises an `ImportError` with installation instructions.
132
133
Each page becomes a separate `DocumentChunk` with:
134
135
- `text`: The extracted text content of the page
136
- `page`: The 1-based page number
137
- `metadata.extraction_method`: Which library was used (`pymupdf` or `pdfplumber`)
138
139
To install PDF support:
140
141
```bash
142
pip install 'planopticon[pdf]'
143
# or
144
pip install pymupdf
145
# or
146
pip install pdfplumber
147
```
148
149
### Markdown processor
150
151
The `MarkdownProcessor` splits Markdown files on heading boundaries (lines starting with `#` through `######`). Each heading and its content until the next heading becomes a separate chunk.
152
153
**Splitting behavior:**
154
155
- If the file contains headings, each heading section becomes a chunk. The `section` field records the heading text.
156
- Content before the first heading is captured as a `(preamble)` chunk.
157
- If the file contains no headings, it falls back to paragraph-based chunking (same as plaintext).
158
159
For example, a file with this structure:
160
161
```markdown
162
Some intro text.
163
164
# Architecture
165
166
The system uses a microservices architecture...
167
168
## Components
169
170
There are three main components...
171
172
# Deployment
173
174
Deployment is handled via...
175
```
176
177
Produces four chunks: `(preamble)`, `Architecture`, `Components`, and `Deployment`.
178
179
### Plaintext processor
180
181
The `PlaintextProcessor` handles `.txt`, `.text`, `.log`, and `.csv` files. It splits text on paragraph boundaries (double newlines) and groups paragraphs into chunks with a configurable maximum size.
182
183
**Chunking parameters:**
184
185
| Parameter | Default | Description |
186
|-----------|---------|-------------|
187
| `max_chunk_size` | 2000 characters | Maximum size of each chunk |
188
| `overlap` | 200 characters | Number of characters from the end of one chunk to repeat at the start of the next |
189
190
The overlap ensures that entities or context that spans a paragraph boundary are not lost. Chunks are created by accumulating paragraphs until the next paragraph would exceed `max_chunk_size`, at which point the current chunk is flushed and a new one begins.
191
192
## The ingestion pipeline
193
194
Document ingestion follows this pipeline:
195
196
```
197
File on disk
198
|
199
v
200
Processor selection (by file extension)
201
|
202
v
203
Text extraction (PDF pages / Markdown sections / plaintext paragraphs)
204
|
205
v
206
DocumentChunk objects (text + metadata)
207
|
208
v
209
Source registration (provenance tracking in the KG)
210
|
211
v
212
KG content addition (LLM entity/relationship extraction per chunk)
213
|
214
v
215
Knowledge graph storage (.db + .json)
216
```
217
218
### Step 1: Processor selection
219
220
PlanOpticon maintains a registry of processors keyed by file extension. When you call `ingest_file()`, it looks up the appropriate processor using `get_processor(path)`. If no processor is registered for the file extension, a `ValueError` is raised.
221
222
### Step 2: Text extraction
223
224
The selected processor reads the file and produces a list of `DocumentChunk` objects. Each chunk contains:
225
226
| Field | Type | Description |
227
|-------|------|-------------|
228
| `text` | `str` | The extracted text content |
229
| `source_file` | `str` | Path to the source file |
230
| `chunk_index` | `int` | Sequential index of this chunk within the file |
Each ingested file is registered as a source in the knowledge graph with provenance metadata:
238
239
- `source_id`: A SHA-256 hash of the absolute file path (first 12 characters), unless you provide a custom ID
240
- `source_type`: Always `"document"`
241
- `title`: The file stem (filename without extension)
242
- `path`: The file path
243
- `mime_type`: Detected MIME type
244
- `ingested_at`: ISO-8601 timestamp
245
- `metadata`: Chunk count and file extension
246
247
### Step 4: Entity and relationship extraction
248
249
Each chunk's text is passed to `knowledge_graph.add_content()`, which uses the configured LLM provider to extract entities and relationships. The content source is tagged with the document name and either the page number or section name:
250
251
- `document:report.pdf:page:3`
252
- `document:spec.md:section:Architecture`
253
- `document:notes.txt` (no page or section)
254
255
### Step 5: Storage
256
257
The knowledge graph is saved in both `.db` (SQLite-backed FalkorDB) and `.json` formats.
258
259
## Combining with video analysis
260
261
A common workflow is to analyze a video recording and then ingest related documents into the same knowledge graph:
The resulting knowledge graph contains entities and relationships from all sources -- video transcripts, meeting agendas, specs, and reference documents -- with full provenance tracking so you can trace any entity back to its source.
281
282
## Python API
283
284
### Ingesting a single file
285
286
```python
287
from pathlib import Path
288
from video_processor.integrators.knowledge_graph import KnowledgeGraph
289
from video_processor.processors.ingest import ingest_file
290
291
kg = KnowledgeGraph(db_path=Path("knowledge_graph.db"))
292
chunk_count = ingest_file(Path("document.pdf"), kg)
293
print(f"Processed {chunk_count} chunks")
294
295
kg.save(Path("knowledge_graph.db"))
296
```
297
298
### Ingesting a directory
299
300
```python
301
from pathlib import Path
302
from video_processor.integrators.knowledge_graph import KnowledgeGraph
303
from video_processor.processors.ingest import ingest_directory
304
305
kg = KnowledgeGraph(db_path=Path("knowledge_graph.db"))
306
results = ingest_directory(
307
Path("./docs"),
308
kg,
309
recursive=True,
310
extensions=[".md", ".pdf"], # Optional: filter by extension
311
)
312
313
for filepath, chunks in results.items():
314
print(f" {filepath}: {chunks} chunks")
315
316
kg.save(Path("knowledge_graph.db"))
317
```
318
319
### Listing supported extensions
320
321
```python
322
from video_processor.processors.base import list_supported_extensions