PlanOpticon

planopticon / docs / architecture / pipeline.md
Source Rendered
Blame History Raw 336 lines
1
# Processing Pipeline
2
3
PlanOpticon has four main pipelines: **video analysis**, **document ingestion**, **source connector**, and **export**. Each pipeline can operate independently, and they connect through the shared knowledge graph.
4
5
---
6
7
## Single video pipeline
8
9
The core video analysis pipeline processes a single video file through eight sequential steps with checkpoint/resume support.
10
11
```mermaid
12
sequenceDiagram
13
participant CLI
14
participant Pipeline
15
participant FrameExtractor
16
participant AudioExtractor
17
participant Provider
18
participant DiagramAnalyzer
19
participant KnowledgeGraph
20
participant Exporter
21
22
CLI->>Pipeline: process_single_video()
23
24
Note over Pipeline: Step 1: Extract frames
25
Pipeline->>FrameExtractor: extract_frames()
26
Note over FrameExtractor: Change detection + periodic capture (every 30s)
27
FrameExtractor-->>Pipeline: frame_paths[]
28
29
Note over Pipeline: Step 2: Filter people frames
30
Pipeline->>Pipeline: filter_people_frames()
31
Note over Pipeline: OpenCV face detection removes webcam/people frames
32
33
Note over Pipeline: Step 3: Extract + transcribe audio
34
Pipeline->>AudioExtractor: extract_audio()
35
Pipeline->>Provider: transcribe_audio()
36
Note over Provider: Supports speaker hints via --speakers flag
37
38
Note over Pipeline: Step 4: Analyze visuals
39
Pipeline->>DiagramAnalyzer: process_frames()
40
loop Each frame (up to 10 standard / 20 comprehensive)
41
DiagramAnalyzer->>Provider: classify (vision)
42
alt High confidence diagram
43
DiagramAnalyzer->>Provider: full analysis
44
Note over Provider: Extract description, text, mermaid, chart data
45
else Medium confidence
46
DiagramAnalyzer-->>Pipeline: screengrab fallback
47
end
48
end
49
50
Note over Pipeline: Step 5: Build knowledge graph
51
Pipeline->>KnowledgeGraph: register_source()
52
Pipeline->>KnowledgeGraph: process_transcript()
53
Pipeline->>KnowledgeGraph: process_diagrams()
54
Note over KnowledgeGraph: Writes knowledge_graph.db (SQLite) + .json
55
56
Note over Pipeline: Step 6: Extract key points + action items
57
Pipeline->>Provider: extract key points
58
Pipeline->>Provider: extract action items
59
60
Note over Pipeline: Step 7: Generate report
61
Pipeline->>Pipeline: generate markdown report
62
Note over Pipeline: Includes mermaid diagrams, tables, cross-references
63
64
Note over Pipeline: Step 8: Export formats
65
Pipeline->>Exporter: export_all_formats()
66
Note over Exporter: HTML report, PDF, SVG/PNG renderings, chart reproductions
67
68
Pipeline-->>CLI: VideoManifest
69
```
70
71
### Pipeline steps in detail
72
73
| Step | Name | Checkpointable | Description |
74
|------|------|----------------|-------------|
75
| 1 | Extract frames | Yes | Change detection + periodic capture. Skipped if `frames/frame_*.jpg` exist on disk. |
76
| 2 | Filter people frames | No | Inline with step 1. OpenCV face detection removes webcam frames. |
77
| 3 | Extract + transcribe audio | Yes | Skipped if `transcript/transcript.json` exists. Speaker hints passed if `--speakers` provided. |
78
| 4 | Analyze visuals | Yes | Skipped if `diagrams/` is populated. Evenly samples frames (not just first N). |
79
| 5 | Build knowledge graph | Yes | Skipped if `results/knowledge_graph.db` exists. Registers source, processes transcript and diagrams. |
80
| 6 | Extract key points + actions | Yes | Skipped if `results/key_points.json` and `results/action_items.json` exist. |
81
| 7 | Generate report | Yes | Skipped if `results/analysis.md` exists. |
82
| 8 | Export formats | No | Always runs. Renders mermaid to SVG/PNG, reproduces charts, generates HTML/PDF. |
83
84
---
85
86
## Batch pipeline
87
88
The batch pipeline wraps the single-video pipeline and adds cross-video knowledge graph merging.
89
90
```mermaid
91
flowchart TD
92
A[Scan input directory] --> B[Match video files by pattern]
93
B --> C{For each video}
94
C --> D[process_single_video]
95
D --> E{Success?}
96
E -->|Yes| F[Collect manifest + KG]
97
E -->|No| G[Log error, continue]
98
F --> H[Next video]
99
G --> H
100
H --> C
101
C -->|All done| I[Merge knowledge graphs]
102
I --> J[Fuzzy matching + conflict resolution]
103
J --> K[Generate batch summary]
104
K --> L[Write batch manifest]
105
L --> M[batch_manifest.json + batch_summary.md + merged KG]
106
```
107
108
### Knowledge graph merge strategy
109
110
During batch merging, `KnowledgeGraph.merge()` applies:
111
112
1. **Case-insensitive exact matching** for entity names
113
2. **Fuzzy matching** via `SequenceMatcher` (threshold >= 0.85) for near-duplicates
114
3. **Type conflict resolution** using a specificity ranking (e.g., `technology` > `concept`)
115
4. **Description union** across all sources
116
5. **Relationship deduplication** by (source, target, type) tuple
117
118
---
119
120
## Document ingestion pipeline
121
122
The document ingestion pipeline processes files (Markdown, plaintext, PDF) into knowledge graphs without video analysis.
123
124
```mermaid
125
flowchart TD
126
A[Input: file or directory] --> B{File or directory?}
127
B -->|File| C[get_processor by extension]
128
B -->|Directory| D[Glob for supported extensions]
129
D --> E{Recursive?}
130
E -->|Yes| F[rglob all files]
131
E -->|No| G[glob top-level only]
132
F --> H[For each file]
133
G --> H
134
H --> C
135
C --> I[DocumentProcessor.process]
136
I --> J[DocumentChunk list]
137
J --> K[Register source in KG]
138
K --> L[Add chunks as content]
139
L --> M[KG extracts entities + relationships]
140
M --> N[knowledge_graph.db]
141
```
142
143
### Supported document types
144
145
| Extension | Processor | Notes |
146
|-----------|-----------|-------|
147
| `.md` | `MarkdownProcessor` | Splits by headings into sections |
148
| `.txt` | `PlaintextProcessor` | Splits into fixed-size chunks |
149
| `.pdf` | `PdfProcessor` | Requires `pymupdf` or `pdfplumber`. Falls back gracefully between libraries. |
150
151
### Adding documents to an existing graph
152
153
The `--db-path` flag lets you ingest documents into an existing knowledge graph:
154
155
```bash
156
planopticon ingest spec.md --db-path existing.db
157
planopticon ingest ./docs/ -o ./output --recursive
158
```
159
160
---
161
162
## Source connector pipeline
163
164
Source connectors fetch content from cloud services, note-taking apps, and web sources. Each source implements the `BaseSource` ABC with three methods: `authenticate()`, `list_videos()`, and `download()`.
165
166
```mermaid
167
flowchart TD
168
A[Source command] --> B[Authenticate with provider]
169
B --> C{Auth success?}
170
C -->|No| D[Error: check credentials]
171
C -->|Yes| E[List files in folder]
172
E --> F[Filter by pattern / type]
173
F --> G[Download to local path]
174
G --> H{Analyze or ingest?}
175
H -->|Video| I[process_single_video / batch]
176
H -->|Document| J[ingest_file / ingest_directory]
177
I --> K[Knowledge graph]
178
J --> K
179
```
180
181
### Available sources
182
183
PlanOpticon includes connectors for:
184
185
| Category | Sources |
186
|----------|---------|
187
| Cloud storage | Google Drive, S3, Dropbox |
188
| Meeting recordings | Zoom, Google Meet, Microsoft Teams |
189
| Productivity suites | Google Workspace (Docs/Sheets/Slides), Microsoft 365 (SharePoint/OneDrive/OneNote) |
190
| Note-taking apps | Obsidian, Logseq, Apple Notes, Google Keep, Notion |
191
| Web sources | YouTube, Web (URL), RSS, Podcasts |
192
| Developer platforms | GitHub, arXiv |
193
| Social media | Reddit, Twitter/X, Hacker News |
194
195
Each source authenticates via environment variables (API keys, OAuth tokens) specific to the provider.
196
197
---
198
199
## Planning agent pipeline
200
201
The planning agent consumes a knowledge graph and uses registered skills to generate planning artifacts.
202
203
```mermaid
204
flowchart TD
205
A[Knowledge graph] --> B[Load into AgentContext]
206
B --> C[GraphQueryEngine]
207
C --> D[Taxonomy classification]
208
D --> E[Agent orchestrator]
209
E --> F{Select skill}
210
F --> G[ProjectPlan skill]
211
F --> H[PRD skill]
212
F --> I[Roadmap skill]
213
F --> J[TaskBreakdown skill]
214
F --> K[DocGenerator skill]
215
F --> L[WikiGenerator skill]
216
F --> M[NotesExport skill]
217
F --> N[ArtifactExport skill]
218
F --> O[GitHubIntegration skill]
219
F --> P[RequirementsChat skill]
220
G --> Q[Artifact output]
221
H --> Q
222
I --> Q
223
J --> Q
224
K --> Q
225
L --> Q
226
M --> Q
227
N --> Q
228
O --> Q
229
P --> Q
230
Q --> R[Write to disk / push to service]
231
```
232
233
### Skill execution flow
234
235
1. The `AgentContext` is populated with the knowledge graph, query engine, provider manager, and any planning entities from taxonomy classification
236
2. Each `Skill` checks `can_execute()` against the context (requires at minimum a knowledge graph and provider manager)
237
3. The skill's `execute()` method generates an `Artifact` with a name, content, type, and format
238
4. Artifacts are collected and can be exported to disk or pushed to external services (GitHub issues, wiki pages, etc.)
239
240
---
241
242
## Export pipeline
243
244
The export pipeline converts knowledge graphs and analysis artifacts into various output formats.
245
246
```mermaid
247
flowchart TD
248
A[knowledge_graph.db] --> B{Export command}
249
B --> C[export markdown]
250
B --> D[export obsidian]
251
B --> E[export notion]
252
B --> F[export exchange]
253
B --> G[wiki generate]
254
B --> H[kg convert]
255
C --> I[7 document types + entity briefs + CSV]
256
D --> J[Obsidian vault with frontmatter + wiki-links]
257
E --> K[Notion-compatible markdown + CSV database]
258
F --> L[PlanOpticonExchange JSON payload]
259
G --> M[GitHub wiki pages + sidebar + home]
260
H --> N[Convert between .db / .json / .graphml / .csv]
261
```
262
263
All export commands accept a `knowledge_graph.db` (or `.json`) path as input. No API key is required for template-based exports (markdown, obsidian, notion, wiki, exchange, convert). Only the planning agent skills that generate new content require a provider.
264
265
---
266
267
## How pipelines connect
268
269
```mermaid
270
flowchart LR
271
V[Video files] --> VP[Video Pipeline]
272
D[Documents] --> DI[Document Ingestion]
273
S[Cloud Sources] --> SC[Source Connectors]
274
SC --> V
275
SC --> D
276
VP --> KG[(knowledge_graph.db)]
277
DI --> KG
278
KG --> QE[Query Engine]
279
KG --> EP[Export Pipeline]
280
KG --> PA[Planning Agent]
281
PA --> AR[Artifacts]
282
AR --> EP
283
```
284
285
All pipelines converge on the knowledge graph as the central data store. The knowledge graph is the shared interface between ingestion (video or document), querying, exporting, and planning.
286
287
---
288
289
## Error handling
290
291
Error handling follows consistent patterns across all pipelines:
292
293
| Scenario | Behavior |
294
|----------|----------|
295
| Video fails in batch | Batch continues. Failed video recorded in manifest with error details. |
296
| Diagram analysis fails | Falls back to screengrab (captioned screenshot). |
297
| LLM extraction fails | Returns empty results gracefully. Key points and action items will be empty arrays. |
298
| Document processor not found | Raises `ValueError` with list of supported extensions. |
299
| Source authentication fails | Returns `False` from `authenticate()`. CLI prints error message. |
300
| Checkpoint file found | Step is skipped entirely and results are loaded from disk. |
301
| Progress callback fails | Warning logged. Pipeline continues without progress updates. |
302
303
---
304
305
## Progress callback system
306
307
The pipeline supports a `ProgressCallback` protocol for real-time progress tracking. This is used by the CLI's progress bars and can be implemented by external integrations (web UIs, CI systems, etc.).
308
309
```python
310
from video_processor.models import ProgressCallback
311
312
class MyCallback:
313
def on_step_start(self, step: str, index: int, total: int) -> None:
314
print(f"Starting step {index}/{total}: {step}")
315
316
def on_step_complete(self, step: str, index: int, total: int) -> None:
317
print(f"Completed step {index}/{total}: {step}")
318
319
def on_progress(self, step: str, percent: float, message: str = "") -> None:
320
print(f" {step}: {percent:.0%} {message}")
321
```
322
323
Pass the callback to `process_single_video()`:
324
325
```python
326
from video_processor.pipeline import process_single_video
327
328
manifest = process_single_video(
329
input_path="recording.mp4",
330
output_dir="./output",
331
progress_callback=MyCallback(),
332
)
333
```
334
335
The callback methods are called within a try/except wrapper, so a failing callback never interrupts the pipeline. If a callback method raises an exception, a warning is logged and processing continues.
336

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button