Navegador

navegador / docs / guide / cluster.md

Source Rendered

Blame History Raw 281 lines

1	`# Cluster Mode`
2
3	`Cluster mode lets multiple machines share a single navegador graph over Redis. Use it when your team runs large ingestion jobs, want shared context across agents and CI, or need partitioned work processing.`
4
5	`---`
6
7	`## Prerequisites`
8
9	`- Redis 7+ (or a managed Redis service — Upstash, Redis Cloud, etc.)`
10	- `pip install "navegador[redis]"`
11
12	`---`
13
14	`## Setup`
15
16	`### 1. Initialize cluster mode`
17
18	`Point navegador at your Redis instance and run init:`
19
20	```bash
21	`navegador init --cluster --redis redis://your-redis-host:6379`
22	```
23
24	This writes cluster config to `navegador.toml`:
25
26	```toml
27	`[cluster]`
28	`enabled = true`
29	`redis_url = "redis://your-redis-host:6379"`
30	`graph_name = "navegador"`
31	`node_id = "worker-1" # auto-generated; override with --node-id`
32	```
33
34	`### 2. Verify connectivity`
35
36	```bash
37	`navegador cluster status`
38	```
39
40	`Output:`
41
42	```
43	`Cluster: connected`
44	`Redis: redis://your-redis-host:6379`
45	`Graph: navegador (47,231 nodes, 189,043 edges)`
46	`Workers: 3 online (worker-1, worker-2, ci-runner-7)`
47	`Queue: 0 tasks pending`
48	```
49
50	`---`
51
52	`## Shared graph`
53
54	`All cluster members read from and write to the same FalkorDB graph stored in Redis. Any ingestion or annotation from any node is immediately visible to all other nodes.`
55
56	```bash
57	`# on any machine in the cluster`
58	`navegador ingest ./src`
59
60	`# on any other machine — sees the result immediately`
61	`navegador explain AuthService`
62	```
63
64	`### Local snapshots`
65
66	`To work offline or reduce Redis round-trips, snapshot the graph to a local SQLite file:`
67
68	```bash
69	`# pull a snapshot from the shared graph`
70	`navegador cluster snapshot --pull .navegador/local.db`
71
72	`# use the snapshot for queries`
73	`navegador --db .navegador/local.db explain AuthService`
74
75	`# push local changes back to the shared graph`
76	`navegador cluster snapshot --push .navegador/local.db`
77	```
78
79	Snapshots are point-in-time copies. They do not auto-sync. Use `--pull` to refresh and `--push` to merge back.
80
81	`---`
82
83	`## Task queue`
84
85	The cluster task queue distributes ingestion and analysis jobs across workers. Instead of running `navegador ingest` directly, submit a task:
86
87	```bash
88	`# submit an ingestion task`
89	`navegador cluster enqueue ingest ./src --clear`
90
91	`# submit an analysis task`
92	`navegador cluster enqueue analyze impact validate_token`
93
94	`# list pending and active tasks`
95	`navegador cluster queue`
96	```
97
98	`Workers pick up tasks from the queue automatically. See [work partitioning](#work-partitioning) for multi-worker ingestion.`
99
100	`### Starting a worker`
101
102	```bash
103	`navegador cluster worker start`
104	```
105
106	`The worker polls the queue and processes tasks. Run one worker per machine.`
107
108	```bash
109	`# run in background`
110	`navegador cluster worker start --daemon`
111
112	`# stop the worker`
113	`navegador cluster worker stop`
114	```
115
116	`---`
117
118	`## Work partitioning`
119
120	`For large monorepos, split ingestion across multiple workers:`
121
122	```bash
123	`# partition a directory across N workers`
124	`navegador cluster partition ./src --workers 4`
125
126	`# each worker then runs its assigned slice`
127	`navegador cluster worker start --partition 0 --of 4 # worker 0`
128	`navegador cluster worker start --partition 1 --of 4 # worker 1`
129	`# ...`
130	```
131
132	`Partitioning splits the file list across workers by file count. All workers write to the same shared graph. The final graph is the union of all partitions.`
133
134	`### In CI`
135
136	```yaml
137	`# .github/workflows/ingest.yml`
138	`jobs:`
139	`ingest:`
140	`strategy:`
141	`matrix:`
142	`partition: [0, 1, 2, 3]`
143	`steps:`
144	`- run: navegador cluster worker start --partition ${{ matrix.partition }} --of 4 --run-once`
145	```
146
147	`--run-once` processes the current queue and exits rather than running as a daemon.
148
149	`---`
150
151	`## Sessions`
152
153	`Sessions let multiple agents coordinate on the same task without interfering with each other.`
154
155	```bash
156	`# start a session (returns a session ID)`
157	`SESSION=$(navegador cluster session start --name "feature/auth-refactor")`
158	`echo "Session: $SESSION"`
159
160	`# run commands scoped to the session`
161	`navegador --session $SESSION ingest ./src/auth`
162	`navegador --session $SESSION explain AuthService`
163
164	`# end the session`
165	`navegador cluster session end $SESSION`
166	```
167
168	`Sessions create a namespaced view of the graph. Writes within a session are visible to other session members but isolated from the main graph until committed.`
169
170	```bash
171	`# commit session changes to the main graph`
172	`navegador cluster session commit $SESSION`
173
174	`# discard session changes`
175	`navegador cluster session discard $SESSION`
176	```
177
178	`---`
179
180	`## Locking`
181
182	For writes that must not overlap (e.g., `--clear` ingest), navegador acquires a distributed lock:
183
184	```bash
185	`navegador ingest ./src --clear`
186	`# automatically acquires the graph write lock; other writers block until it releases`
187	```
188
189	`You can also acquire locks explicitly:`
190
191	```bash
192	`# acquire a named lock`
193	`LOCK=$(navegador cluster lock acquire "ingest-lock" --ttl 300)`
194
195	`# ... run your operations ...`
196
197	`# release the lock`
198	`navegador cluster lock release $LOCK`
199	```
200
201	`Locks have a TTL (seconds) and release automatically if the holder crashes.`
202
203	`---`
204
205	`## Messaging`
206
207	`Workers and agents can exchange messages via the cluster bus:`
208
209	```bash
210	`# publish a message to a channel`
211	`navegador cluster publish "ingest.complete" '{"repo": "myorg/myrepo", "nodes": 12450}'`
212
213	`# subscribe to a channel (blocks; prints messages as they arrive)`
214	`navegador cluster subscribe "ingest.complete"`
215	```
216
217	`Useful for triggering downstream steps (e.g., notify agents that a fresh ingest is ready) without polling.`
218
219	`---`
220
221	`## Observability`
222
223	`### Cluster metrics`
224
225	```bash
226	`navegador cluster metrics`
227	```
228
229	`Output:`
230
231	```
232	`Graph`
233	`Nodes: 47,231`
234	`Edges: 189,043`
235	`Last ingest: 2026-03-23T14:22:11Z (worker-2)`
236
237	`Workers (3 online)`
238	`worker-1 idle last seen 4s ago`
239	`worker-2 idle last seen 2s ago`
240	`ci-runner-7 processing task: ingest ./src/payments`
241
242	`Queue`
243	`Pending: 0`
244	`Active: 1`
245	`Completed: 847 (last 24h)`
246	`Failed: 2 (last 24h)`
247	```
248
249	`### Logs`
250
251	`Workers emit structured JSON logs. Stream them:`
252
253	```bash
254	`navegador cluster logs --follow`
255	`navegador cluster logs --worker worker-2 --since 1h`
256	```
257
258	`### Health check`
259
260	```bash
261	`navegador cluster health`
262	`# exits 0 if healthy, 1 if degraded, 2 if unavailable`
263	```
264
265	`Suitable for use in load balancer health checks and PagerDuty integrations.`
266
267	`---`
268
269	`## Configuration reference`
270
271	All cluster settings can be set in `navegador.toml` or as environment variables:
272
273	`\| Setting \| Env var \| Default \| Description \|`
274	`\|---\|---\|---\|---\|`
275	\| `redis_url` \| `NAVEGADOR_REDIS_URL` \| `redis://localhost:6379` \| Redis connection URL \|
276	\| `graph_name` \| `NAVEGADOR_GRAPH_NAME` \| `navegador` \| FalkorDB graph name \|
277	\| `node_id` \| `NAVEGADOR_NODE_ID` \| auto \| Unique identifier for this worker \|
278	\| `lock_ttl` \| `NAVEGADOR_LOCK_TTL` \| `300` \| Default lock TTL in seconds \|
279	\| `worker_poll_interval` \| `NAVEGADOR_POLL_INTERVAL` \| `2` \| Queue poll interval in seconds \|
280	\| `snapshot_dir` \| `NAVEGADOR_SNAPSHOT_DIR` \| `.navegador/snapshots` \| Local snapshot directory \|
281

Navegador

Keyboard Shortcuts