Navegador

navegador / docs / guide / cluster.md
Source Blame History 280 lines
89816aa… lmata 1 # Cluster Mode
89816aa… lmata 2
89816aa… lmata 3 Cluster mode lets multiple machines share a single navegador graph over Redis. Use it when your team runs large ingestion jobs, want shared context across agents and CI, or need partitioned work processing.
89816aa… lmata 4
89816aa… lmata 5 ---
89816aa… lmata 6
89816aa… lmata 7 ## Prerequisites
89816aa… lmata 8
89816aa… lmata 9 - Redis 7+ (or a managed Redis service — Upstash, Redis Cloud, etc.)
89816aa… lmata 10 - `pip install "navegador[redis]"`
89816aa… lmata 11
89816aa… lmata 12 ---
89816aa… lmata 13
89816aa… lmata 14 ## Setup
89816aa… lmata 15
89816aa… lmata 16 ### 1. Initialize cluster mode
89816aa… lmata 17
89816aa… lmata 18 Point navegador at your Redis instance and run init:
89816aa… lmata 19
89816aa… lmata 20 ```bash
89816aa… lmata 21 navegador init --cluster --redis redis://your-redis-host:6379
89816aa… lmata 22 ```
89816aa… lmata 23
89816aa… lmata 24 This writes cluster config to `navegador.toml`:
89816aa… lmata 25
89816aa… lmata 26 ```toml
89816aa… lmata 27 [cluster]
89816aa… lmata 28 enabled = true
89816aa… lmata 29 redis_url = "redis://your-redis-host:6379"
89816aa… lmata 30 graph_name = "navegador"
89816aa… lmata 31 node_id = "worker-1" # auto-generated; override with --node-id
89816aa… lmata 32 ```
89816aa… lmata 33
89816aa… lmata 34 ### 2. Verify connectivity
89816aa… lmata 35
89816aa… lmata 36 ```bash
89816aa… lmata 37 navegador cluster status
89816aa… lmata 38 ```
89816aa… lmata 39
89816aa… lmata 40 Output:
89816aa… lmata 41
89816aa… lmata 42 ```
89816aa… lmata 43 Cluster: connected
89816aa… lmata 44 Redis: redis://your-redis-host:6379
89816aa… lmata 45 Graph: navegador (47,231 nodes, 189,043 edges)
89816aa… lmata 46 Workers: 3 online (worker-1, worker-2, ci-runner-7)
89816aa… lmata 47 Queue: 0 tasks pending
89816aa… lmata 48 ```
89816aa… lmata 49
89816aa… lmata 50 ---
89816aa… lmata 51
89816aa… lmata 52 ## Shared graph
89816aa… lmata 53
89816aa… lmata 54 All cluster members read from and write to the same FalkorDB graph stored in Redis. Any ingestion or annotation from any node is immediately visible to all other nodes.
89816aa… lmata 55
89816aa… lmata 56 ```bash
89816aa… lmata 57 # on any machine in the cluster
89816aa… lmata 58 navegador ingest ./src
89816aa… lmata 59
89816aa… lmata 60 # on any other machine — sees the result immediately
89816aa… lmata 61 navegador explain AuthService
89816aa… lmata 62 ```
89816aa… lmata 63
89816aa… lmata 64 ### Local snapshots
89816aa… lmata 65
89816aa… lmata 66 To work offline or reduce Redis round-trips, snapshot the graph to a local SQLite file:
89816aa… lmata 67
89816aa… lmata 68 ```bash
89816aa… lmata 69 # pull a snapshot from the shared graph
89816aa… lmata 70 navegador cluster snapshot --pull .navegador/local.db
89816aa… lmata 71
89816aa… lmata 72 # use the snapshot for queries
89816aa… lmata 73 navegador --db .navegador/local.db explain AuthService
89816aa… lmata 74
89816aa… lmata 75 # push local changes back to the shared graph
89816aa… lmata 76 navegador cluster snapshot --push .navegador/local.db
89816aa… lmata 77 ```
89816aa… lmata 78
89816aa… lmata 79 Snapshots are point-in-time copies. They do not auto-sync. Use `--pull` to refresh and `--push` to merge back.
89816aa… lmata 80
89816aa… lmata 81 ---
89816aa… lmata 82
89816aa… lmata 83 ## Task queue
89816aa… lmata 84
89816aa… lmata 85 The cluster task queue distributes ingestion and analysis jobs across workers. Instead of running `navegador ingest` directly, submit a task:
89816aa… lmata 86
89816aa… lmata 87 ```bash
89816aa… lmata 88 # submit an ingestion task
89816aa… lmata 89 navegador cluster enqueue ingest ./src --clear
89816aa… lmata 90
89816aa… lmata 91 # submit an analysis task
89816aa… lmata 92 navegador cluster enqueue analyze impact validate_token
89816aa… lmata 93
89816aa… lmata 94 # list pending and active tasks
89816aa… lmata 95 navegador cluster queue
89816aa… lmata 96 ```
89816aa… lmata 97
89816aa… lmata 98 Workers pick up tasks from the queue automatically. See [work partitioning](#work-partitioning) for multi-worker ingestion.
89816aa… lmata 99
89816aa… lmata 100 ### Starting a worker
89816aa… lmata 101
89816aa… lmata 102 ```bash
89816aa… lmata 103 navegador cluster worker start
89816aa… lmata 104 ```
89816aa… lmata 105
89816aa… lmata 106 The worker polls the queue and processes tasks. Run one worker per machine.
89816aa… lmata 107
89816aa… lmata 108 ```bash
89816aa… lmata 109 # run in background
89816aa… lmata 110 navegador cluster worker start --daemon
89816aa… lmata 111
89816aa… lmata 112 # stop the worker
89816aa… lmata 113 navegador cluster worker stop
89816aa… lmata 114 ```
89816aa… lmata 115
89816aa… lmata 116 ---
89816aa… lmata 117
89816aa… lmata 118 ## Work partitioning
89816aa… lmata 119
89816aa… lmata 120 For large monorepos, split ingestion across multiple workers:
89816aa… lmata 121
89816aa… lmata 122 ```bash
89816aa… lmata 123 # partition a directory across N workers
89816aa… lmata 124 navegador cluster partition ./src --workers 4
89816aa… lmata 125
89816aa… lmata 126 # each worker then runs its assigned slice
89816aa… lmata 127 navegador cluster worker start --partition 0 --of 4 # worker 0
89816aa… lmata 128 navegador cluster worker start --partition 1 --of 4 # worker 1
89816aa… lmata 129 # ...
89816aa… lmata 130 ```
89816aa… lmata 131
89816aa… lmata 132 Partitioning splits the file list across workers by file count. All workers write to the same shared graph. The final graph is the union of all partitions.
89816aa… lmata 133
89816aa… lmata 134 ### In CI
89816aa… lmata 135
89816aa… lmata 136 ```yaml
89816aa… lmata 137 # .github/workflows/ingest.yml
89816aa… lmata 138 jobs:
89816aa… lmata 139 ingest:
89816aa… lmata 140 strategy:
89816aa… lmata 141 matrix:
89816aa… lmata 142 partition: [0, 1, 2, 3]
89816aa… lmata 143 steps:
89816aa… lmata 144 - run: navegador cluster worker start --partition ${{ matrix.partition }} --of 4 --run-once
89816aa… lmata 145 ```
89816aa… lmata 146
89816aa… lmata 147 `--run-once` processes the current queue and exits rather than running as a daemon.
89816aa… lmata 148
89816aa… lmata 149 ---
89816aa… lmata 150
89816aa… lmata 151 ## Sessions
89816aa… lmata 152
89816aa… lmata 153 Sessions let multiple agents coordinate on the same task without interfering with each other.
89816aa… lmata 154
89816aa… lmata 155 ```bash
89816aa… lmata 156 # start a session (returns a session ID)
89816aa… lmata 157 SESSION=$(navegador cluster session start --name "feature/auth-refactor")
89816aa… lmata 158 echo "Session: $SESSION"
89816aa… lmata 159
89816aa… lmata 160 # run commands scoped to the session
89816aa… lmata 161 navegador --session $SESSION ingest ./src/auth
89816aa… lmata 162 navegador --session $SESSION explain AuthService
89816aa… lmata 163
89816aa… lmata 164 # end the session
89816aa… lmata 165 navegador cluster session end $SESSION
89816aa… lmata 166 ```
89816aa… lmata 167
89816aa… lmata 168 Sessions create a namespaced view of the graph. Writes within a session are visible to other session members but isolated from the main graph until committed.
89816aa… lmata 169
89816aa… lmata 170 ```bash
89816aa… lmata 171 # commit session changes to the main graph
89816aa… lmata 172 navegador cluster session commit $SESSION
89816aa… lmata 173
89816aa… lmata 174 # discard session changes
89816aa… lmata 175 navegador cluster session discard $SESSION
89816aa… lmata 176 ```
89816aa… lmata 177
89816aa… lmata 178 ---
89816aa… lmata 179
89816aa… lmata 180 ## Locking
89816aa… lmata 181
89816aa… lmata 182 For writes that must not overlap (e.g., `--clear` ingest), navegador acquires a distributed lock:
89816aa… lmata 183
89816aa… lmata 184 ```bash
89816aa… lmata 185 navegador ingest ./src --clear
89816aa… lmata 186 # automatically acquires the graph write lock; other writers block until it releases
89816aa… lmata 187 ```
89816aa… lmata 188
89816aa… lmata 189 You can also acquire locks explicitly:
89816aa… lmata 190
89816aa… lmata 191 ```bash
89816aa… lmata 192 # acquire a named lock
89816aa… lmata 193 LOCK=$(navegador cluster lock acquire "ingest-lock" --ttl 300)
89816aa… lmata 194
89816aa… lmata 195 # ... run your operations ...
89816aa… lmata 196
89816aa… lmata 197 # release the lock
89816aa… lmata 198 navegador cluster lock release $LOCK
89816aa… lmata 199 ```
89816aa… lmata 200
89816aa… lmata 201 Locks have a TTL (seconds) and release automatically if the holder crashes.
89816aa… lmata 202
89816aa… lmata 203 ---
89816aa… lmata 204
89816aa… lmata 205 ## Messaging
89816aa… lmata 206
89816aa… lmata 207 Workers and agents can exchange messages via the cluster bus:
89816aa… lmata 208
89816aa… lmata 209 ```bash
89816aa… lmata 210 # publish a message to a channel
89816aa… lmata 211 navegador cluster publish "ingest.complete" '{"repo": "myorg/myrepo", "nodes": 12450}'
89816aa… lmata 212
89816aa… lmata 213 # subscribe to a channel (blocks; prints messages as they arrive)
89816aa… lmata 214 navegador cluster subscribe "ingest.complete"
89816aa… lmata 215 ```
89816aa… lmata 216
89816aa… lmata 217 Useful for triggering downstream steps (e.g., notify agents that a fresh ingest is ready) without polling.
89816aa… lmata 218
89816aa… lmata 219 ---
89816aa… lmata 220
89816aa… lmata 221 ## Observability
89816aa… lmata 222
89816aa… lmata 223 ### Cluster metrics
89816aa… lmata 224
89816aa… lmata 225 ```bash
89816aa… lmata 226 navegador cluster metrics
89816aa… lmata 227 ```
89816aa… lmata 228
89816aa… lmata 229 Output:
89816aa… lmata 230
89816aa… lmata 231 ```
89816aa… lmata 232 Graph
89816aa… lmata 233 Nodes: 47,231
89816aa… lmata 234 Edges: 189,043
89816aa… lmata 235 Last ingest: 2026-03-23T14:22:11Z (worker-2)
89816aa… lmata 236
89816aa… lmata 237 Workers (3 online)
89816aa… lmata 238 worker-1 idle last seen 4s ago
89816aa… lmata 239 worker-2 idle last seen 2s ago
89816aa… lmata 240 ci-runner-7 processing task: ingest ./src/payments
89816aa… lmata 241
89816aa… lmata 242 Queue
89816aa… lmata 243 Pending: 0
89816aa… lmata 244 Active: 1
89816aa… lmata 245 Completed: 847 (last 24h)
89816aa… lmata 246 Failed: 2 (last 24h)
89816aa… lmata 247 ```
89816aa… lmata 248
89816aa… lmata 249 ### Logs
89816aa… lmata 250
89816aa… lmata 251 Workers emit structured JSON logs. Stream them:
89816aa… lmata 252
89816aa… lmata 253 ```bash
89816aa… lmata 254 navegador cluster logs --follow
89816aa… lmata 255 navegador cluster logs --worker worker-2 --since 1h
89816aa… lmata 256 ```
89816aa… lmata 257
89816aa… lmata 258 ### Health check
89816aa… lmata 259
89816aa… lmata 260 ```bash
89816aa… lmata 261 navegador cluster health
89816aa… lmata 262 # exits 0 if healthy, 1 if degraded, 2 if unavailable
89816aa… lmata 263 ```
89816aa… lmata 264
89816aa… lmata 265 Suitable for use in load balancer health checks and PagerDuty integrations.
89816aa… lmata 266
89816aa… lmata 267 ---
89816aa… lmata 268
89816aa… lmata 269 ## Configuration reference
89816aa… lmata 270
89816aa… lmata 271 All cluster settings can be set in `navegador.toml` or as environment variables:
89816aa… lmata 272
89816aa… lmata 273 | Setting | Env var | Default | Description |
89816aa… lmata 274 |---|---|---|---|
89816aa… lmata 275 | `redis_url` | `NAVEGADOR_REDIS_URL` | `redis://localhost:6379` | Redis connection URL |
89816aa… lmata 276 | `graph_name` | `NAVEGADOR_GRAPH_NAME` | `navegador` | FalkorDB graph name |
89816aa… lmata 277 | `node_id` | `NAVEGADOR_NODE_ID` | auto | Unique identifier for this worker |
89816aa… lmata 278 | `lock_ttl` | `NAVEGADOR_LOCK_TTL` | `300` | Default lock TTL in seconds |
89816aa… lmata 279 | `worker_poll_interval` | `NAVEGADOR_POLL_INTERVAL` | `2` | Queue poll interval in seconds |
89816aa… lmata 280 | `snapshot_dir` | `NAVEGADOR_SNAPSHOT_DIR` | `.navegador/snapshots` | Local snapshot directory |

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button