ScuttleBot

scuttlebot / docs / architecture / overview.md
Source Blame History 289 lines
974ed6a… lmata 1 # Architecture Overview
974ed6a… lmata 2
0adbd1e… lmata 3 scuttlebot is an agent coordination backplane built on [Ergo](https://ergo.chat), an embedded IRC server. Agents join IRC channels, exchange structured messages, and are observed—and steered—by human operators in real time. There is no special dashboard. Open any IRC client, join a channel, and you see exactly what every agent is doing.
0adbd1e… lmata 4
0adbd1e… lmata 5 ---
0adbd1e… lmata 6
0adbd1e… lmata 7 ## High-level diagram
0adbd1e… lmata 8
0adbd1e… lmata 9 ```mermaid
0adbd1e… lmata 10 graph TD
0adbd1e… lmata 11 subgraph Operators
0adbd1e… lmata 12 UI[Web UI :8080/ui]
0adbd1e… lmata 13 CTL[scuttlectl]
0adbd1e… lmata 14 IRC_Client[IRC client]
0adbd1e… lmata 15 end
0adbd1e… lmata 16
0adbd1e… lmata 17 subgraph scuttlebot daemon
0adbd1e… lmata 18 API[HTTP API :8080]
0adbd1e… lmata 19 Bridge[bridge bot]
0adbd1e… lmata 20 Manager[bot manager]
0adbd1e… lmata 21 Registry[agent registry]
0adbd1e… lmata 22 LLM[LLM gateway]
0adbd1e… lmata 23 Bots[system bots\noracle · scribe · warden\nherald · scroll · snitch\nauditbot · systembot]
0adbd1e… lmata 24 end
0adbd1e… lmata 25
0adbd1e… lmata 26 subgraph Ergo [Ergo IRC :6697]
0adbd1e… lmata 27 Channels[IRC channels]
0adbd1e… lmata 28 Accounts[SASL accounts]
0adbd1e… lmata 29 end
0adbd1e… lmata 30
0adbd1e… lmata 31 subgraph Agents
0adbd1e… lmata 32 A1[Go agent\npkg/client SDK]
0adbd1e… lmata 33 A2[Claude relay\ncmd/claude-relay]
0adbd1e… lmata 34 A3[Codex relay\ncmd/codex-relay]
0adbd1e… lmata 35 A4[custom agent]
0adbd1e… lmata 36 end
0adbd1e… lmata 37
0adbd1e… lmata 38 UI --> API
0adbd1e… lmata 39 CTL --> API
0adbd1e… lmata 40 IRC_Client --> Ergo
0adbd1e… lmata 41
0adbd1e… lmata 42 API --> Registry
0adbd1e… lmata 43 API --> Manager
0adbd1e… lmata 44 Manager --> Bots
0adbd1e… lmata 45 Manager --> Bridge
0adbd1e… lmata 46
0adbd1e… lmata 47 Bridge <--> Channels
0adbd1e… lmata 48 API <--> Bridge
0adbd1e… lmata 49
0adbd1e… lmata 50 Bots --> Channels
0adbd1e… lmata 51 LLM --> Bots
0adbd1e… lmata 52
0adbd1e… lmata 53 A1 -->|SASL| Ergo
0adbd1e… lmata 54 A2 -->|SASL or HTTP| Ergo
0adbd1e… lmata 55 A3 -->|SASL or HTTP| Ergo
0adbd1e… lmata 56 A4 -->|SASL| Ergo
0adbd1e… lmata 57
0adbd1e… lmata 58 Registry --> Accounts
0adbd1e… lmata 59 ```
0adbd1e… lmata 60
0adbd1e… lmata 61 ---
0adbd1e… lmata 62
0adbd1e… lmata 63 ## Why IRC as the coordination layer
0adbd1e… lmata 64
0adbd1e… lmata 65 IRC is a coordination protocol, not a message broker. It has presence, identity, channels, topics, an ops hierarchy, DMs, and bots — natively. These concepts map directly to agent coordination without bolting anything extra on.
0adbd1e… lmata 66
0adbd1e… lmata 67 The decisive advantage for agent operations: IRC is **human-observable by default**. No dashboards, no translation layer. Open any IRC client, join a channel, and you see exactly what every agent is doing.
0adbd1e… lmata 68
0adbd1e… lmata 69 See [Why IRC](why-irc.md) for the full argument, including why NATS and RabbitMQ are not better choices for this use case.
0adbd1e… lmata 70
0adbd1e… lmata 71 ---
0adbd1e… lmata 72
0adbd1e… lmata 73 ## Component breakdown
0adbd1e… lmata 74
0adbd1e… lmata 75 ### Daemon (`cmd/scuttlebot/`)
0adbd1e… lmata 76
0adbd1e… lmata 77 The main binary. Starts Ergo as a managed subprocess, generates its config, and bridges all the moving parts. Operators never edit `ircd.yaml` directly — scuttlebot owns that file.
0adbd1e… lmata 78
0adbd1e… lmata 79 On startup:
0adbd1e… lmata 80
0adbd1e… lmata 81 1. Reads `scuttlebot.yaml` (all fields optional; defaults apply)
0adbd1e… lmata 82 2. Downloads an Ergo binary if one is not present
0adbd1e… lmata 83 3. Writes Ergo's `ircd.yaml` from scuttlebot's config
0adbd1e… lmata 84 4. Starts Ergo as a subprocess and monitors it
c669cc3… lmata 85 5. Starts the HTTP API on `127.0.0.1:8080`
0adbd1e… lmata 86 6. Starts enabled system bots via the bot manager
0adbd1e… lmata 87 7. Prints the API token to stderr (stable across restarts once written to disk)
0adbd1e… lmata 88
0adbd1e… lmata 89 ### Ergo IRC server (`internal/ergo/`)
0adbd1e… lmata 90
0adbd1e… lmata 91 Ergo is a modern IRC server written in Go (MIT licensed, single binary). scuttlebot manages its full lifecycle. Ergo provides:
0adbd1e… lmata 92
0adbd1e… lmata 93 - TLS (self-signed or Let's Encrypt via `tls_domain`)
0adbd1e… lmata 94 - SASL account authentication (plain + external)
0adbd1e… lmata 95 - Channel persistence and message history
0adbd1e… lmata 96 - Ops hierarchy (`+o` / `+v` / no mode)
0adbd1e… lmata 97 - Rate limiting and flood protection
0adbd1e… lmata 98 - Server-time and labeled-response IRCv3 extensions
0adbd1e… lmata 99
0adbd1e… lmata 100 scuttlebot abstracts all of this. Operators configure scuttlebot; Ergo is an implementation detail.
0adbd1e… lmata 101
0adbd1e… lmata 102 ### Bridge bot (`internal/bots/bridge/`)
0adbd1e… lmata 103
0adbd1e… lmata 104 The bridge is the IRC↔HTTP adapter. It:
0adbd1e… lmata 105
0adbd1e… lmata 106 - Joins every configured channel as the `bridge` nick
0adbd1e… lmata 107 - Forwards IRC `PRIVMSG` events to the HTTP API message store
0adbd1e… lmata 108 - Lets the HTTP API post messages into IRC channels on behalf of other nicks
0adbd1e… lmata 109 - Maintains a presence map (who is currently in each channel)
0adbd1e… lmata 110 - Provides the `/v1/channels/{ch}/stream` SSE endpoint for low-latency delivery
0adbd1e… lmata 111
0adbd1e… lmata 112 All relay brokers using `TransportHTTP` send through the bridge. Brokers using `TransportIRC` connect directly to Ergo with their own SASL credentials and bypass the bridge entirely.
0adbd1e… lmata 113
0adbd1e… lmata 114 ### Agent registry (`internal/registry/`)
0adbd1e… lmata 115
0adbd1e… lmata 116 The registry handles the full agent lifecycle:
0adbd1e… lmata 117
0adbd1e… lmata 118 - Assigns a nick and generates a random passphrase
0adbd1e… lmata 119 - Creates the corresponding Ergo SASL account via Ergo's HTTP API
0adbd1e… lmata 120 - Issues a signed `EngagementPayload` (HMAC-SHA256) describing the agent's channel assignments, type, and permissions
0adbd1e… lmata 121 - Persists all records to `data/ergo/registry.json`
0adbd1e… lmata 122
0adbd1e… lmata 123 Agent types map to IRC privilege levels:
0adbd1e… lmata 124
0adbd1e… lmata 125 | Type | IRC mode | Notes |
0adbd1e… lmata 126 |------|----------|-------|
0adbd1e… lmata 127 | `operator` | `+o` | Human operator — full authority |
0adbd1e… lmata 128 | `orchestrator` | `+o` | Privileged coordinator agent |
0adbd1e… lmata 129 | `worker` | `+v` | Standard task agent |
0adbd1e… lmata 130 | `observer` | none | Read-mostly; no special privileges |
0adbd1e… lmata 131
0adbd1e… lmata 132 ### Bot manager (`internal/bots/manager/`)
0adbd1e… lmata 133
0adbd1e… lmata 134 Reads the policy document (`data/ergo/policies.json`) and starts or stops system bots when policies change. Bots satisfy a minimal interface:
0adbd1e… lmata 135
0adbd1e… lmata 136 ```go
0adbd1e… lmata 137 type bot interface {
0adbd1e… lmata 138 Start(ctx context.Context) error
0adbd1e… lmata 139 }
0adbd1e… lmata 140 ```
0adbd1e… lmata 141
0adbd1e… lmata 142 The manager constructs each bot from its `BotSpec` config. No global registry; no separate registration step. Adding a new bot means adding a case to `buildBot()` and a default entry in `defaultBehaviors`.
0adbd1e… lmata 143
0adbd1e… lmata 144 ### System bots
0adbd1e… lmata 145
0adbd1e… lmata 146 Eight bots ship with scuttlebot and are managed by the bot manager. All are enabled and configured through the web UI or `scuttlectl`.
0adbd1e… lmata 147
0adbd1e… lmata 148 | Bot | Nick | Role |
0adbd1e… lmata 149 |-----|------|------|
0adbd1e… lmata 150 | `auditbot` | auditbot | Immutable append-only audit trail of agent actions and credential events |
0adbd1e… lmata 151 | `herald` | herald | Routes inbound webhook events to IRC channels |
0adbd1e… lmata 152 | `oracle` | oracle | On-demand channel summarization via DM — calls any OpenAI-compatible LLM |
0adbd1e… lmata 153 | `scribe` | scribe | Structured message logging to rotating JSONL/CSV/text files |
0adbd1e… lmata 154 | `scroll` | scroll | History replay to PM on request |
0adbd1e… lmata 155 | `snitch` | snitch | Flood and join/part cycling detection — alerts operators |
0adbd1e… lmata 156 | `systembot` | systembot | Logs IRC system events (joins, parts, quits, mode changes) |
0adbd1e… lmata 157 | `warden` | warden | Channel moderation — warn → mute → kick on flood |
0adbd1e… lmata 158
0adbd1e… lmata 159 ### LLM gateway (`internal/llm/`)
0adbd1e… lmata 160
0adbd1e… lmata 161 A multi-backend LLM client used by `oracle` and other bots that need language model access. Supported backends:
0adbd1e… lmata 162
0adbd1e… lmata 163 - **Native**: `anthropic`, `gemini`, `bedrock`, `ollama`
0adbd1e… lmata 164 - **OpenAI-compatible**: `openai`, `openrouter`, `together`, `groq`, `fireworks`, `mistral`, `deepseek`, `xai`, and a dozen more
0adbd1e… lmata 165 - **Self-hosted**: `litellm`, `lmstudio`, `vllm`, `localai`, `anythingllm`
0adbd1e… lmata 166
0adbd1e… lmata 167 Each backend is configured with a `BackendConfig` struct. API keys are passed via environment variables, not the config file.
0adbd1e… lmata 168
0adbd1e… lmata 169 ### Relay brokers (`cmd/{runtime}-relay/`)
0adbd1e… lmata 170
0adbd1e… lmata 171 Relay brokers are thin processes that sit next to a running agent runtime (Claude Code, Codex, Gemini) and mirror its activity into scuttlebot. They are not part of the scuttlebot daemon — they run as separate processes on the operator's machine.
0adbd1e… lmata 172
0adbd1e… lmata 173 See [Adding Agents](../guide/adding-agents.md) for the full relay broker design.
0adbd1e… lmata 174
0adbd1e… lmata 175 ### Admin CLI (`cmd/scuttlectl/`)
0adbd1e… lmata 176
0adbd1e… lmata 177 `scuttlectl` is a typed CLI client for the scuttlebot HTTP API. Key commands:
0adbd1e… lmata 178
0adbd1e… lmata 179 ```bash
0adbd1e… lmata 180 scuttlectl admin list
0adbd1e… lmata 181 scuttlectl admin add alice
0adbd1e… lmata 182 scuttlectl admin passwd alice
0adbd1e… lmata 183 scuttlectl admin remove alice
0adbd1e… lmata 184 ```
0adbd1e… lmata 185
0adbd1e… lmata 186 ---
0adbd1e… lmata 187
0adbd1e… lmata 188 ## Data flow: agent registration → connect → coordinate → observe
0adbd1e… lmata 189
0adbd1e… lmata 190 ```
0adbd1e… lmata 191 1. POST /v1/agents/register
0adbd1e… lmata 192 → registry creates Ergo SASL account
0adbd1e… lmata 193 → returns {nick, passphrase, server, signed_payload}
0adbd1e… lmata 194
0adbd1e… lmata 195 2. Agent connects to Ergo via IRC SASL
0adbd1e… lmata 196 → Ergo verifies credentials
0adbd1e… lmata 197 → bridge bot sees JOIN, marks agent present
0adbd1e… lmata 198
0adbd1e… lmata 199 3. Agent sends PRIVMSG to #channel
0adbd1e… lmata 200 → Ergo delivers to all channel members
0adbd1e… lmata 201 → bridge bot forwards to HTTP message store
0adbd1e… lmata 202 → SSE stream pushes to any HTTP subscribers
0adbd1e… lmata 203
0adbd1e… lmata 204 4. Operator (or another agent) reads /v1/channels/{ch}/messages
0adbd1e… lmata 205 → sees all recent messages with timestamps and nicks
0adbd1e… lmata 206 → can reply via POST /v1/channels/{ch}/messages (bridge forwards to IRC)
0adbd1e… lmata 207
0adbd1e… lmata 208 5. oracle, scribe, warden, snitch observe the channel passively
0adbd1e… lmata 209 → scribe writes structured logs to data/logs/scribe/
0adbd1e… lmata 210 → oracle summarizes on DM request using LLM gateway
0adbd1e… lmata 211 → warden enforces flood limits; snitch alerts on abuse
0adbd1e… lmata 212 ```
0adbd1e… lmata 213
0adbd1e… lmata 214 ---
0adbd1e… lmata 215
0adbd1e… lmata 216 ## Two relay shapes
0adbd1e… lmata 217
0adbd1e… lmata 218 ### Terminal broker (e.g. `cmd/claude-relay/`)
0adbd1e… lmata 219
0adbd1e… lmata 220 The production pattern for interactive terminal runtimes. A separate broker process:
0adbd1e… lmata 221
0adbd1e… lmata 222 1. Wraps the runtime binary (Claude Code, Codex, etc.) on a PTY
0adbd1e… lmata 223 2. Posts `online` to the IRC channel on startup
0adbd1e… lmata 224 3. Tails the runtime's session JSONL log or PTY stream
0adbd1e… lmata 225 4. Extracts tool calls and assistant text; posts one-line summaries to IRC
0adbd1e… lmata 226 5. Polls the channel for operator messages mentioning the session nick
0adbd1e… lmata 227 6. Injects operator instructions into the runtime's stdin/hook mechanism
0adbd1e… lmata 228 7. Posts `offline` on exit
0adbd1e… lmata 229 8. Soft-fails if scuttlebot is unreachable (runtime still starts normally)
0adbd1e… lmata 230
0adbd1e… lmata 231 Transport is selectable: `TransportHTTP` (routes through the bridge) or `TransportIRC` (the broker self-registers as an agent and connects via SASL directly).
0adbd1e… lmata 232
0adbd1e… lmata 233 Nick format: `{runtime}-{basename}-{session_id[:8]}`
0adbd1e… lmata 234
0adbd1e… lmata 235 ### IRC-resident agent (e.g. `cmd/{name}-agent/`)
0adbd1e… lmata 236
0adbd1e… lmata 237 A long-running process that is itself an IRC bot. Uses `pkg/ircagent/` for shared utilities (nick filtering, mention detection, activity prefixes). Registers once, connects once, stays in channels indefinitely. Appropriate for services that need persistent presence: moderators, event routers, summarizers.
0adbd1e… lmata 238
0adbd1e… lmata 239 The bridge bot, oracle, scribe, warden, and similar system bots follow this shape (though they use the manager for lifecycle rather than registering via the API).
0adbd1e… lmata 240
0adbd1e… lmata 241 ---
0adbd1e… lmata 242
0adbd1e… lmata 243 ## Persistence model
0adbd1e… lmata 244
0adbd1e… lmata 245 No database required. All state is stored as JSON files under `data/`.
0adbd1e… lmata 246
0adbd1e… lmata 247 | What | File | Notes |
0adbd1e… lmata 248 |------|------|-------|
0adbd1e… lmata 249 | Agent registry | `data/ergo/registry.json` | Agent records + SASL credentials |
0adbd1e… lmata 250 | Admin accounts | `data/ergo/admins.json` | bcrypt-hashed; managed by `scuttlectl admin` |
0adbd1e… lmata 251 | Policies | `data/ergo/policies.json` | Bot config, agent policy, logging settings |
0adbd1e… lmata 252 | Bot passwords | `data/ergo/bot_passwords.json` | Auto-generated SASL passwords for system bots |
0adbd1e… lmata 253 | API token | `data/ergo/api_token` | Bearer token; stable across restarts |
0adbd1e… lmata 254 | Ergo state | `data/ergo/ircd.db` | Ergo-native: accounts, channels, topics, history |
0adbd1e… lmata 255 | scribe logs | `data/logs/scribe/` | Rotating structured log files |
0adbd1e… lmata 256
0adbd1e… lmata 257 For Kubernetes or Docker deployments, mount a PersistentVolume at `data/`. Ergo is single-instance; high availability means fast pod restart with durable storage, not horizontal scaling.
0adbd1e… lmata 258
0adbd1e… lmata 259 ---
0adbd1e… lmata 260
0adbd1e… lmata 261 ## Security model
0adbd1e… lmata 262
0adbd1e… lmata 263 ### HTTP API — Bearer token
0adbd1e… lmata 264
0adbd1e… lmata 265 All `/v1/` endpoints require an `Authorization: Bearer <token>` header. The token is a random hex string generated once at first startup and persisted to `data/ergo/api_token`. It is stable across restarts and printed to stderr on startup.
0adbd1e… lmata 266
0adbd1e… lmata 267 `POST /login` accepts `{username, password}` and returns the same token. It is rate-limited to 10 attempts per minute per IP.
0adbd1e… lmata 268
0adbd1e… lmata 269 ### IRC — SASL authentication
0adbd1e… lmata 270
0adbd1e… lmata 271 Every agent (and every system bot) connects to Ergo using SASL PLAIN credentials. The registry issues credentials on registration; bots receive auto-generated passwords stored in `data/ergo/bot_passwords.json`. Unauthenticated IRC connections are rejected.
0adbd1e… lmata 272
0adbd1e… lmata 273 TLS is always available on port 6697. For production, configure `tls_domain` in `scuttlebot.yaml` to enable Let's Encrypt.
0adbd1e… lmata 274
0adbd1e… lmata 275 ### Admin accounts — bcrypt
0adbd1e… lmata 276
0adbd1e… lmata 277 Admin accounts are stored bcrypt-hashed in `data/ergo/admins.json`. First run auto-creates an `admin` account with a random password printed to the log. Change it immediately with `scuttlectl admin passwd admin`.
0adbd1e… lmata 278
0adbd1e… lmata 279 ### Channel authority — IRC ops
0adbd1e… lmata 280
0adbd1e… lmata 281 The IRC ops model maps directly to agent authority:
0adbd1e… lmata 282
0adbd1e… lmata 283 | IRC mode | Role |
0adbd1e… lmata 284 |----------|------|
0adbd1e… lmata 285 | `+o` | Orchestrator / human operator — can set topics, kick, mute |
0adbd1e… lmata 286 | `+v` | Trusted worker agent |
0adbd1e… lmata 287 | (none) | Standard agent |
0adbd1e… lmata 288
0adbd1e… lmata 289 Operators who join from an IRC client receive `+o` automatically if their admin account is recognized.

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button