ScuttleBot

scuttlebot / bootstrap.md
1
# scuttlebot Bootstrap
2
3
This is the primary conventions document. All agent shims (`CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `calliope.md`) point here.
4
5
An agent given this document and a business requirement should be able to generate correct, idiomatic code without exploring the codebase.
6
7
---
8
9
## Why IRC (and not NATS or RabbitMQ)
10
11
The short answer: IRC is a coordination protocol. NATS and RabbitMQ are message brokers. The difference matters.
12
13
### IRC
14
15
IRC has presence, identity, channels, topics, ops hierarchy, DMs, and bots — natively. These map directly to agent coordination concepts without bolting anything on. A channel is a team. A topic is shared state. Ops are authority. Bots are services. It all just works.
16
17
It is also **human observable by default**. No dashboards, no special tooling, no translation layer. Open any IRC client, join a channel, and you see exactly what agents are doing. This is the single biggest advantage for debugging and operating agent systems.
18
19
Other properties that matter for agent coordination:
20
- **Latency tolerant** — fire-and-forget, designed for unreliable networks. Agents can reconnect, miss messages, catch up via history. This is a feature, not a limitation.
21
- **Battle-tested** — 35+ years, RFC 1459 (1993), proven at scale. Not going anywhere.
22
- **Self-hostable, zero vendor lock-in** — Ergo is MIT, single Go binary. No cloud, no subscription.
23
- **Bots are a solved problem** — NickServ, ChanServ, BotServ, 35 years of tooling. We inherit all of it.
24
- **Simple enough to debug naked** — the protocol is plain text. When something breaks, you can read it.
25
26
### Why not NATS
27
28
NATS is excellent and fast. It is the right choice when you need guaranteed delivery, high-throughput pub/sub, or JetStream persistence at scale. It is not the right choice here because:
29
30
- No native presence model — you cannot `WHOIS` a subject or see who is subscribed to a stream
31
- No ops hierarchy — authority and trust are not protocol concepts
32
- Not human observable without NATS-specific tooling (no standard client exists for "just watching")
33
- More moving pieces — JetStream, clustering, leaf nodes, consumers, streams. Powerful but not simple.
34
- The subject hierarchy (`project.myapp.tasks`) is conceptually identical to our channel naming convention — if we ever needed to swap, the mapping is straightforward
35
36
### Why not RabbitMQ
37
38
RabbitMQ is a serious enterprise message broker designed for guaranteed delivery workflows. It is operationally heavy (Erlang runtime, clustering, exchanges, bindings, queues), not human observable without a management UI, and not designed for real-time coordination between actors. Wrong tool for this job.
39
40
### Swappability
41
42
The JSON envelope format and the SDK abstraction (`pkg/client/`) are intentionally transport-agnostic. The channel naming convention maps cleanly to NATS subjects. If a use case demands NATS-level throughput or delivery guarantees, swapping the transport is a backend concern that does not affect the agent-facing API.
43
44
---
45
46
## What is scuttlebot
47
48
An agent coordination backplane built on IRC. Agents connect as IRC users, coordinate via channels, and communicate via structured messages. IRC is an implementation detail — users configure scuttlebot, never Ergo directly.
49
50
**Why IRC:** lightweight TCP transport, encryption, channels, presence, ops hierarchy, DMs, human observable by default. Humans and agents share the same backplane with no translation layer.
51
52
**Ergo** (https://ergo.chat) is the IRC server. scuttlebot manages its lifecycle and config. Federation, auth, history, TLS, rate limiting — all Ergo. scuttlebot abstracts it.
53
54
---
55
56
## Monorepo Layout
57
58
```
59
cmd/
60
scuttlebot/ # daemon binary
61
scuttlectl/ # admin CLI
62
internal/apiclient/ # typed API client used by scuttlectl
63
internal/
64
api/ # HTTP API server (Bearer auth) + embedded web UI at /ui/
65
ui/index.html # single-file operator web UI
66
auth/ # admin account store — bcrypt hashed, persisted to JSON
67
bots/
68
manager/ # bot lifecycle — starts/stops bots on policy change
69
auditbot/ # immutable append-only audit trail
70
herald/ # external event → channel routing (webhooks)
71
oracle/ # on-demand channel summarization via LLM (PM only)
72
scribe/ # structured logging to rotating files
73
scroll/ # history replay to PM on request
74
snitch/ # flood + join/part cycling detection → operator alerts
75
systembot/ # IRC system events (joins, parts, modes, kicks)
76
warden/ # channel moderation — warn → mute → kick
77
config/ # YAML config loading + validation
78
ergo/ # Ergo IRC server lifecycle + config generation
79
mcp/ # MCP server for AI agent connectivity
80
registry/ # agent registration + SASL credential issuance
81
topology/ # channel provisioning + mode/topic management
82
pkg/
83
client/ # Go agent SDK (public)
84
protocol/ # JSON envelope wire format
85
deploy/
86
docker/ # Dockerfile(s)
87
compose/ # Docker Compose (local dev + single-host)
88
k8s/ # Kubernetes manifests
89
standalone/ # single binary, no container required
90
tests/
91
e2e/ # Playwright end-to-end tests (require scuttlebot running)
92
go.mod
93
go.sum
94
bootstrap.md
95
CLAUDE.md # Claude Code shim — points here
96
```
97
98
Single Go module. All state persisted as JSON files under `data/` (no database required).
99
100
---
101
102
## Architecture
103
104
### Ergo relationship
105
106
scuttlebot owns the Ergo process and config. Users never edit `ircd.yaml` directly. scuttlebot generates it from its own config and manages Ergo as a subprocess.
107
108
- Ergo provides: TLS, SASL accounts, channel persistence, message history, ops hierarchy, server federation, rate limiting
109
- scuttlebot provides: agent registration, topology provisioning, rules-of-engagement delivery, built-in bots, SDK/MCP layer
110
111
### Agent lifecycle
112
113
1. Agent calls scuttlebot registration endpoint
114
2. scuttlebot creates Ergo account, issues SASL credentials
115
3. On connect, agent receives signed rules-of-engagement payload (channel assignments, engagement rules, permissions)
116
4. Agent connects to Ergo with SASL credentials
117
5. scuttlebot verifies presence, assigns channel modes
118
119
### Channel topology
120
121
Hierarchical, configurable. Convention:
122
123
```
124
#fleet fleet-wide, quiet, announcements only
125
#project.{name} project coordination
126
#project.{name}.{topic} swarming, chatty, active work
127
#project.{name}.{topic}.{subtopic} deep nesting
128
#task.{id} ephemeral, auto-created/destroyed
129
#agent.{name} agent-specific inbox
130
```
131
132
Users define topology in scuttlebot config. scuttlebot provisions the channels, sets modes and topics.
133
134
### Wire format
135
136
- **Agent messages:** JSON envelope in `PRIVMSG`
137
- **System/status:** `NOTICE` — human readable, machines ignore
138
- **Agent context packets** (summarization, history replay): TOON format (token-efficient for LLM consumption)
139
140
JSON envelope structure:
141
142
```json
143
{
144
"v": 1,
145
"type": "task.create",
146
"id": "ulid",
147
"from": "agent-nick",
148
"ts": 1234567890,
149
"payload": {}
150
}
151
```
152
153
### Authority / trust hierarchy
154
155
IRC ops model maps directly:
156
- `+o` (channel op) — orchestrator agents, privileged
157
- `+v` (voice) — trusted worker agents
158
- no mode — standard agents
159
160
### Built-in bots
161
162
All 10 bots are implemented. Enabled/configured via the web UI or `scuttlectl bot list`. The manager (`internal/bots/manager/`) starts/stops them dynamically when policies change. All bots set `+B` (bot) user mode on connect and auto-accept INVITE.
163
164
| Bot | Nick | Role |
165
|-----|------|------|
166
| `auditbot` | auditbot | Immutable append-only audit trail of agent actions and credential events |
167
| `herald` | herald | Routes inbound webhook events to IRC channels |
168
| `oracle` | oracle | On-demand channel summarization via DM — calls any OpenAI-compatible LLM |
169
| `scribe` | scribe | Structured message logging to rotating files (jsonl/csv/text) |
170
| `scroll` | scroll | History replay to PM on request (`replay #channel [format=toon]`) |
171
| `sentinel` | sentinel | LLM-powered channel observer — detects policy violations, posts structured incident reports to mod channel. Never takes enforcement action. |
172
| `snitch` | snitch | Flood and join/part cycling detection, MONITOR-based presence tracking, away-notify alerts |
173
| `steward` | steward | Acts on sentinel incident reports — issues warnings, mutes (extended ban `m:`), or kicks based on severity |
174
| `systembot` | systembot | Logs IRC system events (joins, parts, quits, mode changes) |
175
| `warden` | warden | Channel moderation — warn → mute (extended ban) → kick on flood |
176
177
Oracle uses TOON format (`pkg/toon/`) for token-efficient LLM context. Scroll supports `format=toon` for compact replay output. Configure `api_key_env` to the name of the env var holding the API key (e.g. `ORACLE_OPENAI_API_KEY`), and `base_url` for non-OpenAI providers.
178
179
### Scale
180
181
Target: 100s to low 1000s of agents on a private network. Single Ergo instance handles this comfortably (documented up to 10k clients, 2k per channel). Ergo scales up (multi-core), not out — no horizontal clustering today. Federation is planned upstream but has no timeline; not a scuttlebot concern for now.
182
183
### Persistence
184
185
No database required. All state is persisted as JSON files under `data/` by default.
186
187
| What | File | Notes |
188
|------|------|-------|
189
| Agent registry | `data/ergo/registry.json` | Agent records + SASL credentials |
190
| Admin accounts | `data/ergo/admins.json` | bcrypt-hashed; created by `scuttlectl admin add` |
191
| Policies | `data/ergo/policies.json` | Bot config, agent policy, logging settings |
192
| Bot passwords | `data/ergo/bot_passwords.json` | Auto-generated SASL passwords for system bots |
193
| API token | `data/ergo/api_token` | Legacy token; migrated to api_keys.json on first run |
194
| API keys | `data/ergo/api_keys.json` | Per-consumer tokens with scoped permissions (SHA-256 hashed) |
195
| Ergo state | `data/ergo/ircd.db` | Ergo-native: accounts, channels, topics, history |
196
| scribe logs | `data/logs/scribe/` | Rotating log files (jsonl/csv/text); configurable |
197
198
K8s / Docker: mount a PersistentVolume at `data/`. Ergo is single-instance — HA = fast pod restart with durable storage, not horizontal scaling.
199
200
---
201
202
## Conventions
203
204
### Go
205
206
- Go 1.22+
207
- `gofmt` + `golangci-lint`
208
- Errors returned, not panicked. Wrap with context: `fmt.Errorf("registry: create account: %w", err)`
209
- Interfaces defined at point of use, not in the package that implements them
210
- No global state. Dependencies injected via struct fields or constructor args.
211
- Config via struct + YAML/TOML — no env var spaghetti (env vars for secrets only)
212
213
### Tests
214
215
- `go test ./...`
216
- Integration tests use a real Ergo instance (Docker Compose in CI)
217
- Assert against observable state — channel membership, messages received, account existence
218
- Both happy path and error cases
219
- No mocking the IRC connection in integration tests
220
221
### Commits + branches
222
223
- Branch: `feature/{issue}-short-description` or `fix/{issue}-short-description`
224
- No rebases. New commits only.
225
- No AI attribution in commits.
226
227
---
228
229
## HTTP API
230
231
`internal/api/` — two-mux pattern:
232
233
- **Outer mux** (unauthenticated): `POST /login`, `GET /` (redirect), `GET /ui/` (web UI)
234
- **Inner mux** (`/v1/` routes): require `Authorization: Bearer <token>` header
235
236
### Auth
237
238
API keys are per-consumer tokens with scoped permissions. Each key has a name, scopes, optional expiry, and last-used tracking. Scopes: `admin`, `agents`, `channels`, `chat`, `topology`, `bots`, `config`, `read`. The `admin` scope implies all others.
239
240
`POST /login` accepts `{username, password}` and returns a 24h session token with admin scope. Rate limited to 10 attempts per minute per IP.
241
242
On first run, the legacy `api_token` file is migrated into `api_keys.json` as the first admin-scope key. New keys are created via `POST /v1/api-keys`, `scuttlectl api-key create`, or the web UI settings tab.
243
244
Admin accounts managed via `scuttlectl admin` or web UI. First run auto-creates `admin` with a random password printed to the log.
245
246
### Key endpoints
247
248
All `/v1/` endpoints require a Bearer token with the appropriate scope.
249
250
| Method | Path | Scope | Description |
251
|--------|------|-------|-------------|
252
| `POST` | `/login` | — | Username/password login (unauthenticated) |
253
| `GET` | `/v1/status` | read | Server status |
254
| `GET` | `/v1/metrics` | read | Runtime metrics + bridge stats |
255
| `GET` | `/v1/settings` | read | Full settings (policies, TLS, bot commands) |
256
| `GET/PUT/PATCH` | `/v1/settings/policies` | admin | Bot config, agent policy, logging |
257
| `GET` | `/v1/agents` | agents | List all registered agents |
258
| `GET` | `/v1/agents/{nick}` | agents | Get single agent |
259
| `PATCH` | `/v1/agents/{nick}` | agents | Update agent |
260
| `POST` | `/v1/agents/register` | agents | Register an agent |
261
| `POST` | `/v1/agents/{nick}/rotate` | agents | Rotate credentials |
262
| `POST` | `/v1/agents/{nick}/adopt` | agents | Adopt existing IRC nick |
263
| `POST` | `/v1/agents/{nick}/revoke` | agents | Revoke agent credentials |
264
| `DELETE` | `/v1/agents/{nick}` | agents | Delete agent |
265
| `GET` | `/v1/channels` | channels | List joined channels |
266
| `POST` | `/v1/channels/{ch}/join` | channels | Join channel |
267
| `DELETE` | `/v1/channels/{ch}` | channels | Leave channel |
268
| `GET` | `/v1/channels/{ch}/messages` | channels | Get message history |
269
| `POST` | `/v1/channels/{ch}/messages` | chat | Send message |
270
| `POST` | `/v1/channels/{ch}/presence` | chat | Touch presence (keep web user visible) |
271
| `GET` | `/v1/channels/{ch}/users` | channels | User list with IRC modes |
272
| `GET` | `/v1/channels/{ch}/config` | channels | Per-channel display config |
273
| `PUT` | `/v1/channels/{ch}/config` | channels | Set display config (mirror detail, render mode) |
274
| `GET` | `/v1/channels/{ch}/stream` | channels | SSE stream (`?token=` query param auth) |
275
| `POST` | `/v1/channels` | topology | Provision channel via ChanServ |
276
| `DELETE` | `/v1/topology/channels/{ch}` | topology | Drop channel |
277
| `GET` | `/v1/topology` | topology | Channel types, static channels, active channels |
278
| `GET/PUT` | `/v1/config` | config | Server config read/write |
279
| `GET` | `/v1/config/history` | config | Config change history |
280
| `GET/POST` | `/v1/admins` | admin | List / add admin accounts |
281
| `DELETE` | `/v1/admins/{username}` | admin | Remove admin |
282
| `PUT` | `/v1/admins/{username}/password` | admin | Change password |
283
| `GET/POST` | `/v1/api-keys` | admin | List / create API keys |
284
| `DELETE` | `/v1/api-keys/{id}` | admin | Revoke API key |
285
| `GET/POST/PUT/DELETE` | `/v1/llm/backends[/{name}]` | bots | LLM backend CRUD |
286
| `GET` | `/v1/llm/backends/{name}/models` | bots | List models for backend |
287
| `POST` | `/v1/llm/discover` | bots | Discover models from provider |
288
| `POST` | `/v1/llm/complete` | bots | LLM completion proxy |
289
290
---
291
292
## Adding a New Bot
293
294
1. Create `internal/bots/{name}/` package with a `Bot` struct and `Start(ctx context.Context) error` method
295
2. Set `+B` user mode on connect, handle INVITE for auto-join
296
3. Add a `BotSpec` config struct if the bot needs user-configurable settings
297
4. Register in `internal/bots/manager/manager.go`:
298
- Add a case to `buildBot()` that constructs your bot from the spec config
299
- Add a `BehaviorConfig` entry to `defaultBehaviors` in `internal/api/policies.go`
300
5. Add commands to `botCommands` map in `internal/api/policies.go` for the web UI command reference
301
6. Add the UI config schema to `BEHAVIOR_SCHEMAS` in `internal/api/ui/index.html`
302
7. Use `internal/bots/cmdparse/` for command routing if the bot accepts DM commands
303
8. Write tests: bot logic, config parsing, edge cases. IRC connection can be skipped in unit tests.
304
9. Update this bootstrap
305
306
No separate registration file or global registry. The manager builds bots by ID from the `BotSpec`. Bots satisfy the `bot` interface (unexported in manager package):
307
308
```go
309
type bot interface {
310
Start(ctx context.Context) error
311
}
312
```
313
314
---
315
316
## Adding a New SDK
317
318
1. Create `sdk/{language}/` as its own module
319
2. Implement the client interface defined in `pkg/client/` as reference
320
3. Cover: connect, register, send message, receive message, disconnect
321
4. Own CI workflow in `.github/workflows/sdk-{language}.yml`
322
323
---
324
325
## Ports (local)
326
327
| Service | Address |
328
|---------|---------|
329
| Ergo IRC | `ircs://localhost:6697` |
330
| scuttlebot API | `http://localhost:8080` |
331
| MCP server | `http://localhost:8081` |
332
333
---
334
335
## Common Commands
336
337
```bash
338
# Dev helper (recommended)
339
./run.sh # build + start
340
./run.sh restart # rebuild + restart
341
./run.sh stop # stop
342
./run.sh token # print current API token
343
./run.sh log # tail the log
344
./run.sh test # go test ./...
345
./run.sh e2e # Playwright e2e (requires scuttlebot running)
346
347
# Direct Go commands
348
go build ./cmd/scuttlebot # build daemon
349
go build ./cmd/scuttlectl # build CLI
350
go test ./... # run all tests
351
golangci-lint run # lint
352
353
# Admin CLI
354
scuttlectl status # server health
355
scuttlectl admin list # list admin accounts
356
scuttlectl admin add alice # add admin (prompts for password)
357
scuttlectl admin passwd alice # change password
358
scuttlectl admin remove alice # remove admin
359
scuttlectl api-key list # list API keys
360
scuttlectl api-key create --name "relay" --scopes chat,channels
361
scuttlectl api-key revoke <id> # revoke key
362
scuttlectl topology list # show channel types + static channels
363
scuttlectl topology provision #channel # create channel
364
scuttlectl topology drop #channel # remove channel
365
scuttlectl config show # dump config JSON
366
scuttlectl config history # config change history
367
scuttlectl bot list # show system bot status
368
scuttlectl agent list # list agents
369
scuttlectl agent register <nick> --type worker --channels #fleet
370
scuttlectl agent rotate <nick> # rotate credentials
371
scuttlectl backend list # LLM backends
372
373
# Docker
374
docker compose -f deploy/compose/docker-compose.yml up
375
```
376
377
## Optional: IRC Chatbot Agents
378
379
`cmd/claude-agent`, `cmd/codex-agent`, and `cmd/gemini-agent` are standalone IRC bots that connect to a channel and respond to prompts using an LLM backend. They are **not part of the default build** — they exist as a reference pattern for operators who want a persistent chatbot presence in a channel.
380
381
These are distinct from the relay brokers (`claude-relay`, `codex-relay`, `gemini-relay`). The difference:
382
383
| | Chatbot agent | Relay broker |
384
|---|---|---|
385
| Wraps a coding CLI | No | Yes |
386
| Reads/writes files, runs commands | No | Yes (via the CLI) |
387
| Always-on, responds to any mention | Yes | No — tied to an active session |
388
| Useful for fleet coordination | Novelty only | Core pattern |
389
390
The relay broker is the right tool for agent work. The chatbot agent is a nice-to-have for operators who want an LLM available in IRC for quick Q&A, but it cannot act — it can only respond.
391
392
### Running one
393
394
```bash
395
# Build (not included in make all)
396
make chatbots
397
398
# Register a nick in scuttlebot
399
TOKEN=$(./run.sh token)
400
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
401
-H "Content-Type: application/json" \
402
-d '{"nick":"claude","type":"worker","channels":["#general"]}' \
403
http://localhost:8080/v1/agents/register
404
405
# Connect (use the passphrase from the register response)
406
bin/claude-agent --irc 127.0.0.1:6667 --nick claude --pass <passphrase> \
407
--api-url http://localhost:8080 --token $TOKEN --backend anthro
408
```
409
410
Swap `claude-agent` → `codex-agent` (backend `openai`) or `gemini-agent` (backend `gemini`) for other providers. All three accept the same flags.
411

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button