# GenSwarms — full documentation

> Concatenated from the GenSwarms repository for LLM ingestion.
> Source: https://github.com/genlayerlabs/genswarms

---

---
name: operating-genswarms
description: Operate and orchestrate genswarms agent swarms — author swarm configs (.exs/.json/.yaml), build the CLI, and start/manage/observe/scale swarms via the genswarms CLI or REST API. Use when working in a genswarms repo, defining or running a swarm, wiring agent topologies/backends/skills, or driving swarms over HTTP/WebSocket. Do NOT use for developing the genswarms Elixir codebase itself, or for unrelated multi-agent frameworks.
---

# genswarms

GenSwarms is an Elixir/OTP orchestrator for swarms of `subzeroclaw` agents. A swarm is a
declarative set of **agents** (pluggable backends: Local / Docker / SSH / Bwrap / Mock),
optional non-agentic **objects**, and a directed-graph **topology** connecting them. Swarms run
as independent daemon processes; an optional Phoenix server exposes a JSON REST API + WebSocket.
The CLI, API, and daemons coordinate through SQLite at `.genswarms/swarms.db`.

Full docs live in [`docs/`](docs/README.md) — link out rather than duplicating. This skill is the
how-to-operate quick path.

## Prerequisites & install

Requires **Elixir 1.14+** and **Erlang/OTP 27+** (Nix dev shell pins Elixir 1.17 / OTP 27 / Node 20).
Node 20 is only needed to build agent container images.

```bash
nix develop              # recommended dev shell (or install Elixir/OTP yourself)
mix deps.get
mix escript.build        # produces ./genswarms escript binary
cp genswarms ~/.local/bin/   # optional: put it on PATH
```

Every subcommand also runs via Mix: `genswarms status` == `mix genswarms.status`
(hyphens become underscores, e.g. `list-skills` → `mix genswarms.list_skills`).
See [docs/getting-started.md](docs/getting-started.md).

Only `SUBZEROCLAW_API_KEY` is required to run real agents. The CLI auto-loads `.env` from the
working directory (searching up to 5 parents). Key env vars (full list in getting-started.md):

| Var | Purpose | Default |
|-----|---------|---------|
| `SUBZEROCLAW_API_KEY` | LLM provider key (required) | — |
| `SUBZEROCLAW_MODEL` | Default agent model | `anthropic/claude-sonnet-4` |
| `SUBZEROCLAW_MOCK_SCRIPT` | Run real agents w/o LLM (canned responses) | — |
| `PORT` | API server port | `4000` |
| `SWARM_API_URL` | Base URL CLI uses to reach the server | `http://localhost:4000` |
| `GENSWARMS_API_TOKEN` | Bearer token securing the REST + WebSocket API | — (unset = loopback-only) |

## Core workflow

Copy this checklist and check off each step as you go:

```
Swarm bring-up:
- [ ] 1. Scaffold a project        (genswarms init)
- [ ] 2. Set SUBZEROCLAW_API_KEY   (cp .env.example .env)
- [ ] 3. Validate the config       (genswarms config validate)
- [ ] 4. Start the API server      (genswarms up — optional)
- [ ] 5. Start the swarm daemon    (genswarms start)
- [ ] 6. Inspect status            (genswarms status)
- [ ] 7. Task an agent             (genswarms task)
- [ ] 8. Observe logs/events       (genswarms logs -f / events)
- [ ] 9. Stop                      (genswarms stop / down)
```

```bash
# 1. Scaffold a project (swarms/, skills/, docker/, .env.example, .genswarms/)
genswarms init my-project && cd my-project
cp .env.example .env            # add SUBZEROCLAW_API_KEY

# 2. Validate the config before running it
genswarms config validate swarms/example_swarm.exs

# 3. Start the API server (background) — optional but enables HTTP/WS + restart-agent
genswarms up                    # --port N to override; --foreground to run inline

# 4. Start the swarm as a daemon (name comes from the config's name:, not the filename)
genswarms start swarms/example_swarm.exs

# 5. Inspect
genswarms status                # server + all swarms
genswarms status example-swarm  # agents, objects, topology, backends, skills

# 6. Task an agent (queued in SQLite, daemon polls every 500ms)
genswarms task example-swarm researcher "Find papers on transformers"

# 7. Observe
genswarms logs example-swarm -f         # follow live logs (use this for live streaming)
genswarms events --errors               # one-shot error query

# 8. Stop
genswarms stop example-swarm            # one swarm
genswarms down                          # all swarms + the API server
```

Walkthrough: [docs/getting-started.md](docs/getting-started.md).

## Minimal swarm config

A config is a map (`.exs`, `.json`, or `.yaml`). The swarm name comes from `name:`.

```elixir
%{
  name: "example-swarm",
  agents: [
    %{name: :researcher, backend: :local, skills: ["web.md"], model: "anthropic/claude-sonnet-4"},
    %{name: :coder, backend: {:docker, "coder"}, skills: ["code.md"], presets: [:base, :code]}
  ],
  objects: [
    # optional deterministic, non-agentic components
    %{name: :evaluator, handler: MyApp.Objects.Evaluator, config: %{}}
  ],
  topology: [
    {:researcher, :coder},   # directed edge: researcher may send to coder
    {:coder, :evaluator},
    {:evaluator, :researcher}
  ]
}
```

Key facts (see [docs/configuration.md](docs/configuration.md)):
- `name` and `agents` (non-empty) are required; `objects` and `topology` default to `[]`.
- `backend` defaults to `:bwrap` if omitted. Forms: `:local`, `{:docker, "name"}`,
  `{:docker, "name", %{opts}}`, `{:ssh, "user@host"}`, `{:ssh, "user@host", %{opts}}`,
  `:bwrap`, `{:bwrap, %{opts}}`, `:mock`, `{:mock, %{script: [...]}}`.
- **JSON/YAML only support scalar-string backends** (`"local"`, `"bwrap"`, `"mock"`). Docker/SSH
  and option-map backends require `.exs`.
- Topology is directional — declare both directions for two-way comms. System objects
  `:metrics`, `:tick`, `:gateway` are always routable without an edge.
- There is **no `count:` key** — one agent def == one agent. Run pools via runtime scaling.
- For bwrap agents, `config:` is split into backend keys (`workspace`, `extra_path`,
  `extra_ro_binds`, `extra_env`, `memory_limit`, `cpu_shares`, `tasks_max`, `subzeroclaw_path`,
  `presets`) and domain keys (anything else, passed through to the agent).

Backends in depth: [docs/backends.md](docs/backends.md). Objects: [docs/objects.md](docs/objects.md).
Driving swarms from Elixir directly: [docs/programmatic.md](docs/programmatic.md).

## Essential CLI commands

These are dispatched by the `genswarms` **escript binary** (verified against `lib/genswarms/cli.ex`):

| Command | Purpose |
|---------|---------|
| `genswarms init [dir]` | Scaffold a project (`--force` to overwrite) |
| `genswarms up` / `down` | Start API server / stop server + swarms (`up` = `dashboard start`) |
| `genswarms dashboard [start\|stop\|status]` | Manage the API/dashboard server explicitly |
| `genswarms start <config>` | Start a swarm daemon (`--foreground` to run inline) |
| `genswarms stop <name>` | Stop a swarm daemon |
| `genswarms restart <name>` | Restart, reloading config (`--delete` for clean slate) |
| `genswarms status [name]` | Server + swarm status |
| `genswarms task <swarm> <agent> <msg>` | Send a task to an agent |
| `genswarms msg <swarm> <from> <to> <msg>` | Route a message (validated against topology) |
| `genswarms logs [swarm] [agent]` | View/stream logs (`-f` follow, `--tail N`, `--stdout`, `--events`, `--all`) |
| `genswarms events` | One-shot event query (`--errors`, `-s <swarm>`, `-a <agent>`, `--category`, `--type`, `--limit`) |
| `genswarms scale <swarm> <base> <n>` | Scale an agent group to N (`base_1`, `base_2`, …) |
| `genswarms overlay <swarm>` / `snapshot <swarm>` | Inspect/clear runtime overlay; emit effective `.exs` |
| `genswarms config validate <file>` (alias `check`) | Validate a config |
| `genswarms list-skills` | List available skills |
| `genswarms build [img\|--all]` | Build agent container images via Nix |
| `genswarms env [list\|get\|set\|unset]` | Manage `.env` variables |

**Mix-task only** (NOT escript subcommands — `genswarms clean`/`pause`/etc. error with "Unknown command"):

| Command | Purpose |
|---------|---------|
| `mix genswarms.pause <name>` / `mix genswarms.resume <name>` | Freeze / unfreeze the swarm's Docker containers |
| `mix genswarms.delete <name>` | Delete a swarm + all its data (`--force` to skip prompt) |
| `mix genswarms.clean [--all]` | Remove stopped/crashed swarms (`--all` also clears events) |
| `mix genswarms.restart_agent <swarm> <agent>` | Restart one agent (requires the API server) |

Note: `genswarms events --follow`/`--stats` are silent no-ops today — use `genswarms logs <swarm> -f`
for live streaming. Full reference: [docs/cli.md](docs/cli.md).

## Driving via REST API

Server runs at `http://localhost:4000` (JSON only). Start it with `genswarms up`.
Many runtime ops (add/remove agents, edit topology, restart-agent, edit skills) are API-only.
When `GENSWARMS_API_TOKEN` is set, send `Authorization: Bearer $GENSWARMS_API_TOKEN`
on every request (the CLI does this automatically); unset, the API accepts
loopback callers only. See [Security](docs/security.md) before exposing the server.

```bash
curl http://localhost:4000/api/swarms                          # list swarms

curl -X POST http://localhost:4000/api/swarms \                # create from server-side config
  -H 'Content-Type: application/json' \
  -d '{"config_path": "swarms/example_swarm.exs"}'             # or {"config": {...}} inline

curl -X POST http://localhost:4000/api/swarms/example-swarm/agents/researcher/task \
  -H 'Content-Type: application/json' \
  -d '{"task": "Summarize the latest results."}'

curl -X POST http://localhost:4000/api/swarms/example-swarm/agents \   # add + wire a live agent
  -H 'Content-Type: application/json' \
  -d '{"name": "reviewer", "backend": {"type": "docker", "image": "code"}, "incoming": ["coder"]}'
```

Other useful routes: `GET /api/swarms/:name` (detailed status), `GET .../topology`,
`PATCH .../topology` (`{"add":[...],"remove":[...]}`), `POST .../agents/:base/scale` (`{"count":N}`),
`GET /api/events` (filterable), `POST /api/config/validate`, `GET /api/skills`. Real-time streaming is
over the WebSocket channel `swarm:{name}` at `/swarm`. Full reference:
[docs/rest-api.md](docs/rest-api.md) and [WebSocket](docs/websocket.md).

## Defining skills for agents

Skills are plain markdown files (no schema) that become an agent's instructions. Reference them by
the `skills:` list in config; a simple filename resolves against the skills dir (`priv/skills`, set by
`SKILLS_DIR`), and `./`-relative or absolute paths are also accepted.

Three template variables are substituted per agent at deploy time, so one file serves a whole pool:
`{{agent_name}}`, `{{swarm_name}}`, `{{workspace}}`.

```markdown
# Planner Skill
You are {{agent_name}}, the planner for {{swarm_name}}.
Break tasks into steps and delegate them with @agent_name: prefixes.
Write output to {{workspace}}.
```

Built-ins: `web.md`, `code.md`, `review.md`, `swarm_architect.md`, `swarm-fixer.md`, `secret.md`.
List/read/edit deployed skills over the API (`GET /api/skills`, `PUT .../agents/:agent/skills/:skill`).
Authoring details: [docs/skills.md](docs/skills.md). Inter-agent messaging (`@agent:`, `@all:`,
`.inbox/`/`.outbox/`): [docs/messaging.md](docs/messaging.md).

## Common gotchas

- **`genswarms clean` is not a real escript command** — it errors. Use `mix genswarms.clean` or
  `POST /api/swarms/clean`. Same for `pause`/`resume`/`delete`/`restart-agent` (Mix-task only).
- **Swarm name ≠ filename.** `swarms/example_swarm.exs` defines a swarm named `example-swarm` (from
  `name:`). Use the file path to `start`/`validate`, the hyphenated name everywhere else.
- **Agent won't start:** confirm the `subzeroclaw` binary is reachable (config → `../subzeroclaw/subzeroclaw`
  → `SUBZEROCLAW_PATH` → PATH) and `SUBZEROCLAW_API_KEY` is set (or `SUBZEROCLAW_MOCK_SCRIPT` for no-LLM).
- **Messages not routing:** the topology must permit `source -> target` (directional). Check the agent
  emits correct `@agent:` syntax; inspect `GET /api/swarms/:name/messages`.
- **`:mock` backend is a no-op stub** (spawns nothing, no responses). To run *real* agents without an
  LLM, set `SUBZEROCLAW_MOCK_SCRIPT` instead.
- **Tasks to daemon swarms are async** — queued in `.genswarms/swarms.db` (`tasks` table), polled every
  500ms. Confirm the daemon is up (`genswarms status`) and check `genswarms events --category agent`.
- **Docker agents** are named `szc-{swarm}-{agent}` and run with `--rm` (no container left on crash —
  check `genswarms events --category backend`).

More: [docs/troubleshooting.md](docs/troubleshooting.md) and [docs/observability.md](docs/observability.md).

---

---
description: Documentation for GenSwarms — the declared Elixir/OTP runtime for swarms of AI agents. Start here, then jump to configuration, the CLI, backends, and the API.
---

# GenSwarms documentation

GenSwarms is an Elixir/OTP orchestrator for swarms of agents, including
`subzeroclaw` workers and persistent interactive coding clients, with pluggable
backends, arbitrary directed-graph topologies, per-agent skills, and fault
tolerance via OTP supervision trees.

This is the documentation index. If you are new, start with
[Getting started](getting-started.md) and work through the area you need.

## Entry points

- [`README.md`](https://github.com/genlayerlabs/genswarms#readme) (repo root) — project overview and quick start.
- [`SKILL.md`](https://github.com/genlayerlabs/genswarms/blob/main/SKILL.md) (repo root) — the how-to-operate quick path for driving swarms via the `genswarms` CLI and REST API.
- [Getting started](getting-started.md) — the full first-swarm walkthrough.

## Getting started

- [Getting started](getting-started.md) — install, build the `genswarms` CLI, run your first swarm, and the key environment variables.
- [Configuration](configuration.md) — the swarm config DSL: agents, objects, topology, backends, and the `.exs` / `.json` / `.yaml` formats.
- [CLI reference](cli.md) — every `genswarms` / `mix genswarms.*` command with its flags and examples.

## Operating

- [Messaging](messaging.md) — the `@agent:` syntax, topology-gated routing, file inbox/outbox, and the `swarm-msg` helper.
- [Skills](skills.md) — per-agent markdown skills and the `{{agent_name}}` / `{{swarm_name}}` / `{{workspace}}` template variables.
- [Objects](objects.md) — non-agentic Elixir components that participate in the topology and run deterministic code.
- [Observability](observability.md) — the event spine: events, logging, telemetry, and streaming.
- [Testing and development](testing.md) — ExUnit unit tests, the e2e harness (`mix genswarms.test`), and the Mock backend.
- [Troubleshooting](troubleshooting.md) — common problems with startup, routing, backends, tasks, and the API server.

## Architecture and internals

- [Architecture](architecture.md) — the OTP supervision tree, the daemon model, and supported deployment topologies.
- [Backends](backends.md) — Local, persistent tmux TUIs, Docker, Apple container, SSH, Bwrap, and Mock execution backends and their shared contract.
- [Containers and sandboxes](containers.md) — NixOS container images, tool presets, and the bwrap sandbox internals.

## Reference

- [REST API](rest-api.md) — the complete pure-JSON HTTP API served by Phoenix.
- [WebSocket API](websocket.md) — real-time agent output, message routing, and event streams over the `swarm:{name}` channel.
- [Programmatic API](programmatic.md) — driving GenSwarms directly as an Elixir library.

---

---
description: Install GenSwarms, build the CLI, set your API key, and run your first AI agent swarm — with the full bring-up checklist and key environment variables.
---

# Getting started

This guide walks you through installing GenSwarms, building the `genswarms` CLI, and running your first swarm end to end. By the end you will have an API server running, a swarm started from config, and a task delivered to an agent.

## Prerequisites

GenSwarms is an Elixir/OTP application. You can get the toolchain either through the bundled Nix dev shell or by installing the runtimes yourself.

### With Nix (recommended)

The flake's dev shell pins the exact versions used by the project:

| Tool | Version |
|------|---------|
| Elixir | 1.17 |
| Erlang/OTP | 27 |
| Node.js | 20 |

```bash
nix develop
```

The dev shell also provides `git`, `inotify-tools`, and `colmena` (for bare-metal deploys), and sets `MIX_HOME`/`HEX_HOME` inside the project directory.

### Without Nix

Install the runtimes manually:

- Elixir 1.14+ (`mix.exs` requires `~> 1.14`; the Nix dev shell pins 1.17)
- Erlang/OTP 27+

Node.js 20 is only needed if you build agent container images.

## Install dependencies

From the project root:

```bash
mix deps.get
```

## Build the CLI

GenSwarms ships a single `genswarms` CLI, built as an Elixir escript:

```bash
mix escript.build
```

This produces a `./genswarms` binary. Install it somewhere on your `PATH`:

```bash
sudo cp genswarms /usr/local/bin/   # system-wide
cp genswarms ~/.local/bin/          # user-local (ensure ~/.local/bin is on PATH)
```

### Running commands through Mix instead of the binary

Most CLI commands are also available through Mix as `mix genswarms.<cmd>`, if you
prefer not to install the binary. The escript dispatches each command to a
matching `Mix.Tasks.Genswarms.*` task (see [`lib/genswarms/cli.ex`](https://github.com/genlayerlabs/genswarms/blob/main/lib/genswarms/cli.ex)).

For hyphenated commands the Mix task name uses an underscore, not a hyphen:

| CLI command | Mix task |
|-------------|----------|
| `genswarms start` | `mix genswarms.start` |
| `genswarms status` | `mix genswarms.status` |
| `genswarms task` | `mix genswarms.task` |
| `genswarms list-skills` | `mix genswarms.list_skills` |

> Note: `genswarms up` and `genswarms down` are convenience aliases. `up` maps to
> `mix genswarms.dashboard start`, and `down` maps to `mix genswarms.down`.
> A few task modules (such as `mix genswarms.restart_agent`) exist only as Mix
> tasks and are not wired as top-level `genswarms` subcommands.

## Quick start

### 1. Create a project

Scaffold a new project directory with example configs and an `.env.example`:

```bash
genswarms init my-project
cd my-project
```

This creates `swarms/example_swarm.exs` (a two-agent demo), example skills under
`skills/`, a `docker/` directory, and a `.genswarms/` runtime directory.

> Note: `genswarms init` prints a "Next steps" hint that still uses the legacy
> binary name `swarm` (e.g. `swarm up`, `swarm start ...`). Use `genswarms`
> instead — the commands are otherwise identical.

### 2. Configure your environment

Copy the example file and add your LLM provider API key:

```bash
cp .env.example .env
# edit .env and set SUBZEROCLAW_API_KEY
```

The CLI auto-loads `.env` from the current directory (searching up to 5 parent
directories) on every invocation; you can also `source .env` yourself. To confirm
which file was loaded, run with `SWARM_DEBUG` set — the CLI prints
`Loaded environment from <path>`.

### 3. Start the API server

The Phoenix API server runs in the background and exposes the REST API and WebSocket:

```bash
genswarms up
genswarms status
```

The server listens on `http://localhost:4000` by default (override the port with
`PORT` or `genswarms up --port <n>`).

### 4. Start a swarm

Start a swarm from a config file. Swarms run as independent daemon processes, separate from the API server:

```bash
genswarms start swarms/example_swarm.exs
```

Check its status:

```bash
genswarms status                 # server + all swarms
genswarms status example-swarm   # one swarm in detail
```

> The swarm's name (`example-swarm`) comes from the `name:` field inside the
> config, not the filename.

### 5. Send a task to an agent

```bash
genswarms task example-swarm researcher "find papers on transformers"
```

Tasks sent to a daemon swarm are queued in SQLite and picked up by the daemon, which polls the queue every 500ms.

### 6. View logs and status

```bash
genswarms logs example-swarm              # all agents
genswarms logs example-swarm researcher   # one agent
genswarms logs example-swarm -f           # follow mode
```

### 7. Stop things

```bash
genswarms stop example-swarm   # stop one swarm
genswarms down                 # stop all swarms and the API server
```

## Local data directory

GenSwarms keeps cross-process state under `.genswarms/` in the project directory, including the SQLite database at `.genswarms/swarms.db` plus events and logs. This is where swarm state, the task queue, and the event log live.

## Environment variables

Set these in `.env` (or your shell). Only `SUBZEROCLAW_API_KEY` is required to run real agents. The defaults below are the values read by the code; entries marked "(production release)" only apply to the bundled release runtime config, not the CLI dev path. The **model is set per agent** (the `model` config key, or `request_extra` for router routing) — there is no `SUBZEROCLAW_MODEL` environment variable; `subzeroclaw` dropped it.

| Variable | Description | Default |
|----------|-------------|---------|
| `SUBZEROCLAW_API_KEY` | API key for the LLM provider (OpenRouter, Anthropic, OpenAI, etc.) | - |
| `SUBZEROCLAW_ENDPOINT` | API endpoint URL passed through to the agent backend | Provider default (from `subzeroclaw`) |
| `SUBZEROCLAW_PATH` | Path to the `subzeroclaw` binary | `subzeroclaw` (resolved from PATH) |
| `SUBZEROCLAW_SRC` | Source directory mounted/built into Docker and Apple container agents | `../subzeroclaw` |
| `SUBZEROCLAW_MOCK_SCRIPT` | Path to a mock script JSON; passed through to the bwrap sandbox so the agent returns canned responses instead of calling the LLM | - |
| `SUBZEROCLAW_RECORD_SCRIPT` | Path passed through to the bwrap sandbox to record agent interactions for later mock replay | - |
| `SWARM_DATA_DIR` | Swarm data directory (`:swarm_data_dir` app config) | `~/.subzeroclaw/swarms` |
| `SKILLS_DIR` | Skills directory (`:skills_dir` app config) | `priv/skills` |
| `SWARM_TOPOLOGY` | (Container only) comma-separated targets for `swarm-msg list`; set automatically per agent | Auto-set |
| `SWARM_API_URL` | Base URL the CLI uses to reach the API server | `http://localhost:4000` |
| `SWARM_DEBUG` | When set, the CLI prints the loaded `.env` path on startup | - |
| `PORT` | HTTP port for the API server | `4000` |
| `SECRET_KEY_BASE` | Phoenix secret key (production release; required, no default) | - |
| `PHX_HOST` | Phoenix host. CLI/dev path defaults to `localhost`; the production release runtime defaults to `example.com` | `localhost` (CLI) / `example.com` (production release) |

## See also

- [Configuration](configuration.md)
- [CLI reference](cli.md)
- [Backends](backends.md)
- [Troubleshooting](troubleshooting.md)

---

---
description: The GenSwarms swarm configuration DSL: declare agents, deterministic objects, topology edges, and backends in .exs, .json, or .yaml.
---

# Swarm configuration

The Swarm Configuration DSL is the declarative format that defines a swarm: its name, its agents and objects, and the topology that connects them. Configs can be written as Elixir (`.exs`), JSON (`.json`), or YAML (`.yaml`/`.yml`) and are validated by `Genswarms.Config.SwarmConfig` when loaded.

## Top-level structure

A configuration is a map with the following keys.

| Key | Type | Required | Description |
|-----|------|----------|-------------|
| `name` | string or atom | Yes | Unique swarm identifier. Must start with a letter and contain only alphanumerics, `_`, or `-`. Atoms are converted to strings. |
| `agents` | list of maps | Yes | One or more agent definitions. Must be non-empty. |
| `objects` | list of maps | No | Non-agentic Elixir/backend components. Defaults to `[]`. |
| `topology` | list of `{from, to}` tuples | No | Directed communication edges. Defaults to `[]`. |
| `skills_base_dir` | string | No | Base directory to resolve skill files from. Stored on the parsed struct; not otherwise validated. |
| `options` | map | No | Free-form additional settings. Defaults to `%{}`. |

```elixir
%{
  name: "example-swarm",
  agents: [
    %{name: :researcher, backend: :local, skills: ["web.md"]},
    %{name: :coder, backend: {:docker, "coder"}, skills: ["code.md"]}
  ],
  objects: [],
  topology: [
    {:researcher, :coder},
    {:coder, :researcher}
  ]
}
```

## Agent configuration

Each entry in `agents` is a map. Only the keys below are recognized by the validator and serializer; unknown keys are passed through into the agent map but are not validated.

| Key | Type | Required | Default | Description |
|-----|------|----------|---------|-------------|
| `name` | atom or string | Yes | — | Unique agent identifier. Strings are normalized to atoms. |
| `backend` | backend spec | No | `:bwrap` | Where/how the agent runs (see [Backend value forms](#backend-value-forms)). If omitted, the parser fills in `:bwrap`. |
| `model` | string | No | — | LLM model (e.g. `provider/model-name`). Passed through to `subzeroclaw` as `SUBZEROCLAW_REQUEST_EXTRA = {"model": …}`. There is **no `SUBZEROCLAW_MODEL` env fallback** (it is a dead variable). When unset, `subzeroclaw` uses its own default. For router routing, set `request_extra` (a `policy_ir`) instead of `model`. |
| `endpoint` | string | No | — | API endpoint URL. When unset, the backend passes through `SUBZEROCLAW_ENDPOINT`; otherwise `subzeroclaw` auto-detects from the API key. |
| `skills` | list of strings | No | `[]` | Skill markdown filenames to deploy. All entries must be strings. |
| `presets` | list of atoms | No | `[]` | NixOS tool presets. Must be drawn from the valid preset set below. |
| `tools` | list of atoms | No | `[]` | Individual tools. Must be drawn from the valid tool set below. |
| `config` | map | No | `%{}` | Backend-specific and domain-specific configuration (see [Backend config separation](#backend-config-separation)). |

```elixir
%{
  name: :researcher,
  backend: :local,
  model: "anthropic/claude-sonnet-4",
  endpoint: "https://openrouter.ai/api/v1/chat/completions",
  skills: ["web.md", "summarize.md"],
  presets: [:base, :web]
}
```

### Valid presets

Presets are validated against the `@valid_presets` set in `Genswarms.Config.SwarmConfig` (the corresponding NixOS package sets live in `nix/tool-presets.nix`):

`:base`, `:web`, `:code`, `:python`, `:node`, `:data`, `:docs`, `:network`, `:system`, `:security`, `:containers`, `:cloud`, `:ai`

A `presets` value that is not a list of atoms fails with `:invalid_presets_format`.

### Valid tools

Individual tools are validated against the complete `@valid_tools` set in `Genswarms.Config.SwarmConfig`:

`:git`, `:curl`, `:wget`, `:jq`, `:yq`, `:tree`, `:htop`, `:ripgrep`, `:rg`, `:fd`, `:fzf`, `:ag`, `:vim`, `:neovim`, `:nano`, `:python`, `:python3`, `:node`, `:nodejs`, `:ruby`, `:go`, `:rustc`, `:cargo`, `:make`, `:cmake`, `:gcc`, `:clang`, `:sqlite`, `:postgresql`, `:mysql`, `:redis`, `:duckdb`, `:pandoc`, `:pdftotext`, `:ssh`, `:rsync`, `:netcat`, `:httpie`, `:docker`, `:podman`, `:kubectl`, `:gh`, `:glab`, `:miller`, `:csvkit`, `:xsv`, `:ffmpeg`, `:imagemagick`, `:pytest`, `:ruff`, `:mypy`, `:black`, `:flake8`, `:pip`, `:poetry`, `:uv`.

A `tools` value that is not a list of atoms fails with `:invalid_tools_format`. Atoms outside the set above fail with `{:unknown_tools, invalid}` (where `invalid` is the list of offending atoms); likewise unknown presets fail with `{:unknown_presets, invalid}`.

## Backend value forms

The `backend` key accepts any of the following forms. See [backends.md](backends.md) for behavioral details.

| Form | Backend | Notes |
|------|---------|-------|
| `:local` | Local | Runs as an Elixir Port subprocess. |
| `{:docker, "name"}` | Docker | Container image/name; resolves to `%{image: "name"}`. |
| `{:docker, "name", %{opts}}` | Docker | Options map merged over `%{image: "name"}`. |
| `:apple_container` | Apple container | Apple `container` CLI backend; image is picked from presets/defaults. |
| `{:apple_container, "image"}` | Apple container | Container image; resolves to `%{image: "image"}`. |
| `{:apple_container, "image", %{opts}}` | Apple container | Options map merged over `%{image: "image"}`. |
| `{:ssh, "user@host"}` | SSH | Resolves to `%{host: "user@host"}`. |
| `{:ssh, "user@host", %{opts}}` | SSH | Options map merged over `%{host: ...}`. |
| `{:tmux, :codex}` | Tmux TUI | Persistent interactive Codex pane; `:claude` and `:opencode` are also supported. Runs on the host unless a runner is selected. |
| `{:tmux, client, %{opts}}` | Tmux TUI | Persistent pane with host, per-agent Docker, or per-agent bwrap execution. |
| `:bwrap` | Bwrap | Bubblewrap sandbox; default when `backend` is omitted. |
| `{:bwrap, %{opts}}` | Bwrap | Options map (see below). |
| `:mock` | Mock | No process; a stub for testing. |
| `{:mock, %{script: [...]}}` | Mock | `script` is stored for test introspection only — the backend does not generate responses (see [backends.md](backends.md)). |

Any other value fails validation with `{:invalid_backend, backend}`.

> JSON/YAML limitation: the loader normally converts only a **scalar string** backend to an atom (`"local"` → `:local`, `"bwrap"` → `:bwrap`, etc.). It does **not** turn an array like `["docker", "coder"]` into a tuple, so Docker/SSH/Apple tuple forms still require `.exs`. Tmux is the explicit exception and accepts a data-only object: `{"type":"tmux","client":"codex","opts":{...}}` (the same shape works in YAML).

## Backend config separation

For agents, the agent's `config` map is split at deploy time (in `Genswarms.Agents.AgentServer`) into backend keys and domain keys. Backend keys override the values returned by `SwarmConfig.backend_config/1` and control the execution environment; all remaining keys stay as domain config available to the agent's skills and logic.

The shared/backend-specific keys are:

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `workspace` | string | `/tmp/szc-workspace/<sandbox_id>` | Working directory mounted read-write into the sandbox. |
| `container_name` | string | `szc-<swarm>-<agent>` | Explicit Apple container name. |
| `env` | map | `%{}` | Extra environment variables for Docker/Apple container agents. |
| `volumes` | list of `{host, container}` | `[]` | Extra Docker/Apple container bind mounts. |
| `cmd` | string or argv list | backend default | Override the Docker/Apple in-container command. |
| `extra_path` | list of strings | `[]` | Additional directories prepended to `PATH`. |
| `extra_ro_binds` | list of `{host, container}` | `[]` | Extra read-only bind mounts. |
| `extra_rw_binds` | list of `{host, container}` | `[]` | Additional writable mounts for tmux Docker/bwrap runners. The subzeroclaw bwrap backend does not currently apply this key; use `workspace` there. |
| `extra_env` | map | `%{}` | Extra environment variables passed into the sandbox. |
| `memory_limit` | string | subzeroclaw bwrap: `"256M"`; tmux bwrap rootless: unset; tmux bwrap cgroup: `"2G"` | Memory ceiling (e.g. `"2G"`). Rootless uses `RLIMIT_AS`; JS runtimes can reserve large virtual heaps, so set it deliberately. |
| `memory_swap` | string | Docker default | Docker-only RAM+swap cap. |
| `cpu_limit` | number/string | backend default | Docker/Apple CPU cap. |
| `cpu_shares` | integer | `100` | Relative CPU weight. |
| `pids_limit` | integer | Docker default | Docker-only process-count cap. |
| `tasks_max` | integer | `50` | Max number of tasks/processes. |
| `subzeroclaw_path` | string | resolved | Explicit path to the `subzeroclaw` binary. |
| `subzeroclaw_src` | string | resolved | Source directory mounted into Docker/Apple containers for in-container build. |
| `api_key` / `model` / `endpoint` | string | env/provider defaults | LLM settings passed through to real backends. Prefer top-level `model` / `endpoint` for normal configs. |
| `request_extra` / `compact_extra` | JSON string or map | none | Advanced subzeroclaw request/compaction routing payloads. |
| `presets` | list of atoms | `[:base]` | The same agent-level `presets` key (above), forwarded to the sandbox. The bwrap backend falls back to `[:base]` when none are given. |
| `network` | `:open` \| `:none` \| `:isolated` \| string | `:open` | Subzeroclaw Docker/bwrap use `:isolated` for LLM-only egress. An isolated tmux TUI runner supports `:none` (no network); Docker also accepts a named network. Tmux TUI runners reject `:isolated` until their clients have a safe LLM-forwarding contract. Apple container also rejects `:isolated`. See [Security › network isolation](security.md#agent-network-isolation). |
| `client` / `resume` | tmux client / boolean or string | required / `false` | Select a coding TUI and optionally resume its latest/named conversation. |
| `runner` | `:host` \| `:docker` \| `:bwrap` | `:host` | Where a tmux pane executes its coding client. tmux itself remains host-side and attachable. |
| `client_source` | `:runtime` \| `:host_nix` | Docker: `:runtime`; bwrap: `:host_nix` | Use a client already in the isolated runtime, or mount the selected host Nix closure read-only. |
| `privilege_mode` | `:rootless` \| `:cgroup` | Tmux bwrap: `:rootless`; subzeroclaw bwrap: `:cgroup` | TUI bwrap defaults rootless so the client keeps a direct PTY; select cgroup explicitly for systemd-enforced limits on a compatible host. |
| `image` / `state_dir` | string / string | preset image / private per-agent path | Docker TUI image and persistent client home mounted as `/root`. |
| `pass_env` / `runner_env` | list of names / map | `[]` / `%{}` | Environment passed into an isolated TUI runner. Docker and bwrap use private mode-0600 env files so values are not placed in host-visible CLI argv. |
| `keepalive_command` | argv list | `["sleep", "infinity"]` | Command that keeps a per-agent Docker TUI container alive between client restarts. |
| `docker_executable` / `bwrap_executable` | string | resolved from `PATH` | Isolation CLI overrides for a tmux runner. |
| `tmux_socket` / `tmux_executable` | string | `genswarms` / `tmux` | Persistent tmux transport selection. |
| `executable` / `args` | string / list of strings | client default / `[]` | Tmux client executable and extra argv; values are never interpolated into a host shell. |
| `approval_policy` / `sandbox` | string/atom | client default | Codex permissions. |
| `permission_mode` / `effort` | string/atom | client default | Claude permissions and effort. |
| `dangerously_bypass` / `auto_approve` | boolean | `false` | Explicit unsafe client modes; never enabled by default. |
| `poll_interval_ms` / `ready_quiet_ms` | positive integer | `250` / `1000` | Tmux observation timing. Quiet readiness is ignored unless `quiet_ready_fallback: true`. |
| `submit_delay_ms` / `submit_retry_after_ms` | non-negative / positive integer | `100` / `1000` | Separates literal input from the first Enter, then bounds the active-cursor verification grace. |
| `submit_max_attempts` / `submit_check_max_errors` | positive integer | `2` / `3` | Bounded Enter-only recovery and cursor-query failures before `needs_attention`. |

For example, this JSON shape creates an attachable OpenCode pane backed by its
own Docker container; the image must contain `opencode`:

```json
{
  "type": "tmux",
  "client": "opencode",
  "opts": {
    "runner": "docker",
    "image": "coding-tuis:latest",
    "client_source": "runtime",
    "network": "open"
  }
}
```

On NixOS, use `"client_source": "host_nix"` to mount the installed client's
minimal Nix closure instead. `network: "none"` is a complete network cutoff and
therefore also blocks cloud model APIs; it is not equivalent to the
subzeroclaw-only `network: "isolated"` LLM forwarder.

Any key in `config` not listed above (for example `population_size` or `max_iterations`) is preserved as domain config and is not interpreted by the backend.

```elixir
%{
  name: :fixer,
  backend: :bwrap,
  config: %{
    # Backend keys
    workspace: "/tmp/workspace",
    extra_path: ["/opt/tools/bin"],
    extra_ro_binds: [{"/home/user/project", "/project"}],
    memory_limit: "512M",
    # Domain keys
    population_size: 10,
    max_iterations: 50
  }
}
```

## Objects

Objects are non-agentic components that participate in topology but run deterministic code instead of LLM calls. Each object must specify either a `handler` (native Elixir) or a `backend` (Docker/Apple container/SSH). See [objects.md](objects.md) for the handler behaviour.

| Key | Type | Required | Description |
|-----|------|----------|-------------|
| `name` | atom or string | Yes | Unique object identifier. Normalized to an atom. |
| `handler` | module | For native objects | Module implementing `init/1` and `handle_message/3` from `Genswarms.Objects.ObjectHandler`. |
| `backend` | backend spec | For Docker/Apple container/SSH objects | Same forms as agent backends. |
| `config` | map | No | Passed to the handler's `init/1` (or to the backend). |

If a `handler` module is already loaded, the validator checks it exports `init/1` and `handle_message/3` (raising `{:invalid_handler, handler, ...}` otherwise); if the module is not yet loaded (it may live in the host application), validation is deferred. An object map with neither `handler` nor `backend` fails with `:invalid_object_config`.

```elixir
objects: [
  %{
    name: :evaluator,
    handler: MyApp.Objects.Evaluator,
    config: %{parallel: true, timeout: 300_000}
  }
]
```

## Topology

`topology` is a list of directed edges, each a `{from, to}` tuple. Every endpoint must be the name of a defined agent or object. Both endpoints may be atoms or strings (strings are normalized to atoms).

```elixir
topology: [
  {:researcher, :coder},  # researcher can send to coder
  {:coder, :researcher}   # and back
]
```

An edge `{a, b}` permits messages from `a` to `b`. For two-way communication, declare both directions explicitly. The topology may be empty.

Validation collects all edge errors and returns them wrapped as `{:invalid_topology, errors}`. Each entry is either `{:unknown_agent, name}` (an endpoint that is not a defined agent or object) or `{:invalid_edge_format, idx, edge}` (an edge that is not a `{from, to}` tuple of atoms/strings).

### System objects

The router always permits messages to the system objects `:metrics`, `:tick`, and `:gateway`, even without an explicit topology edge (`@system_objects` in `lib/genswarms/routing/router.ex`). You do not need to declare edges to these targets. See [messaging.md](messaging.md) for routing details.

## Config formats

The file extension determines the parser: `.exs` (Elixir term), `.json`, or `.yaml`/`.yml`. All formats produce the same validated structure. String keys are atomized and scalar string backend values are converted to atoms during loading.

### Elixir (.exs)

The file must evaluate to a configuration map. This is the only format that supports Elixir-native values such as module atoms for object handlers, tuple-form backends, and dynamic expressions.

```elixir
%{
  name: "example-swarm",
  agents: [
    %{name: :researcher, backend: :local, skills: ["web.md"]},
    %{name: :coder, backend: {:docker, "coder"}, skills: ["code.md"]}
  ],
  topology: [
    {:researcher, :coder},
    {:coder, :researcher}
  ]
}
```

### JSON

Topology edges are two-element arrays. Backends are normally scalar strings;
see the JSON/YAML limitation above. Tmux additionally accepts an object such as
`{"type":"tmux","client":"codex","opts":{"workspace":"/work"}}`.

```json
{
  "name": "example-swarm",
  "agents": [
    { "name": "researcher", "backend": "local", "skills": ["web.md"] },
    { "name": "coder", "backend": "bwrap", "skills": ["code.md"] }
  ],
  "topology": [
    ["researcher", "coder"],
    ["coder", "researcher"]
  ]
}
```

### YAML

Same rule as JSON, including the data-only tmux backend object exception.

```yaml
name: example-swarm
agents:
  - name: researcher
    backend: local
    skills:
      - web.md
  - name: coder
    backend: bwrap
    skills:
      - code.md
topology:
  - [researcher, coder]
  - [coder, researcher]
```

## Per-agent models

Each agent can run on a different model and endpoint. The `model` is passed to `subzeroclaw` as `SUBZEROCLAW_REQUEST_EXTRA = {"model": …}` — there is no `SUBZEROCLAW_MODEL` env fallback (it is a dead variable); when unset, `subzeroclaw` uses its own default. The endpoint falls back to `SUBZEROCLAW_ENDPOINT` and is otherwise auto-detected from the API key. To route through an unhardcoded router, set `request_extra` with a `policy_ir` instead of a bare `model`.

```elixir
agents: [
  %{name: :researcher, backend: :local, model: "anthropic/claude-sonnet-4", skills: ["web.md"]},
  %{name: :coder, backend: :local, model: "deepseek/deepseek-chat", skills: ["code.md"]}
]
```

## Skill templating

Skill files listed under an agent's `skills:` are copied into the agent at deploy
time, and three template variables are substituted per agent:

| Variable | Resolves to |
|----------|-------------|
| `{{agent_name}}` | the agent's name |
| `{{swarm_name}}` | the swarm name |
| `{{workspace}}` | the agent's `workspace` path (empty if unset) |

This lets one skill file serve many agents. See [skills.md](skills.md) for
authoring details and built-in skills.

## Full annotated example

```elixir
%{
  # Required: unique identifier (letter-led, alphanumeric/_/-)
  name: "example-swarm",

  agents: [
    # Local agent with a per-agent model and tool presets
    %{
      name: :researcher,
      backend: :local,
      model: "anthropic/claude-sonnet-4",
      skills: ["web.md", "summarize.md"],
      presets: [:base, :web]
    },

    # Sandboxed bwrap agent: backend keys + domain keys in `config`
    %{
      name: :coder,
      backend: :bwrap,
      skills: ["code.md"],
      config: %{
        # Backend keys (consumed by BwrapBackend)
        workspace: "/tmp/example-swarm/coder",
        memory_limit: "512M",
        presets: [:base, :code],
        # Domain key (available to the agent's logic)
        max_iterations: 25
      }
    }
  ],

  # Optional: deterministic, non-agentic component
  objects: [
    %{
      name: :evaluator,
      handler: MyApp.Objects.Evaluator,
      config: %{parallel: true}
    }
  ],

  # Directed edges; system objects (:metrics, :tick, :gateway)
  # are routable without explicit edges
  topology: [
    {:researcher, :coder},
    {:coder, :evaluator},
    {:evaluator, :researcher}
  ]
}
```

## See also

- [backends.md](backends.md) — backend types and their options
- [containers.md](containers.md) — building NixOS container images for Docker and Apple container agents
- [objects.md](objects.md) — the `ObjectHandler` behaviour and object patterns
- [skills.md](skills.md) — authoring and deploying agent skill files
- [cli.md](cli.md) — validating and running configs from the command line

---

---
description: Complete GenSwarms CLI reference — every genswarms and mix genswarms.* command for starting, tasking, scaling, and observing swarms.
---

# CLI reference

The `genswarms` command-line interface manages the full lifecycle of a swarm: creating projects, starting and stopping swarms, sending tasks and messages, querying events, and performing advanced runtime operations like scaling and snapshotting. This page documents every subcommand, grounded in the actual task modules under `lib/mix/tasks/genswarms/`.

## Building and invoking the CLI

Build the standalone escript binary with Mix:

```bash
mix escript.build   # produces ./genswarms in the project root
```

Every subcommand can be invoked two ways, with the same arguments and flags:

```bash
# As the escript binary
genswarms status
genswarms start swarms/example_swarm.exs

# As a Mix task (dot-separated subcommand)
mix genswarms.status
mix genswarms.start swarms/example_swarm.exs
```

A `.env` file in the working directory is auto-loaded before most commands run.

### Global help and version

```bash
genswarms                 # print help (command list + examples)
genswarms help            # same as above
genswarms version         # print version
genswarms --version       # print version
genswarms -v              # print version
```

Most commands also accept `--help` / `-h` for command-specific usage:

```bash
genswarms start --help
genswarms events -h
```

## Command table

These commands are dispatched by the `genswarms` escript binary (see
`dispatch/2` in `lib/genswarms/cli.ex`):

| Command | Description |
|---------|-------------|
| `init` | Create a new swarm project with standard directory structure |
| `up` | Start the Phoenix server (REST API + WebSocket) in the background (legacy alias for `dashboard start`) |
| `down` | Stop the dashboard and/or running swarms |
| `dashboard` | Start, stop, or check the web dashboard |
| `start` | Start a swarm from a config file (daemon by default) |
| `stop` | Stop a running swarm daemon |
| `restart` | Restart a swarm, reloading its config |
| `status` | Show status of all swarms or one swarm in detail |
| `logs` | View or stream agent logs and conversation history |
| `events` | Query events from the centralized event store |
| `task` | Send a task to an agent |
| `msg` | Route a message between two agents |
| `env` | Manage environment variables in `.env` files |
| `build` | Build agent Docker images via Nix |
| `config validate` | Validate one or more config files (alias: `check`) |
| `list-skills` | List available skills |
| `scale` | Scale an agent group in a running swarm to a target count |
| `overlay` | Inspect or clear a swarm's dynamic-mutation overlay |
| `snapshot` | Emit a swarm's effective config (seed + overlay) as `.exs` |

### Mix-task only commands

The following operations exist as Mix tasks but are **not** wired into the
escript dispatch — running `genswarms pause …` falls through to
`Unknown command`. Invoke them through Mix instead:

| Command | Description |
|---------|-------------|
| `mix genswarms.pause <name>` | Pause a swarm by freezing its Docker containers |
| `mix genswarms.resume <name>` | Resume a paused swarm |
| `mix genswarms.delete <name>` | Delete a swarm and all of its data |
| `mix genswarms.clean` | Remove stopped/crashed swarms (optionally clear all events) |
| `mix genswarms.restart_agent <swarm> <agent>` | Restart a single agent in a running swarm (requires the API server) |

## Server

### `up`

Start the Phoenix server (REST API + WebSocket, plus the dev dashboard) in the background. Pass `--foreground` to run it inline instead.

```bash
genswarms up                 # start on default port (4000 or $PORT)
genswarms up --port 3000     # custom port
genswarms up --foreground    # run inline instead of backgrounding
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--port PORT` | `-p` | Port to run on (default: 4000 or `$PORT`) |
| `--foreground` | `-f` | Run in foreground instead of background |

### `down`

Stop running services. With no flags it stops both swarms and the dashboard.

```bash
genswarms down                   # stop everything
genswarms down --dashboard-only  # only the dashboard
genswarms down --swarms-only     # only swarms
```

| Flag | Description |
|------|-------------|
| `--dashboard-only` | Only stop the dashboard |
| `--swarms-only` | Only stop swarms |

### `dashboard`

Start, stop, or check the web dashboard. The dashboard runs independently of swarms.

```bash
genswarms dashboard               # start (default subcommand)
genswarms dashboard start -p 3000 # start on port 3000
genswarms dashboard stop          # stop
genswarms dashboard status        # check whether it is running
```

Subcommands: `start` (default), `stop`, `status`.

| Flag | Alias | Description |
|------|-------|-------------|
| `--port PORT` | `-p` | Port to run on (default: 4000 or `$PORT`) |
| `--foreground` | `-f` | Run in foreground instead of background |

## Swarm lifecycle

### `init`

Scaffold a new project (`.env`, `swarms/`, `skills/`, `docker/`, etc.). The generated config is written to `swarms/example_swarm.exs` (note the underscore in the file name) and declares a swarm whose `name:` is `example-swarm` (hyphenated). Use the file path when starting/validating it, and the hyphenated name when referring to the running swarm.

```bash
genswarms init                  # in the current directory
genswarms init my-project       # into a new directory
genswarms init ~/projects/swarm # absolute path
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--force` | `-f` | Overwrite existing files in a non-empty directory |

### `start`

Start a swarm from a config file (`.exs` / `.json` / `.yaml`). Runs as a background daemon by default; state is tracked in `.genswarms/swarms.db`.

```bash
genswarms start swarms/example_swarm.exs
genswarms start swarms/example_swarm.exs --foreground
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--foreground` | `-f` | Run in foreground instead of daemon mode |

### `stop`

Stop a running swarm daemon (sends SIGTERM and updates the registry).

```bash
genswarms stop example-swarm
```

### `restart`

Stop then start a swarm, reloading its config file so config changes take effect.

```bash
genswarms restart example-swarm           # normal restart
genswarms restart example-swarm --delete  # clean restart (wipe old logs/events/data)
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--delete` | `-d` | Delete all logs, events, and data before restarting |

### `mix genswarms.pause` (Mix-only)

Freeze every Docker container belonging to the swarm (`docker pause szc-<swarm>-<agent>`). Processes are suspended but containers stay alive. If no matching running containers are found, the command reports `No running containers found for swarm` and exits non-zero.

```bash
mix genswarms.pause example-swarm
```

### `mix genswarms.resume` (Mix-only)

Unfreeze all paused Docker containers in the swarm.

```bash
mix genswarms.resume example-swarm
```

### `mix genswarms.delete` (Mix-only)

Stop the swarm if running, remove it from the registry, and delete all of its events, logs, and data files.

```bash
mix genswarms.delete example-swarm
mix genswarms.delete example-swarm --force
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--force` | `-f` | Skip the confirmation prompt |

### `mix genswarms.clean` (Mix-only)

Remove all stopped and crashed swarms (and their files) from the registry.

```bash
mix genswarms.clean           # clean stopped/crashed swarms
mix genswarms.clean --all     # also clear all events from the database
mix genswarms.clean --force   # skip confirmation
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--all` | | Also clear all events from the database |
| `--force` | `-f` | Skip the confirmation prompt |

### `status`

Show all registered swarms, or detailed status (agents, objects, topology, backends, skills) for one.

```bash
genswarms status                # all swarms
genswarms status example-swarm  # detailed view of one swarm
```

## Agent operations

### `task`

Send a task to a specific agent. If the API server is running the task is delivered over HTTP; otherwise it is queued in SQLite for the daemon to pick up.

```bash
genswarms task example-swarm researcher "Summarize the latest findings"
```

Usage: `genswarms task <swarm> <agent> <task>`

### `msg`

Route a message from one agent to another. The route is validated against the swarm topology before sending; an invalid route lists the valid targets and exits non-zero.

```bash
genswarms msg example-swarm researcher coder "Can you review this code?"
```

Usage: `genswarms msg <swarm> <from> <to> <message>`

### `mix genswarms.restart_agent` (Mix-only)

Restart a single agent in a running swarm. This requires the API server to be running. It is **not** an escript subcommand — invoke it through Mix.

```bash
mix genswarms.restart_agent example-swarm researcher
```

Usage: `mix genswarms.restart_agent <swarm> <agent>`

### `logs`

View or stream agent logs and conversation history.

```bash
genswarms logs example-swarm                # all agents, conversation only
genswarms logs example-swarm researcher     # one agent
genswarms logs example-swarm researcher -f  # stream in real time
genswarms logs example-swarm --stdout       # show stdout output
genswarms logs example-swarm --events       # all agent events
genswarms logs example-swarm --all          # everything
genswarms logs example-swarm --tail 100     # last 100 entries
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--follow` | `-f` | Stream logs in real time |
| `--tail N` | `-n` | Show the last N entries (default: 50) |
| `--stdout` | | Show agent stdout output |
| `--events` | | Show all agent events (tasks, messages, lifecycle) |
| `--conversation` | | Show conversation only (default) |
| `--all` | | Show all log types |

## Observability and events

### `events`

Query events from the centralized event store. With no flags it prints the last 50 events. (This is a one-shot query — see the note below about `--follow`/`--stats`.)

```bash
genswarms events                          # last 50 events
genswarms events --errors                 # errors only
genswarms events --errors -n 5            # errors from the last 5 minutes
genswarms events -s example-swarm         # filter by swarm
genswarms events -s example-swarm -a coder# filter by swarm + agent
genswarms events --category backend       # backend events only
genswarms events --type message_routed    # filter by event type
genswarms events --limit 200              # raise the result cap
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--errors` | `-e` | Show only error-level events |
| `--warnings` | `-w` | Show warnings and errors |
| `--minutes N` | `-n` | Only events from the last N minutes |
| `--swarm NAME` | `-s` | Filter by swarm name |
| `--agent NAME` | `-a` | Filter by agent name |
| `--category CAT` | | Filter by category: `backend`, `routing`, `agent`, `object`, `swarm`, `system` |
| `--type TYPE` | | Filter by event type (e.g. `stdout`, `message_routed`, `task_sent`) |
| `--limit N` | | Maximum events to return (default: 50) |

> **Not yet implemented:** the parser also accepts `--follow` / `-f` and `--stats`, but they are currently silent no-ops. `run/1` always performs a one-shot query and `build_query_opts/1` never consults either flag — there is no live-stream loop or statistics computation in `lib/mix/tasks/genswarms/events.ex`. For live event streaming, use `genswarms logs <swarm> --follow` instead.

See [observability.md](observability.md) for the full category/event-type catalog.

## Config and skills

### `config validate`

Validate one or more config files using the real loader: file format, required fields, agent/object config, topology validity, skill-file existence, and handler-module existence. Globs are expanded. `check` is a shorthand alias.

```bash
genswarms config validate swarms/example_swarm.exs
genswarms config validate "swarms/*.exs"
genswarms config validate config.json --quiet
genswarms check swarms/example_swarm.exs   # alias
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--quiet` | `-q` | Only output errors |

### `list-skills`

List all skills available in the skills repository.

```bash
genswarms list-skills
```

### `env`

Manage variables in a `.env` file. Sensitive values (matching key, secret, token, etc.) are masked in `list` output.

```bash
genswarms env list                          # list all variables
genswarms env get SUBZEROCLAW_API_KEY       # read one variable
genswarms env set PORT 3000                 # set a variable
genswarms env unset DEBUG                    # remove a variable
genswarms env list --file .env.production    # use a different file
```

Subcommands: `list` (default), `get <key>`, `set <key> <value>`, `unset <key>`.

| Flag | Alias | Description |
|------|-------|-------------|
| `--file FILE` | `-f` | Use a specific `.env` file (default: `.env`) |

### `build`

Build agent Docker images via Nix (falling back to a `docker build` if no flake is found). The available images are the eight defined in `nix/container.nix`: `base`, `web`, `code`, `data`, `full`, `python`, `node`, `devops` — see [Containers → prebuilt images](containers.md) for what each bundles. (Equivalently, build one directly with `nix build .#agentContainer-<name>`.)

```bash
genswarms build base              # build one image
genswarms build --all             # build all images
genswarms build base --push       # build and push (requires DOCKER_REGISTRY)
genswarms build base --tag v1.0   # custom tag
genswarms build base --no-cache   # rebuild without cache
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--all` | `-a` | Build all images |
| `--push` | `-p` | Push to registry after building |
| `--tag TAG` | `-t` | Custom tag (default: `latest`) |
| `--no-cache` | | Build without cache |

## Advanced: dynamic swarm operations

These commands operate on the runtime state of a swarm. Additions and removals are recorded in an *overlay* that is replayed at start so dynamic state survives a restart.

### `scale`

Scale an agent group to a target count. The group is identified by `base-name`; members are named `<base-name>_1`, `<base-name>_2`, .... Extra members are stopped, missing ones are created from an existing member's spec.

```bash
genswarms scale example-swarm researcher 20
```

Usage: `genswarms scale <swarm> <base-name> <count>` — `count` must be a non-negative integer. This command takes no flags.

### `overlay`

Inspect or clear the dynamic-mutation overlay (the event log of runtime additions/removals).

```bash
genswarms overlay example-swarm           # list overlay events
genswarms overlay example-swarm --clear   # wipe the overlay
```

| Flag | Description |
|------|-------------|
| `--clear` | Wipe the overlay, returning the swarm to its pure seed state |

### `snapshot`

Emit a swarm's effective configuration (seed combined with overlay) as an `.exs` source. This does not modify the original config file; the output is a declarative seed you can load with `start`.

```bash
genswarms snapshot example-swarm                    # write to stdout
genswarms snapshot example-swarm --output seed.exs  # write to a file
```

| Flag | Alias | Description |
|------|-------|-------------|
| `--output FILE` | `-o` | Write the snapshot to a file instead of stdout |

## See also

- [getting-started.md](getting-started.md) — first swarm, end to end
- [rest-api.md](rest-api.md) — HTTP API behind several of these commands
- [configuration.md](configuration.md) — the config DSL validated by `config validate`
- [observability.md](observability.md) — event categories and types for `events`

---

---
description: How GenSwarms works: the OTP supervision tree, the daemon model, SQLite coordination, and deployment topologies.
---

# Architecture

GenSwarms is an Elixir/OTP application that orchestrates swarms of subzeroclaw agents. This document describes the supervision tree, the per-swarm processes, the API-first design, the daemon model, and the supported deployment topologies.

## Overview

The OTP application (`:genswarms`, root module `Genswarms`) starts a single top-level supervisor (`Genswarms.Supervisor`, strategy `:one_for_one`) with a flat set of long-lived children. Swarms, agents, and objects are not separate static branches of the tree — they are started dynamically at runtime under shared application-level singletons, all keyed by swarm name.

A key consequence: there is **one** registry and **one** dynamic supervisor for the whole node. Agents and objects from every swarm coexist under them, distinguished by a `{swarm_name, name}` key.

## Supervision tree

The children below are started by `lib/genswarms/application.ex` in this order. The Phoenix endpoint is **not** part of the static tree — it is added dynamically (see [API-first design](#api-first-design)).

```text
Genswarms.Supervisor (one_for_one)
├── Genswarms.Telemetry                         (telemetry supervisor)
├── Phoenix.PubSub (name: Genswarms.PubSub)     (message broadcasting)
├── Genswarms.Observability.LogStore            (centralized event logging)
├── Genswarms.Backends.Bwrap.AgentTelemetry     (ETS ring buffer, 10k+ scale)
├── Registry (keys: :unique, name: Genswarms.AgentRegistry)   (process lookup)
├── Genswarms.Skills.SkillsManager              (ETS-backed skill files)
├── Genswarms.Routing.Router                    (inter-agent message routing)
├── DynamicSupervisor (name: Genswarms.AgentSupervisor, one_for_one)
│       │   (shared by ALL agents AND objects, across all swarms)
│       ├── AgentServer {swarm, agent}  ── Backend + LogWatcher
│       ├── AgentServer {swarm, agent}  ── Backend + LogWatcher
│       └── ObjectServer {swarm, object}
├── Genswarms.SwarmManager                      (swarm lifecycle GenServer)
└── (EventStore.child_specs/0)                  (backend-dependent; none for
                                                 the default stateless SQLite)
```

Before the children start, `Genswarms.Application.start/2` also calls `Genswarms.CLI.EnvManager.auto_load/0` to load a `.env` file if one is present.

After the tree is up, `Genswarms.Observability.TelemetryBridge.attach/0` wires the telemetry event stream into `LogStore` so events are durable, queryable, and streamable over WebSocket.

### How the real tree differs from the README diagram

The diagrams in `README.md` and `CLAUDE.md` are conceptual and do not match the actual process layout. Notable differences, verified against `application.ex`:

| README/CLAUDE diagram says | Actual tree |
|----------------------------|-------------|
| `Registry`, `Router`, `SkillsManager`, `AgentDynSup` are children of `SwarmManager` | They are direct children of the top-level `Genswarms.Supervisor`, siblings of `SwarmManager` |
| Each swarm has its own supervisor subtree | One shared `Genswarms.AgentSupervisor` and one shared `Genswarms.AgentRegistry` serve all swarms |
| Agents and objects have separate supervisors | Objects run under the **same** `Genswarms.AgentSupervisor` as agents (see `objects/object_supervisor.ex`) |
| `SwarmRegistry (SQLite)` is a child of the tree | `SwarmRegistry` is a stateless SQLite helper module, not a supervised process |
| Phoenix is a static child | Phoenix endpoint is started dynamically, not part of the static tree |

## Per-swarm processes

`Genswarms.SwarmManager` is the lifecycle GenServer. It loads configs, tracks per-swarm status (`:starting | :running | :stopping | :stopped | :error`), and starts agents and objects via thin helper modules that delegate to the shared dynamic supervisor.

Every swarm definition and dynamic mutation passes through the [IR](intermediate-representation.md) gate (`Genswarms.IR.Gate`): a config must translate to a valid `swarm.state` before any agent is spawned, and `add_agent`/`scale_agent_group` are bounded by the per-swarm policy. The IR is the pure-data model that validates, mutates, and can drive a swarm.

```text
SwarmManager (single GenServer, tracks swarms: %{name => info})
   │  starts/stops children on the shared supervisor
   ▼
Genswarms.AgentSupervisor (DynamicSupervisor)
   ├── AgentServer  (per agent, registered as {swarm_name, agent_name})
   │      ├── Backend     (Local Port | tmux TUI | Docker | SSH | Bwrap | Mock)
   │      └── LogWatcher  (polls logs + .outbox/ for message routing)
   └── ObjectServer (per object, registered as {swarm_name, object_name})
```

- **`Genswarms.Agents.AgentSupervisor`** and **`Genswarms.Objects.ObjectSupervisor`** are not GenServers; they are helper modules whose `start_*`/`stop_*`/`list_*` functions call `DynamicSupervisor.start_child/2` against the shared `Genswarms.AgentSupervisor` and look up processes in `Genswarms.AgentRegistry`. Both modules hardcode `@supervisor Genswarms.AgentSupervisor`, which is why agents and objects share one supervisor.
- **`AgentServer`** (`lib/genswarms/agents/agent_server.ex`) is a GenServer per agent. On init it starts the configured backend and links a `Genswarms.Agents.LogWatcher`, which polls the agent's logs and `.outbox/` directory and routes outgoing messages through the `Router`.
- **`ObjectServer`** wraps a module implementing the `ObjectHandler` behaviour; objects participate in the same topology as agents but execute deterministic Elixir instead of LLM calls.
- **`Router`** (`lib/genswarms/routing/router.ex`) is a GenServer that holds each swarm's topology as an adjacency map and validates inter-agent messages against the allowed edges before delivering. The system objects `:metrics`, `:tick`, and `:gateway` (the `@system_objects` list in `router.ex`) are always routable without explicit topology edges.

## API-first design

The Phoenix layer exposes a pure JSON REST API plus a WebSocket channel for real-time events. No HTML or bundled frontend is shipped — bring your own client (React, Vue, etc.) or use the CLI. CORS is enabled.

The endpoint is optional and lifecycle-managed at runtime rather than supervised statically:

- `Genswarms.Application.start_web_server/1` adds `GenswarmsWeb.Endpoint` as a dynamic child of `Genswarms.Supervisor` (default port `4000`, overridable via the `PORT` env var or the `:port` option). Calling it twice returns `{:error, :already_running}`.
- When the web server starts on the monitor/API node, it also starts `Genswarms.Observability.EventRelay` (unless `config :genswarms, :event_relay` is set to `false`), which tails the shared SQLite event log and re-broadcasts new events to WebSocket clients — so the API node can stream events produced by daemon swarms running in other BEAM instances.
- `stop_web_server/0` terminates the relay and the endpoint, returning `{:error, :not_running}` if the server is not up.

```text
External client (React, Vue, CLI)        GenSwarms API node (Phoenix)
        │                                        │
        ├── HTTP  ──────────────────────────────►├── REST API  (/api/*)
        ├── WS    ──────────────────────────────►├── WebSocket (swarm:* channel)
        │                                        ├── EventRelay (tails SQLite log)
        │                                        └── SwarmRegistry (SQLite reads/writes)
```

`EventRelay` re-broadcasts each newly-persisted event onto the same PubSub topics (`log_store:events` and `log_store:events:<swarm>`) that `LogStore` uses in-node, so the existing `SwarmChannel` delivers them to WebSocket clients unchanged. It polls on a configurable interval (default 500 ms) and is intended to run **only** on a monitor/API node that does not host swarms in-process, avoiding double-delivery.

See `docs/rest-api.md` and `docs/observability.md` for the API surface and event model.

## Daemon model

Swarms run as **independent OS processes** (daemons), separate from the API node. This isolates a swarm's BEAM from the API server and from other swarms, and lets the CLI manage swarms without a running dashboard.

The CLI is available two ways, and both reach the same task implementations:

- The built escript binary: `genswarms <command> ...` (built with `mix escript.build`; `main_module: Genswarms.CLI`, output name `genswarms`).
- The Mix wrapper task: `mix genswarms <command> ...`, which dispatches each subcommand from `Mix.Tasks.Genswarms.run/1`.

The examples below use the escript form (`genswarms ...`); the `mix genswarms ...` form is equivalent.

### Starting a daemon

`genswarms start <config>` (escript) — equivalently `mix genswarms start <config>` — is implemented by `Mix.Tasks.Genswarms.Start`. It does not run the swarm in its own process. It:

1. Verifies the config file exists and initializes the SQLite registry (`SwarmRegistry.init/0`).
2. Spawns a detached background process via `Port.open/2` running `sh -c 'nohup mix genswarms.start.daemon "<config>" > .genswarms/logs/<swarm>.log 2>&1 & echo $!'`, capturing the child PID from stdout.
3. Waits ~2 s and confirms the daemon is still alive (`SwarmRegistry.process_alive?/1`) before reporting success.

The inner `mix genswarms.start.daemon` task (`Mix.Tasks.Genswarms.Start.Daemon`) is the actual long-running process: it loads `.env`, initializes SQLite, starts the `:genswarms` application, starts the swarm, registers itself in SQLite, then enters a poll loop. (Use `genswarms start <config> --foreground` — alias `-f` — to run in the current process instead of daemonizing.)

### Coordination via SQLite

The API node and CLI never talk to a daemon's BEAM directly. They coordinate through a shared SQLite database at `.genswarms/swarms.db` (managed by `Genswarms.CLI.SwarmRegistry`).

```text
API node / CLI                         Daemon process (genswarms start)
   │                                          │
   ├── query swarm state ──┐        ┌── write swarm state (running/…)
   ├── queue tasks ────────┤ SQLite │── poll tasks every 500ms
   └── queue commands ─────┘ swarms │── poll commands every 500ms
                            .db ─────┘── log events
```

Tables in `.genswarms/swarms.db` (created by `SwarmRegistry.init/0`):

| Table | Purpose |
|-------|---------|
| `swarms` | Daemon swarm state: `name` (primary key), `status` (`running`/`stopped`/`crashed`), `pid`, `config_path`, `log_path`, `started_at`, `stopped_at` |
| `events` | Centralized event log (`id`, `timestamp`, `level`, `category`, `swarm`, `agent`, `event_type`, `message`, `metadata`), indexed by `swarm` and `timestamp` |
| `tasks` | Cross-process task queue (`swarm`, `agent`, `task`, `status`, `created_at`, `processed_at`), with a partial index on pending rows |
| `swarm_overlays` | Dynamic-mutation event log for runtime swarm changes, keyed by `(swarm, seq)` |
| `swarm_commands` | CLI → daemon command bridge (add/remove agents/objects, topology edges, scaling, fetch config, etc.) |

The database runs in WAL mode (`PRAGMA journal_mode=WAL`) with a 5 s busy timeout (`PRAGMA busy_timeout=5000`) for concurrent readers/writers. Each operation opens a fresh connection and closes it when done; `log_events_bulk/1` wraps a batch in a single `BEGIN`/`COMMIT` transaction.

### Poll loop

The daemon's `daemon_loop/2` monitors `Genswarms.Supervisor`; if it goes `:DOWN`, the swarm is marked `crashed`. Otherwise, every 500 ms (the `@task_poll_interval`) it:

1. `process_pending_tasks/1` — drains `SwarmRegistry.get_pending_tasks/1` and delivers each via `SwarmManager.send_task/3`, marking processed on success or leaving the task pending for retry (and logging) on failure.
2. `process_pending_commands/1` — applies queued mutation/control commands
   (add/remove/restart agents, interrupt or inspect persistent sessions,
   add/remove objects or topology edges, scale an agent group, fetch full
   config) and writes results back via `SwarmRegistry.mark_command_done/2`.

### Task delivery paths

`genswarms task <swarm> <agent> <msg>` chooses a path based on whether the API server is up:

- API server running → send over HTTP REST (`APIClient.send_task/3`).
- No API server → enqueue in the `tasks` table (`SwarmRegistry.queue_task/3`); the daemon's poll loop picks it up within ~500 ms.

### Stop, pause, resume

- `genswarms stop <swarm>` sends `SIGTERM` (`kill -TERM <pid>`) to the recorded daemon PID, waits for exit, and marks the swarm stopped.
- Pause/resume for daemon swarms cannot use in-BEAM GenServer calls (the daemon is a separate process), so they act on Docker containers directly, e.g. `docker pause szc-<swarm>-<agent>` / `docker unpause …`. Apple `container` has no equivalent pause/unpause support in the current backend.

## Deployment models

| Model | How agents run | Configuration |
|-------|----------------|---------------|
| Docker (NixOS) | Minimal NixOS containers, one per agent, namespaced `szc-<swarm>-<agent>` | `backend: {:docker, "<image>"}`; build with `nix build .#agentContainer-<preset>` |
| Apple container | OCI containers through Apple's `container` CLI on macOS / Apple silicon | `backend: {:apple_container, "<image>"}`; `container system start` first |
| Bare metal (Colmena + NixOS) | Dedicated NixOS machines provisioned ahead of time, reached over SSH | `colmena apply` to provision, then `backend: {:ssh, "user@host"}` |
| Bwrap | Bubblewrap sandboxes on a single NixOS host (10k+ scale) | `backend: :bwrap` |
| Tmux TUI | Persistent, human-attachable Codex/Claude/OpenCode panes; client may run on host or in a per-agent Docker/bwrap boundary | `backend: {:tmux, :codex, %{runner: :bwrap}}` |
| Hybrid | Any mix of `:local`, `{:tmux, …}`, `{:docker, …}`, `{:apple_container, …}`, `{:ssh, …}`, `:bwrap`, `:mock` in one swarm | per-agent `backend:` |

### Docker (NixOS containers)

Run many isolated agents on one machine using minimal NixOS containers that include only the tools declared via presets/tools. Containers are namespaced by swarm name (`szc-<swarm>-<agent>`), so multiple swarms run simultaneously without interference and pause/resume affects only the targeted swarm's containers.

### Persistent TUI workers

The tmux backend splits the terminal transport from execution. A trusted
host-side tmux server owns the PTY, pane ID, scrollback, attach surface, and
keystroke transport. The pane command is supplied by a runner:

```text
AgentServer / TmuxBackend
          │ durable turn files + lifecycle events
          ▼
 host tmux pane (attachable)
          │
          ├── client process on host
          ├── docker exec -it ──► per-agent persistent container
          └── bwrap ────────────► per-agent copy-on-write sandbox
                                  ├── CodexAdapter
                                  ├── ClaudeAdapter
                                  └── OpenCodeAdapter
```

The runner owns process/filesystem lifecycle and reports normalized metadata;
the adapter owns client argv, resume semantics, readiness/blocked recognition,
and the short path-based task nudge. Raw terminal capture remains diagnostic.
Turn completion is durable filesystem state, not an inference from screen text.
An orchestrator disconnect keeps the pane and runner alive, while explicit
destroy removes both.

### Apple container

Run OCI-style agent containers on macOS / Apple silicon with Apple's `container`
CLI. The backend preserves the same GenSwarms runtime mounts and environment as
Docker where supported, but rejects `network: :isolated` because the current CLI
does not provide equivalent egress isolation semantics.

### Bare metal (Colmena + NixOS)

Deploy fully configured NixOS machines with Colmena, then start the orchestrator, which connects to them over SSH. Point `start` at your own SSH-backed swarm config (one whose agents use `backend: {:ssh, "user@host"}`):

```bash
colmena apply
genswarms start path/to/bare_metal_swarm.exs
```

### Hybrid

Mix backends within a single swarm config:

```elixir
%{
  name: "example-swarm",
  agents: [
    %{name: :researcher, backend: :local},
    %{name: :coder, backend: {:docker, "coder"}},
    %{name: :mac_coder, backend: {:apple_container, "szc-agent-code:latest"}},
    %{name: :remote_1, backend: {:ssh, "root@192.168.1.51"}}
  ]
}
```

See `docs/backends.md` and `docs/containers.md` for backend specifics and container builds.

## See also

- [backends.md](backends.md) — backend types and configuration
- [containers.md](containers.md) — NixOS container builds and presets
- [configuration.md](configuration.md) — swarm config DSL
- [observability.md](observability.md) — events, logging, and streaming
- [rest-api.md](rest-api.md) — REST API reference

---

---
description: Inter-agent messaging in GenSwarms — @agent: syntax, topology-gated routing, broadcast, and the file-based inbox/outbox.
---

# Messaging

GenSwarms agents coordinate by sending messages to one another. Messages flow through a central `Genswarms.Routing.Router`, which validates every hop against the swarm topology before delivering it. This page covers the message syntax agents emit, how topology gates routing, the always-routable system objects, the file-based inbox and outbox channels, and the `swarm-msg` helper available inside agent sandboxes.

## The `@agent_name:` syntax

In their natural-language output, agents address another agent by prefixing a line with `@target:`. The orchestrator parses these prefixes and routes the rest of the message to the named agent.

```
ASST: I've analyzed the paper. @coder: Please implement the algorithm
described in section 3. Here's the pseudocode: ...
```

To reach every agent the sender is connected to, use `@all:`:

```
ASST: @all: Task completed successfully.
```

Under the hood, agent output is translated into structured `SWARM_MSG` markers that `Genswarms.Agents.AgentProtocol` and `Genswarms.Agents.LogWatcher` recognize:

```
<<SWARM_MSG:TO=coder:START>>
Please implement the algorithm.
<<SWARM_MSG:END>>
```

```
<<SWARM_MSG:BROADCAST:START>>
Task completed successfully.
<<SWARM_MSG:END>>
```

A target name must match `[a-zA-Z_][a-zA-Z0-9_]*`. Content is trimmed of surrounding whitespace before routing.

Two code paths recognize these markers:

- **`AgentProtocol.parse_output/1`** scans an agent's stdout when a turn completes. Its send pattern requires a newline immediately after `:START>>`.
- **`LogWatcher`** polls each agent's `*.txt` log files (the `RES:` entries written by the backend) every **500 ms**, extracts the same blocks, and forwards them to the Router. Here the newline after `:START>>` is optional.

Both paths emit `:send` messages (for `TO=<name>`) or `:broadcast` messages (for `BROADCAST`) to the Router.

## How topology gates routing

The Router keeps each swarm's topology as an adjacency map: for every source agent it stores the list of targets that source is allowed to reach. The topology is built from the `topology:` edges in your swarm config.

```elixir
topology: [
  {:researcher, :coder},
  {:coder, :reviewer}
]
```

When a message is routed, the Router checks whether the target is in the source's adjacency list:

- If allowed, the message is delivered to the target (agent or object), logged, and emitted as a `:message_routed` telemetry event and PubSub broadcast (on `swarm:<name>:routing`).
- If not allowed, the message is dropped, a warning is logged, and an `:invalid_route` telemetry event is emitted listing the allowed targets.

A broadcast (`@all:`) is delivered to every target in the source's adjacency list. An agent with no outgoing edges can broadcast, but the message reaches no one.

Edges are directed. `{:researcher, :coder}` lets `researcher` message `coder`, but not the reverse — add `{:coder, :researcher}` for a reply path.

> If the target resolves to neither a registered agent nor object in the swarm (for example, a typo'd name that *is* topology-allowed), the route passes the topology check but delivery fails: the Router logs a `:target_not_found` warning instead.

## System object routing

Three targets are always routable regardless of topology edges, defined as `@system_objects` in the Router (`[:metrics, :tick, :gateway]`):

| Target | Purpose |
|--------|---------|
| `:metrics` | Collect state reports and counters |
| `:tick` | Clock / heartbeat coordination |
| `:gateway` | External ingress/egress |

Any agent or object may send to `:metrics`, `:tick`, or `:gateway` without an explicit topology edge. This lets objects emit state reports, heartbeats, and similar signals without wiring them into every node of the graph. (A handler still has to be registered for the target to actually receive the message — see the `:target_not_found` note above.) See [objects.md](objects.md) for handlers that typically consume these.

## File-based messaging

Sandboxed (bwrap) agents cannot always rely on stdin/stdout. For them, two file-based channels mirror the in-band protocol. Both live under the agent's `workspace`.

### File-inbox (inbound)

Every message delivered to an agent is also written to `{workspace}/.inbox/{seq}_{from}.json`, giving sandboxed agents a reliable place to read incoming messages. The `seq` is a per-agent counter incremented on each delivery and zero-padded to four digits (for example `0001_researcher.json`).

```json
{"from": "researcher", "content": "Please implement the algorithm.", "seq": 1, "timestamp": "2024-01-01T00:00:00Z"}
```

`timestamp` is an ISO-8601 UTC string. The file-inbox is a delivery convenience: it is written in addition to the agent's normal in-process inbox queue, not instead of it. (Writing only happens when the agent has a non-empty `workspace` configured; local/port agents without a workspace skip it.)

### File-outbox (outbound)

Instead of emitting `@agent:` syntax, an agent can drop a JSON file into `{workspace}/.outbox/`. `LogWatcher` polls the outbox every 500 ms, processes `*.json` files in sorted (lexical) filename order, routes each one through the Router, and deletes it afterward.

A directed send:

```json
{"to": "coder", "content": "here is the fixed simulation"}
```

A broadcast:

```json
{"broadcast": true, "content": "task completed"}
```

Files that match neither shape are logged as invalid and removed. Because files are processed in lexical order, zero-padded sequence prefixes (e.g. `0001_`, `0002_`) preserve send order — this is exactly what `swarm-msg` writes for you.

## The `swarm-msg` helper

`swarm-msg` is the agent-side messaging CLI available inside agent sandboxes (the script ships at the repo root and is mounted into the sandbox). It writes outbox files for you, so skills can call it instead of formatting JSON by hand. (The name `swarm-msg` is intentional — it belongs to the agent side and is not renamed.)

| Command | Description |
|---------|-------------|
| `swarm-msg send <agent> <message>` | Send a message to an agent via the outbox |
| `swarm-msg send <agent> -f <file>` | Send a file's contents to an agent (combined with `<message>` if both given) |
| `swarm-msg ask <object> <message>` | Send **and block** for the object's reply, printing a JSON envelope (synchronous — see below) |
| `swarm-msg broadcast <message>` | Broadcast to all connected agents via the outbox |
| `swarm-msg list` | List agents you can message (from the `SWARM_TOPOLOGY` env var) |
| `swarm-msg send-stdout <agent> <msg>` | Legacy: send via the stdout `SWARM_MSG` protocol |
| `swarm-msg broadcast-stdout <msg>` | Legacy: broadcast via the stdout `SWARM_MSG` protocol |
| `swarm-msg help` | Show usage (also `--help`, `-h`, or no args) |

Examples:

```bash
# Send a JSON payload to another agent
swarm-msg send coder '{"action":"fix_result","status":"fixed"}'

# Send a state report to the metrics system object
swarm-msg send metrics '{"action":"state_report","data":{"state":{"count":42}}}'

# Broadcast to everyone you are connected to
swarm-msg broadcast "Phase complete, ready for next"

# Send the contents of a file
swarm-msg send reviewer -f /workspace/fix.patch
```

`send` and `broadcast` write zero-padded JSON files into `/workspace/.outbox/`, which the router picks up automatically. The sequence number is derived from the count of existing `*.json` files already in the outbox, so a directed send becomes e.g. `0001_coder.json` and a broadcast becomes `0001_broadcast.json`. The legacy `send-stdout` and `broadcast-stdout` subcommands instead print `SWARM_MSG` markers to stdout for log-based routing.

### Synchronous `ask` (request/response with an object)

`send` is fire-and-forget. `swarm-msg ask <object> <message>` is the **synchronous** motion: it publishes the same outbox message but tags it with a `reply_to` correlation id and **blocks** until the engine writes the object's reply, then prints exactly one well-formed JSON envelope to stdout — so a shell-tool caller receives the result **inline in the same turn**:

```json
{"ok":true,"result":{...},"error":null,"timeout":false}
```

On timeout (`SWARM_ASK_TIMEOUT` seconds, default 30) it prints an `ok:false`/`timeout:true` envelope instead of hanging; a route denial or missing target likewise returns a typed `ok:false` envelope. An `error.type` of `"permanent"` means retrying the same ask can never succeed. `ask` targets an **object** with a return-path edge back to the agent.

**Task gating while awaiting.** When an agent sends to an object that has a return edge to it (an async reply is expected), the agent enters an *awaiting* state: new **user** tasks are queued in its Inbox rather than forwarded to the backend immediately, preserving reply ordering and preventing mis-correlation. The flag clears when the reply arrives or a safety timeout (default 90 s) fires. (`ask` is the synchronous surface over this mechanism.)

**`reply_to` auto-delivery.** An agent can be configured with a `reply_to:` object in its config; each turn's derived reply text is then delivered to that sink object automatically, once per turn — unless the agent already sent to that target during the turn. This is opt-in, for reply-sink topologies.

> `swarm-msg` JSON-encodes message bodies with `jq` (for `send`) or `python3` (for `broadcast`), falling back to a `sed`/`awk` escaper when those tools are absent — so the preset's available tools affect encoding fidelity for unusual payloads.

### `SWARM_TOPOLOGY` for `swarm-msg list`

Inside a container, `swarm-msg list` reads the `SWARM_TOPOLOGY` environment variable — a comma-separated list of the targets the agent is connected to — and prints them. If `SWARM_TOPOLOGY` is unset, it reports `Topology not available.`

```bash
$ swarm-msg list
Agents you can message:
  - coder
  - reviewer
```

## See also

- [configuration.md](configuration.md) — defining agents, objects, and the `topology:` edges that gate routing
- [objects.md](objects.md) — non-agentic handlers that send and receive routed messages, including system objects
- [skills.md](skills.md) — skill files that drive what agents say and how they call `swarm-msg`

---

---
description: Objects in GenSwarms: deterministic, non-agentic Elixir components that participate in the swarm topology.
---

# Objects

Objects are non-agentic components of a swarm. Where an agent is backed by an LLM
that produces free-form text, an object is a plain Elixir module that runs
deterministic code. Objects participate in the swarm topology exactly like
agents: they receive messages, hold state, and send messages to other agents or
objects. They are the right tool for game referees, evaluators, gateways,
schedulers, validators, and bridges between swarms.

Each object is hosted by a `Genswarms.Objects.ObjectServer` (a GenServer). For a
native object the server delegates to a module that implements the
`Genswarms.Objects.ObjectHandler` behaviour. (Objects can also be backed by a
Docker or SSH process that speaks the same JSON protocol over stdin/stdout; this
guide focuses on native handlers, which are the common case.)

## The ObjectHandler behaviour

A native object is any module that declares `@behaviour
Genswarms.Objects.ObjectHandler` and implements its callbacks.

```elixir
defmodule ExampleSwarm.Objects.Evaluator do
  @behaviour Genswarms.Objects.ObjectHandler

  @impl true
  def init(config) do
    {:ok, %{config: config, results: []}}
  end

  @impl true
  def handle_message(from, content, state) do
    {:reply, "ack", state}
  end

  @impl true
  def interface do
    %{evaluate: %{input: "JSON list of configs", output: "JSON with results"}}
  end
end
```

### Callbacks

| Callback | Required | Purpose |
|----------|----------|---------|
| `init(config)` | yes | Build the handler's initial state from its declared config map. |
| `handle_message(from, content, state)` | yes | React to a message routed from another node. |
| `interface()` | yes | Return a schema describing the object's actions (introspection). |
| `handle_info(msg, state)` | no | Handle process messages such as timers. |
| `terminate(reason, state)` | no | Cleanup when the object stops. |

`handle_info/2` and `terminate/2` are the only optional callbacks — the
behaviour declares `@optional_callbacks [terminate: 2, handle_info: 2]`. The
`ObjectServer` checks at runtime (with `function_exported?/3`) whether the
handler exports them before calling them, so you only implement them if you
need them.

## init/1

`init/1` is called once when the `ObjectServer` starts. `config` is the map you
provide in the swarm configuration under the object's `:config` key.

The behaviour's published typespec covers the two-tuple and the single-send
forms:

```elixir
@callback init(config :: map()) ::
            {:ok, state}
            | {:ok, state, {:send, to, content}}
            | {:error, reason}
```

In addition, the `ObjectServer` also honors a `{:multi, messages}` form at
runtime (see the table below).

| Return value | Semantics |
|--------------|-----------|
| `{:ok, state}` | Initialize with `state`; do nothing else. |
| `{:ok, state, {:send, to, content}}` | Initialize, then send an opening message to `to`. |
| `{:ok, state, {:multi, messages}}` | Initialize, then send several opening messages (see below). |
| `{:error, reason}` | Initialization failed; the object enters its `:error` state and drops any messages delivered to it. |

The `{:ok, state, {:send, to, content}}` form is how an object kicks off a
conversation. The tic-tac-toe game (`examples/tic-tac-toe/objects/game.ex`) uses
it to send the first turn to the opening player:

```elixir
@impl true
def init(_config) do
  board = [[".", ".", "."], [".", ".", "."], [".", ".", "."]]
  state = %{board: board, turn: :player_x, game_over: false, winner: nil, move_count: 0}
  {:ok, state, {:send, :player_x, encode(:your_turn, %{board: board})}}
end
```

The `{:ok, state, {:multi, messages}}` form accepts a list of
`{:send, to, content}` and `{:broadcast, content}` tuples and dispatches all of
them after initialization. (Bare `{target, msg}` pairs are *not* accepted in the
`init/1` multi form; that shorthand only exists for `handle_message/3`'s
`:send_many`.)

## handle_message/3

`handle_message/3` runs for every message routed to the object. `from` is the
sender's name (an atom), `content` is the message string, and `state` is the
current handler state. The return tuple tells the `ObjectServer` what to send
and how to update state.

The behaviour's published typespec covers the four common forms:

```elixir
@callback handle_message(from :: atom(), content :: String.t(), state) ::
            {:reply, response, new_state}
            | {:send, to, content, new_state}
            | {:broadcast, content, new_state}
            | {:noreply, new_state}
```

The handler may also return the multi-message tuples below. The full set of
return tuples honored by the `ObjectServer` dispatch is:

| Return tuple | Semantics |
|--------------|-----------|
| `{:reply, response, new_state}` | Route `response` back to the original sender (`from`). |
| `{:send, to, content, new_state}` | Route `content` to a specific node `to`. |
| `{:broadcast, content, new_state}` | Send `content` to every node connected to this object in the topology. |
| `{:noreply, new_state}` | Update state only; send nothing. |
| `{:send_many, messages, new_state}` | Send several messages at once (flexible item shapes — see below). |
| `{:multi, messages, new_state}` | Send several messages at once (tagged item shapes only). |

All routed targets are subject to the topology: a message only reaches `to` if
there is an edge from this object to `to` (or `to` is a system object — see
below). After any of these returns the object goes back to its `:idle` state and
its `message_count` is incremented.

### `:send_many` vs `:multi`

Both forms emit multiple messages from a single callback return. They differ
only in the item shapes they accept.

`:multi` accepts tagged tuples only — `{:send, to, msg}` and `{:broadcast, msg}`:

```elixir
{:multi,
 [
   {:send, :player_x, "your move"},
   {:broadcast, "game starting"}
 ], new_state}
```

`:send_many` accepts those tagged tuples *and* bare `{target, msg}` pairs, so you
can mix styles:

```elixir
{:send_many,
 [
   {:player_x, "your move"},          # bare {target, msg}
   {:send, :player_o, "stand by"},    # tagged send
   {:broadcast, "game starting"}      # tagged broadcast
 ], new_state}
```

Use `:send_many` when it is convenient to build a keyword-like list of
`{target, msg}` pairs; use `:multi` when you want every item explicitly tagged.

### Worked example: a turn-validating game object

The tic-tac-toe `Game` object (`examples/tic-tac-toe/objects/game.ex`, module
`TicTacToe.Objects.Game`) shows the common return tuples in one handler. It
replies to the sender on an invalid move, sends the next turn to the other
player on a valid move, and broadcasts the final result when the game ends.

```elixir
@impl true
def handle_message(from, content, state) do
  cond do
    state.game_over ->
      {:reply, encode(:error, "Game over. #{winner_msg(state.winner)}"), state}

    from != state.turn ->
      {:reply, encode(:error, "Not your turn, waiting for #{state.turn}"), state}

    true ->
      process_move(from, content, state)
  end
end

defp process_move(from, content, state) do
  # ... validate, update board ...
  case check_result(new_board) do
    {:win, p} ->
      winner = if p == "X", do: :player_x, else: :player_o
      {:broadcast, encode(:game_over, %{board: new_board, winner: winner}), final}

    :draw ->
      {:broadcast, encode(:game_over, %{board: new_board, winner: "draw"}), final}

    :continue ->
      {:send, next, encode(:your_turn, %{board: new_board}), new_state}
  end
end
```

## handle_info/2 for timers and process messages

Objects are GenServers, so they can receive ordinary process messages. Implement
the optional `handle_info/2` callback to react to timers scheduled with
`Process.send_after/3` or other Erlang messages. It returns the same tuples as
`handle_message/3` (including `:send_many` and `:multi`):

```elixir
@impl true
def init(_config) do
  Process.send_after(self(), :tick, 1_000)
  {:ok, %{ticks: 0}}
end

@impl true
def handle_info(:tick, state) do
  Process.send_after(self(), :tick, 1_000)
  {:broadcast, "tick #{state.ticks}", %{state | ticks: state.ticks + 1}}
end
```

A `:reply` returned from `handle_info/2` has no original sender to reply to, so
the `ObjectServer` logs it and treats it as a state-only update.

## interface/0 introspection

`interface/0` returns a map describing the actions the object supports and their
expected input/output. It is surfaced through `ObjectServer.get_interface/2` for
display in tooling and dashboards and does not affect routing. By convention each
key is an action name pointing at a map with `:input` and `:output`
descriptions.

```elixir
@impl true
def interface do
  %{
    move: %{
      input: ~s({"board": [["X",".","."],[".",".","."],[".",".","."]]}),
      output: "Validates move, sends board to next player or announces winner"
    }
  }
end
```

## Logging from an object

Handlers can write structured entries to the centralized event log via
`Genswarms.Objects.ObjectServer.log/5`:

```elixir
alias Genswarms.Objects.ObjectServer

ObjectServer.log(:info, "tic-tac-toe", :game, "Move accepted", %{player: from})
```

The arguments are `level`, `swarm_name`, `object_name`, `message`, and an
optional `metadata` map (defaults to `%{}`):

```elixir
@spec log(level, swarm_name, object_name, message, metadata \\ %{}) :: term()
```

Internally this calls `LogStore.log/4` with `source: :object` and
`event: :custom`, tagging the entry with the swarm and object names. See
[observability.md](observability.md) for how these events are queried and
streamed.

## Declaring objects in a swarm

Objects are listed under the `:objects` key of a swarm configuration. Each entry
needs a `:name` and a `:handler`; the optional `:config` map is passed verbatim
to the handler's `init/1`. Objects appear in `:topology` edges just like agents.

The snippet below is the tic-tac-toe swarm
(`examples/tic-tac-toe/tic_tac_toe_swarm.exs`), trimmed for brevity:

```elixir
# Load the object handler module before referencing it in config.
Code.require_file("objects/game.ex", __DIR__)

%{
  name: "tic-tac-toe",
  agents: [
    %{
      name: :player_x,
      backend: {:docker, "szc-agent-code:latest", %{memory_limit: "512m"}},
      skills: [Path.join([__DIR__, "skills", "player_x.md"])],
      model: "minimax/minimax-m2.7"
    },
    %{
      name: :player_o,
      backend: {:docker, "szc-agent-code:latest", %{memory_limit: "512m"}},
      skills: [Path.join([__DIR__, "skills", "player_o.md"])],
      model: "minimax/minimax-m2.7"
    }
  ],
  objects: [
    %{
      name: :game,
      handler: TicTacToe.Objects.Game,
      config: %{}
    }
  ],
  topology: [
    {:player_x, :game},
    {:game, :player_x},
    {:player_o, :game},
    {:game, :player_o}
  ]
}
```

The `config` map is how you parameterize an object. A bridge object, for
instance, might receive its swarm name and a routing table:

```elixir
objects: [
  %{
    name: :bridge,
    handler: ExampleSwarm.Objects.Bridge,
    config: %{
      swarm_name: "example-swarm",
      routing: %{messenger_a: {"swarm-b", :messenger_b}}
    }
  }
]
```

See [configuration.md](configuration.md) for the full configuration DSL.

## System objects

The router always allows messages to three reserved system object names, even
when no explicit topology edge exists:

| Name | Purpose |
|------|---------|
| `:metrics` | Metrics sink. |
| `:tick` | Clock / scheduling. |
| `:gateway` | External gateway. |

These are defined as `@system_objects [:metrics, :tick, :gateway]` in
`lib/genswarms/routing/router.ex`. Any node may send to them without declaring an
edge; define a handler for them only if you want to act on what they receive.

## See also

- [configuration.md](configuration.md) — declaring objects and topology
- [messaging.md](messaging.md) — how messages are routed between nodes
- [programmatic.md](programmatic.md) — driving swarms from Elixir code
- [observability.md](observability.md) — querying and streaming object log events

---

---
description: Author per-agent skills in GenSwarms — plain markdown instructions with template variables resolved per agent at deploy time.
---

# Skills

Skills are markdown files that define an agent's role, capabilities, and behavior. Each agent is assigned a list of skills in its config; when the agent starts, those files are copied into the agent's own skills directory with template variables resolved per agent.

Two components are involved:

- `Genswarms.Skills.SkillsManager` — a GenServer that loads the skills repository (`priv/skills` by default) into an ETS cache on startup and serves skill content over the REST API.
- `Genswarms.Agents.AgentServer` — at agent start, `prepare_skills/1` resolves each skill entry to a source path, substitutes template variables, and writes the result into the agent's per-agent skills directory.

## What a skill is

A skill is a plain markdown file — there is no special schema. Its contents become part of the agent's instructions. A typical skill describes the agent's role, lists capabilities and guidelines, and explains how to communicate with other agents.

```markdown
# My Custom Skill

You are a specialist in [domain]. Your role is to [description].

## Capabilities
- Capability 1
- Capability 2

## Guidelines
1. Guideline 1
2. Guideline 2

## Communication
When communicating with other agents, use the @agent_name: prefix.
```

See [messaging.md](messaging.md) for the `@agent_name:` syntax skills should reference.

## Assigning skills in config

List skills on each agent with the `skills:` key. Plain filenames are resolved against the skills directory (`priv/skills` by default).

```elixir
%{
  name: "example-swarm",
  agents: [
    %{name: :researcher, backend: :local, skills: ["web.md"]},
    %{name: :coder, backend: :local, skills: ["code.md", "review.md"]}
  ],
  topology: [{:researcher, :coder}]
}
```

Each skill entry is resolved to a source path by `AgentServer.prepare_skills/1` in one of three ways:

| Entry form | Resolved against |
|------------|------------------|
| Absolute path (`/opt/skills/custom.md`) | used as-is |
| Relative path starting with `.` (`./skills/custom.md`, `../shared.md`) | the project root (`:project_root` app env, falling back to the current working directory) |
| Anything else — a simple filename (`web.md`) | the skills directory (`priv/skills` by default) |

In every case only the basename (`Path.basename/1`) is used for the destination file inside the agent's skills directory. The agent's skills directory is `<swarm_data_dir>/<swarm>/<agent>/skills`, where `swarm_data_dir` defaults to `~/.subzeroclaw/swarms`.

## Built-in skills

The repository ships these skills in `priv/skills/`:

| Skill | Description |
|-------|-------------|
| `web.md` | Web research specialist — search, summarize, and cite sources |
| `code.md` | Code implementation specialist — write, refactor, debug, and test code |
| `review.md` | Code review specialist — review for correctness, security, and quality |
| `secret.md` | Minimal example skill used in tests |
| `swarm_architect.md` | Designs swarm topologies and agent configurations |
| `swarm-fixer.md` | Diagnoses and repairs swarm issues |

On startup, `SkillsManager` loads every `*.md` file from the skills directory into an ETS cache. The skills directory defaults to `priv/skills` and is configured by the `:skills_dir` application environment key.

> The `:skills_dir` app env key is populated from the `SKILLS_DIR` OS
> environment variable in `config/config.exs` and `config/runtime.exs`
> (`skills_dir: System.get_env("SKILLS_DIR", "priv/skills")`). So setting the
> `SKILLS_DIR` environment variable and setting the `:genswarms, :skills_dir`
> application key are the same thing — `SKILLS_DIR` is the user-facing knob,
> `:skills_dir` is where it lands internally. See
> [getting-started.md](getting-started.md) for the environment variable list.

## Template variables

Skills support template variables that are substituted when the skill is deployed to a specific agent. Resolution happens at agent start, per agent, in `AgentServer.prepare_skills/1` via a literal string replacement of the following tokens:

| Variable | Resolved to |
|----------|-------------|
| `{{agent_name}}` | the agent's name (e.g. `fixer_3`) |
| `{{swarm_name}}` | the swarm name |
| `{{workspace}}` | the agent's workspace path (the `:workspace` backend config key, or `""` if unset) |

These are the only template variables. Any other `{{...}}` token is left untouched.

```markdown
# Fixer Agent

You are {{agent_name}} in the {{swarm_name}} swarm.
Your workspace is {{workspace}}.

Write output files to your workspace directory.
```

## Creating custom skills

Drop a new markdown file into `priv/skills/` (or point an agent at any path using the relative/absolute forms above), then reference it from the agent's `skills:` list.

```bash
# Add a skill file
$ cat > priv/skills/planner.md <<'EOF'
# Planner Skill

You are {{agent_name}}, the planner for {{swarm_name}}.
Break tasks into steps and delegate them with @agent_name: prefixes.
EOF
```

```elixir
%{name: :planner, backend: :local, skills: ["planner.md"]}
```

`SkillsManager.reload_skills/0` clears the ETS cache and reloads every skill from the skills directory on disk, which is useful while iterating during development:

```elixir
Genswarms.Skills.SkillsManager.reload_skills()
```

Note that this refreshes the repository cache used by the REST API; agents copy their skills at start, so already-running agents keep the skill files they were deployed with until they restart.

## Per-agent workspaces

Each agent's workspace is the `workspace` key inside its `config` map (a backend
key — see [configuration.md](configuration.md)). It is mounted read-write into
the sandbox and is where the file-inbox and file-outbox live (see
[messaging.md](messaging.md)).

```elixir
%{
  name: :fixer,
  backend: :bwrap,
  config: %{workspace: "/tmp/example-swarm/fixer"}
}
```

To run a pool of identical agents, scale the group at runtime with
[`genswarms scale`](cli.md) (or `SwarmManager.scale_agent_group/4`, or the scale
REST endpoint). Scaling uses an existing group member's spec as a template and
creates `fixer_1`, `fixer_2`, … Each replica gets its own `workspace`, derived
by `maybe_rename_workspace/4` in `swarm_manager.ex`:

- If the workspace ends with `/` followed by the template agent's name, that
  suffix is replaced with the replica name.
  `/tmp/example-swarm/fixer` → `/tmp/example-swarm/fixer_1`,
  `/tmp/example-swarm/fixer_2`.
- Otherwise the replica name is appended as a path segment (`Path.join/2`).
  `/tmp/example-swarm/work` →
  `/tmp/example-swarm/work/fixer_1`, `/tmp/example-swarm/work/fixer_2`.

Because `{{workspace}}` and `{{agent_name}}` are resolved per instance, a single
templated skill file produces correct, instance-specific instructions across the
whole pool.

> Note: there is no config-time `count:` key. An agent definition always maps to
> one agent; multiple instances come from runtime scaling.

## Skills over the REST API

`SkillsManager` (the repository) and the per-agent skills directories are exposed through the API. See [rest-api.md](rest-api.md) for full request/response details.

| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/skills` | List available skills in the repository |
| GET | `/api/skills/:name` | Get a skill's content |
| GET | `/api/swarms/:swarm_name/agents/:agent_name/skills` | Get a deployed agent's skills |
| PUT | `/api/swarms/:swarm_name/agents/:agent_name/skills/:skill_name` | Update a deployed agent's skill |

## See also

- [configuration.md](configuration.md) — assigning `skills:` on agents
- [messaging.md](messaging.md) — the `@agent_name:` syntax and `swarm-msg` that skills drive
- [rest-api.md](rest-api.md) — skills endpoints and agent skill management
- [getting-started.md](getting-started.md) — the `SKILLS_DIR` environment variable

---

---
description: GenSwarms execution backends — Local, persistent tmux TUIs, Docker, Apple container, SSH, Bubblewrap, and Mock — and how to choose one per agent.
---

# Backends

A backend is how GenSwarms runs an agent runtime. Most backends launch
`subzeroclaw`; the tmux backend launches an existing interactive coding client.
Every agent declares a `backend:`, and GenSwarms uses the matching module to
start it, deliver input, expose health/session state, and stop it. All backends
implement `Genswarms.Backends.BackendBehaviour`, so the topology and router do
not depend on the selected runtime.

This guide covers each backend: how it runs, the config it accepts, and what you need on the host.

## The backend contract

Every backend implements `Genswarms.Backends.BackendBehaviour` (`lib/genswarms/backends/backend_behaviour.ex`). The callbacks are:

| Callback | Required? | Purpose |
|----------|-----------|---------|
| `start/2` | yes | Start the agent process; returns `{:ok, ref}` or `{:error, term}` |
| `stop/1` | yes | Stop the running agent |
| `send_input/2` | yes | Deliver a message; may return backend metadata such as a turn ID |
| `deploy_skills/2` | yes | Make skills available to the agent |
| `health_check/1` | yes | Report whether the agent is alive (`:ok` or `{:error, reason}`) |
| `backend_type/0` | yes | Return the backend's atom (e.g. `:local`) |
| `handle_output/2` | optional | Parse raw output into messages |
| `capabilities/0` | optional | Advertise readiness events, interrupt, persistence, and raw terminal support |
| `interrupt/1` | optional | Interrupt the current turn without destroying the worker |
| `session_info/1` | optional | Return non-secret attach/session metadata |
| `disconnect/1` / `destroy/1` | optional | Separate orchestrator disconnect from intentional resource destruction |
| `acknowledge/2` | optional | Confirm that a durable turn completion was handled |

Backends without the optional lifecycle callbacks retain the original Port-style behavior. Event-driven backends receive an `event_sink` and a per-start `backend_id`; the ID prevents delayed events from an older backend generation from mutating a restarted agent.

The subzeroclaw backends share the `szc-wrapper` wire protocol, which translates
between JSON lines and subzeroclaw's plain-text interface. Tmux uses typed
lifecycle events and durable per-turn files instead; its terminal stream remains
diagnostic and human-facing.

## Choosing a backend

| Backend | When to use | Isolation level |
|---------|-------------|-----------------|
| `:local` | Development, debugging, single-host runs | None (plain subprocess) |
| `{:tmux, client}` | Warm, human-attachable Codex/Claude/OpenCode sessions | Selectable: host, per-agent Docker, or per-agent bwrap |
| `{:docker, "name"}` | Reproducible tool environments, per-agent images | Container (namespaces + image) |
| `{:apple_container, "name"}` | OCI containers on macOS / Apple silicon without Docker Desktop | Apple container VM |
| `{:ssh, "user@host"}` | Bare-metal / remote NixOS machines | Remote host |
| `:bwrap` | Massive scale (10k+ agents on one box) | Lightweight sandbox (user namespaces) |
| `:mock` | Tests without LLM calls | None (no process spawned) |

Every *real* backend (local, docker, apple_container, ssh, bwrap) resolves the LLM settings from the agent config. `api_key` and `endpoint` fall back to the process environment (`SUBZEROCLAW_API_KEY`, `SUBZEROCLAW_ENDPOINT`) when not set in config. The **`model` has no environment fallback**: `SUBZEROCLAW_MODEL` is a dead variable that `subzeroclaw` no longer reads. A config-level `model` is passed through as `SUBZEROCLAW_REQUEST_EXTRA = {"model": …}`; for router routing you set `request_extra` directly (a `policy_ir`), and when neither is set `subzeroclaw` uses its own default. The `:mock` backend ignores all of this — it never spawns a process.

## Local

The local backend (`lib/genswarms/backends/local_backend.ex`) spawns subzeroclaw as an Elixir `Port` subprocess and communicates over stdin/stdout. It is the simplest backend and the easiest to debug, but provides no isolation — the agent runs as your user with full access to the host. `stop/1` terminates the whole OS process tree (SIGTERM, then SIGKILL after a short grace) rather than just closing stdin, so a wedged agent and its children are cleaned up.

```elixir
%{
  name: :researcher,
  backend: :local,
  skills: ["research.md"],
  model: "anthropic/claude-sonnet-4"
}
```

It launches the `szc-wrapper` script, which in turn runs the `subzeroclaw` binary. Both paths are resolved from config or application environment:

| Config key | Purpose | Resolution order |
|------------|---------|------------------|
| `wrapper_path` | Path to the wrapper script | config `:wrapper_path` → app env `:wrapper_path` → `priv/szc-wrapper-fifo.sh` |
| `subzeroclaw_path` | Path to the subzeroclaw binary | config `:subzeroclaw_path` → app env `:subzeroclaw_path` → `"subzeroclaw"` (from `PATH`) |
| `api_key` | LLM API key | config → `SUBZEROCLAW_API_KEY` env |
| `model` | Model identifier | config only → wrapped into `SUBZEROCLAW_REQUEST_EXTRA` as `{"model": …}` (no `SUBZEROCLAW_MODEL` env fallback — it is dead) |
| `endpoint` | LLM endpoint | config → `SUBZEROCLAW_ENDPOINT` env |
| `request_extra` | Router routing/body-override JSON (`policy_ir`) | config → `SUBZEROCLAW_REQUEST_EXTRA` env |
| `compact_extra` | Async compaction JSON (`keep_recent` + summariser policy) | config → `SUBZEROCLAW_COMPACT_EXTRA` env |

The wrapper is invoked as `<wrapper_path> <name> <subzeroclaw_path> <skills_dir>`. When a `skills_dir` is present, its expanded path is also exported to the subprocess as the `SUBZEROCLAW_SKILLS` environment variable; the agent name is exported as `SUBZEROCLAW_AGENT_NAME`.

Requirements: a `subzeroclaw` binary on the host (on `PATH` or via `subzeroclaw_path`).

## Tmux persistent TUI

The tmux backend (`lib/genswarms/backends/tmux_backend.ex`) runs an existing
interactive coding client in a persistent pane instead of flattening it into a
one-shot command. Supported clients are Codex, Claude Code, and OpenCode:

```elixir
%{
  name: :coder,
  backend: {:tmux, :codex, %{
    workspace: "/home/me/project",
    runner: :docker,
    image: "coding-tuis:latest",
    client_source: :runtime,
    approval_policy: :on_request,
    sandbox: :workspace_write
  }}
}
```

Use `:claude` or `:opencode` as the second tuple element for those clients.
String client names (`"codex"`, `"claude"`, `"opencode"`) are also accepted.
There is intentionally no bare `:tmux` form: the client must be explicit.

tmux itself always runs on the host and remains the observation/control plane.
The pane command can run the client directly (`runner: :host`), enter one
dedicated container with `docker exec -it` (`runner: :docker`), or enter one
dedicated bubblewrap sandbox (`runner: :bwrap`). The latter two give every agent
its own process/filesystem boundary while preserving the same host-side attach
command.

### Lifecycle and durable turns

One tmux socket contains one named session per swarm and one window per agent.
GenSwarms stores the stable pane ID and emits normalized lifecycle states:
`starting`, `ready`, `running`, `blocked`, `needs_attention`, `interrupted`, and
`stopped`. Screen capture is used for human visibility and conservative prompt
detection; it is **not** treated as a delivery acknowledgement.

Each turn gets a directory under:

```text
<workspace>/.genswarms/turns/<swarm>/<agent>/<turn-id>/
├── task.md
├── reply.md
├── done.json
├── ack.json
└── interrupted.json
```

The backend writes `task.md` atomically and sends only a short path-based nudge
with `tmux send-keys`. It briefly separates the literal paste from `Enter`. If
the exact nudge is still present at the active cursor after the grace period,
the backend retries only `Enter` once; it never pastes the task text twice. A
nudge that remains staged moves the turn to `needs_attention`. This is bounded
TUI recovery, not an acceptance acknowledgement. The client completes the
contract by writing `reply.md`
and atomically renaming a `done.json.tmp` receipt to `done.json`. GenSwarms
processes the reply and then writes `ack.json`. The dispatcher still waits for
a fresh recognized prompt before sending queued work. A task without `ack.json` is
recovered as `needs_attention` after an orchestrator restart; a completed but
unacknowledged turn may therefore be delivered again. The stable turn ID makes
that replay visible in the artifacts and session metadata; output delivery is
still at-least-once, not exactly-once.

There are no lock files or automatic retry engine. A follow-up task remains in
the existing GenSwarms inbox until the current turn completes. If the pane is
alive, an AgentServer restart disconnects and reattaches without killing the
conversation. If the pane process is dead, it is respawned; set `resume: true`
or a client session ID to ask the CLI to restore its own conversation. An
explicit agent/swarm stop destroys the tmux window.

### Attaching and interrupting

`GET /api/swarms/<swarm>/agents/<agent>/session` returns the socket, session,
window, pane ID, and argv for read-only or read-write attachment. The equivalent
command is normally:

```bash
tmux -L genswarms attach-session -r -t genswarms-<swarm>:<agent>  # observe only
tmux -L genswarms attach-session -t genswarms-<swarm>:<agent>     # interactive
```

Use `POST /api/swarms/<swarm>/agents/<agent>/interrupt` to send `C-c` to the
current pane without deleting it. The raw captured terminal is also published
on the in-process swarm terminal PubSub topic for BEAM-side UI consumers; it is
not currently forwarded through the public WebSocket channel.

### Options

| Config key | Purpose | Default |
|------------|---------|---------|
| `workspace` | Client working directory and durable turn root | `/tmp/genswarms-tmux/<swarm>/<agent>` |
| `runner` | Execution boundary: `:host`, `:docker`, or `:bwrap` | `:host` |
| `client_source` | `:runtime` (binary already in image/base) or `:host_nix` (read-only host Nix closure) | Docker: `:runtime`; bwrap: `:host_nix` |
| `image` | Persistent per-agent Docker container image | preset/default image |
| `state_dir` | Private host directory mounted as `/root` for client auth/config/session state | per-swarm/agent/client temp path |
| `network` | Host/open, Docker network name, or `:none`; see below | `:open` |
| `pass_env` | Names of existing host variables to pass to the isolated runtime | `[]` |
| `runner_env` | Explicit isolated-runtime environment map | `%{}` |
| `extra_ro_binds` / `extra_rw_binds` | Additional `{host, runtime}` mounts | `[]` |
| `memory_limit` | Per-agent memory limit | bwrap rootless: unset; bwrap cgroup: `"2G"` |
| `privilege_mode` | bwrap launcher: `:rootless` (keeps the pane's PTY) or explicit `:cgroup` | `:rootless` |
| `docker_executable` / `bwrap_executable` | Override isolation CLI executable | resolved from `PATH` |
| `xargs_executable` | Override the NUL-safe bwrap host-argv launcher | resolved from `PATH` |
| `model` | Native client model flag | client default |
| `resume` | `true` for the most recent session, or a client session ID | `false` |
| `tmux_socket` | Dedicated tmux socket name | `genswarms` |
| `tmux_executable` / `executable` | Override tmux/client executable | resolved from `PATH` |
| `args` | Extra client argv strings (never host-shell interpolated) | `[]` |
| `approval_policy` / `sandbox` | Codex approval and sandbox modes | client config/default |
| `permission_mode` / `effort` | Claude permission and effort modes | client config/default |
| `auto_approve` | OpenCode `--auto` | `false` |
| `dangerously_bypass` | Explicit Codex/Claude permission bypass | `false` |
| `poll_interval_ms` | Pane/receipt poll interval | `250` |
| `submit_delay_ms` / `submit_retry_after_ms` | Delay before initial `Enter` / cursor-check grace | `100` / `1000` |
| `submit_max_attempts` / `submit_check_max_errors` | Total Enter attempts / cursor-query errors before attention | `2` / `3` |
| `history_lines` / `state_lines` | Captured history / visible tail used for state recognition | `200` / `24` |
| `quiet_ready_fallback` | Allow a stable non-empty screen to count as ready | `false` |
| `max_reply_bytes` | Maximum accepted `reply.md` size | 1 MiB |

Dangerous bypass flags are never enabled implicitly. A blocked trust or
permission prompt moves the agent to `blocked` so a human can attach and decide.
Because a TUI is not a machine protocol, prompt recognition is deliberately
conservative; `quiet_ready_fallback` is opt-in.

### Per-agent Docker and bwrap runners

The isolated runners use this runtime contract:

| Runtime path | Source | Access |
|--------------|--------|--------|
| `/workspace` | the agent's host workspace | read/write |
| `/root` | the agent's private `state_dir` | read/write |
| `/skills` | deployed skills, when present | read-only |
| `/nix/store/...` | selected client closure with `client_source: :host_nix` | read-only |

Environment selected by `pass_env`/`runner_env` is not placed in the bwrap or
tmux argv. The runner writes a shell-quoted mode-0600 file under the private
`state_dir` and invokes a constant bootstrap path inside `/root`; the agent can
read those values because they are part of its granted authority, while host
process listings and systemd metadata cannot.

A minimal Nix closure can require hundreds of read-only bind arguments, more
than tmux accepts in one control command. The bwrap runner writes those
arguments as a private, NUL-delimited `host-launch.argv0` manifest and invokes
them once with `xargs --null --exit --max-args=<exact count>`. No host shell
parses the manifest, argument boundaries are preserved, and the short tmux
command contains only the manifest path and launcher metadata. GNU `xargs` is
therefore a host requirement for the bwrap TUI runner.

Docker creates one persistent container named `gstui-<swarm>-<agent>` and
refuses to reuse a same-named container whose ownership, mounts, image, network,
resource limits, or configured environment differ. bwrap creates one
copy-on-write root under `/run/swarm/agents/gstui-<swarm>-<agent>` and stores a
private contract fingerprint so a live pane cannot be reattached under changed
mounts, network, limits, executable, or environment. Explicit destroy removes
the container/overlay; an orchestrator disconnect leaves both the pane and its
boundary alive for reattachment.

There are two ways to supply a client:

- `client_source: :runtime` expects `codex`, `claude`, or `opencode` to be in the
  image/base runtime's `PATH`. This is the Docker default and is the most
  portable production setup.
- `client_source: :host_nix` resolves the selected host executable to
  `/nix/store`, queries its minimal closure, and mounts that closure read-only.
  It rejects arbitrary non-Nix host binaries. This is the bwrap default and is
  also useful with a minimal Docker image on NixOS.

Example using the locally installed Nix client in bwrap:

```elixir
%{
  name: :reviewer,
  backend: {:tmux, :claude, %{
    runner: :bwrap,
    client_source: :host_nix,
    privilege_mode: :rootless,
    network: :open
  }}
}
```

`network: :none` gives Docker or bwrap a real no-network namespace. That also
blocks cloud LLM APIs, so it is primarily useful for local providers or
offline/testing work. Docker additionally accepts a named Docker network;
bwrap accepts only `:open` and `:none`. The subzeroclaw-specific
`network: :isolated` mode (LLM-only forwarding) is not yet valid for an
interactive TUI and fails closed with `:tui_egress_isolation_unsupported`.

tmux itself is not the security boundary: it stays on the trusted host. Do not
mount its socket into the agent. Extra read/write binds enlarge the agent's
authority, and the private state directory may contain client credentials.
Dangerous client bypass flags remain explicit. In particular, only set
`dangerously_bypass: true` for Codex/Claude when the outer Docker/bwrap policy is
strong enough for the workload; it is never inferred merely because a runner
was selected.

Requirements: host-side tmux; Docker or bwrap for the selected runner; and
either a runtime image/base containing the selected client or a Nix-installed
host client plus `nix-store`.

## Docker

The Docker backend (`lib/genswarms/backends/docker_backend.ex`) runs each agent in a NixOS-based container. It is the right choice when agents need specific, reproducible tool sets, since the tools are baked into the image rather than your host.

```elixir
%{
  name: :coder,
  backend: {:docker, "coder"},
  presets: [:base, :code],
  skills: ["code.md"]
}
```

You can also pass options as a third tuple element:

```elixir
%{
  name: :coder,
  backend: {:docker, "coder", %{memory_limit: "512m", network: "swarmnet"}},
  skills: ["code.md"]
}
```

### Container naming and multi-swarm namespacing

Containers are named `szc-{swarm}-{agent}` unless you override the name with the `container` key. The swarm name is part of the name, so the same agent name in two different swarms maps to two distinct containers and they never collide. On start, if a container with that name already exists (running, paused, exited, or otherwise), it is forcibly removed (`docker rm -f`) and recreated. The container itself is run with `docker run -i --rm`, so it is also removed automatically when it exits.

### Image selection

The image is chosen in this order:

1. An explicit `image` key.
2. The `container` name used as an image.
3. A pre-built image matched from `presets` (sorted), e.g. `[:base, :web]` → `szc-agent-web:latest`. Unknown combinations fall back to `szc-agent-base:latest`.
4. The default `szc-agent-base:latest`.

If the chosen image is not present locally, the backend attempts to build it with `nix build .#agentContainer-<preset>` (where `<preset>` is derived from `presets`, defaulting to `full` for unrecognized combinations) and then `docker load -i result`. If the build fails the failure is logged and the backend proceeds with the originally selected image name — so make sure your preset images either build or already exist locally.

### Docker options

| Config key | Purpose |
|------------|---------|
| `container` | Explicit container name; also used as an image candidate |
| `image` | Explicit image to run |
| `presets` | NixOS tool presets used to pick/build the image |
| `workspace` | Host path mounted at `/workspace` (default `/tmp/szc-workspace`) |
| `volumes` | Extra mounts as `[{host_path, container_path}]` |
| `network` | Docker network to attach (`--network`) |
| `memory_limit` | Memory cap (`--memory`) |
| `memory_swap` | RAM+swap cap (`--memory-swap`); set equal to `memory_limit` for a true hard RAM ceiling (without it `--memory` allows ~2x via swap) |
| `cpu_limit` | CPU cap (`--cpus`) |
| `pids_limit` | Max process count (`--pids-limit`); bounds fork-bombs / runaway spawns |
| `env` | Extra env vars (a map); `${VAR}` / `$VAR` are expanded from the host. Empty/`nil` values are dropped |
| `cmd` | Override the in-container command |
| `api_key` / `model` / `endpoint` | LLM settings (fall back to env) |

The skills directory, if set, is mounted read-only at `/skills`, and a sibling `logs/` directory is mounted at `/root/.subzeroclaw/logs`. The workspace is mounted at `/workspace` (unless your own `volumes` already mount something under `/workspace`), the host `/tmp` is shared, and the subzeroclaw source directory is mounted read-only at `/src/subzeroclaw` for in-container compilation. Agent name and LLM settings are passed as `-e` env vars, and topology connections are exported as `SWARM_TOPOLOGY` so `swarm-msg list` works inside the container.

Requirements: Docker, and Nix if you want images built on demand. For details on NixOS containers, presets, and how the images are assembled, see [containers.md](containers.md).

## Apple container

The Apple container backend (`lib/genswarms/backends/apple_container_backend.ex`) runs each agent with Apple's `container` CLI. It is for macOS / Apple silicon hosts that want OCI-style agent containers without Docker Desktop.

```elixir
%{
  name: :coder,
  backend: {:apple_container, "szc-agent-code:latest"},
  presets: [:base, :code],
  skills: ["code.md"]
}
```

Options are passed as the third tuple element:

```elixir
%{
  name: :coder,
  backend: {:apple_container, "szc-agent-code:latest", %{
    memory_limit: "2g",
    cpu_limit: 2,
    workspace: "/tmp/genswarms/coder"
  }},
  skills: ["code.md"]
}
```

Use `:apple_container` when you want the backend to pick an image from presets/defaults. Do not use bare `:container`; that name is intentionally not accepted because it is ambiguous.

### Service and image selection

Apple's tool requires its API server to be running before agents start:

```bash
container system start
container system status --format json
```

If `container system status --format json` does not report a running service, GenSwarms fails the agent start with `:apple_container_not_ready`.

The image is chosen in the same order as Docker: explicit `image`, then `container_name`, then a preset-derived image such as `szc-agent-code:latest`, then `szc-agent-base:latest`. If the selected image is not present, the backend attempts `nix build .#agentContainer-<preset> -o result` and then asks Apple `container` to load the result. Current Nix `agentContainer-*` outputs are Docker archives, while Apple `container image load` expects an OCI archive, so operators should pre-load a compatible image by converting the Nix result to OCI or by pulling from a registry. If Nix is unavailable or the build/load fails, the agent still starts with the selected image name and the `container` CLI reports the final image error.

### Apple container options

| Config key | Purpose |
|------------|---------|
| `container_name` | Explicit container name; default `szc-{swarm}-{agent}` |
| `image` | Explicit image to run |
| `presets` | NixOS tool presets used to pick/build the image |
| `workspace` | Host path mounted at `/workspace` (default `/tmp/szc-workspace`) |
| `volumes` | Extra mounts as `[{host_path, container_path}]` |
| `network` | Apple container network to attach (`--network`); `:isolated` / `"isolated"` is rejected |
| `memory_limit` | Memory cap (`--memory`) |
| `cpu_limit` | CPU cap (`--cpus`) |
| `env` | Extra env vars (a map); values are passed as discrete argv entries |
| `cmd` | Override the in-container command. A string runs through `sh -c` inside the container; a list is used as argv |
| `api_key` / `model` / `endpoint` | LLM settings (fall back to env) |

The runtime contract matches Docker where Apple's CLI supports it: skills are mounted read-only at `/skills`, logs at `/root/.subzeroclaw/logs`, the workspace at `/workspace`, host `/tmp` is shared, and the subzeroclaw source directory is mounted read-only at `/src/subzeroclaw` when it can be found. Agent name, LLM request routing, extra request/compaction settings, and topology are passed as environment variables. Container commands are assembled as argv lists for the host `container` process, not shell-built host commands.

`network: :isolated` is not implemented for Apple `container` because the current command set does not expose the Docker/bwrap-style egress-forwarding primitive GenSwarms uses. The backend fails closed with `{:unsupported_network, :isolated}` instead of silently running with open network. Use Docker or bwrap for isolated untrusted-content agents.

Apple's CLI also does not currently expose Docker-style pause/unpause semantics, so GenSwarms pause/resume remains Docker-only.

Requirements: macOS on Apple silicon, Apple's `container` CLI, the `container-apiserver` service running, and Nix if you want images built on demand.

## SSH

The SSH backend (`lib/genswarms/backends/ssh_backend.ex`) runs subzeroclaw on a remote machine over an SSH connection. It targets bare-metal NixOS hosts that have been provisioned (via Colmena) with the agent module — tools installed, skills directory at `/var/lib/subzeroclaw/skills`, and a `subzeroclaw` user set up — but also works on plain hosts.

```elixir
%{
  name: :researcher,
  backend: {:ssh, "agent@192.168.1.51", %{
    key_path: "~/.ssh/id_ed25519",
    nixos: true
  }},
  presets: [:base, :web],
  skills: ["web.md"]
}
```

### SSH options

| Config key | Purpose | Default |
|------------|---------|---------|
| `host` | `user@host` (taken from the tuple) | required |
| `port` | SSH port | `22` |
| `key_path` | Private key path | keys in `~/.ssh` |
| `password` | Password auth (added alongside any key) | none |
| `nixos` | Treat host as a provisioned NixOS machine | `true` |
| `remote_skills_dir` | Where skills are deployed | `/var/lib/subzeroclaw/skills` (NixOS) or `~/.subzeroclaw/skills` |
| `remote_user` | User to run the agent as (NixOS only) | `subzeroclaw` |
| `subzeroclaw_path` | Remote binary path | `subzeroclaw` |
| `api_key` / `model` / `endpoint` | LLM settings (fall back to env) | — |

Authentication: if `key_path` points to an existing file, its directory is used as the SSH `user_dir`; otherwise the backend falls back to `~/.ssh`. A `password`, if given, is added in addition. Host keys are accepted automatically (`silently_accept_hosts: true`, `user_interaction: false`), so this backend trusts whatever host it connects to — pin keys yourself if that matters.

When `nixos: true`, the agent is launched as the `remote_user` (`subzeroclaw` by default) via `sudo -u <user> env … subzeroclaw`. On non-NixOS hosts set `nixos: false`; the agent then runs as the SSH login user (the `remote_user` key is ignored), and you must install subzeroclaw and its tools yourself. If a local `skills_dir` is set, its files are copied to the remote skills directory over SFTP at start time (and again on each `deploy_skills` call). The agent is started with `SUBZEROCLAW_AGENT_NAME`, `SUBZEROCLAW_SKILLS`, and the LLM env vars set on the remote command line.

Requirements: SSH access to the host; on non-NixOS hosts, subzeroclaw and tools installed yourself.

## Bwrap

The bubblewrap backend (`lib/genswarms/backends/bwrap_backend.ex`) sandboxes each agent with Linux user namespaces instead of a full container. It is built for scale — roughly 500KB RAM and ~50ms startup per agent — which is what makes 10k+ agents on a single NixOS machine practical, with no external daemon.

```elixir
# Defaults
%{
  name: :researcher,
  backend: :bwrap,
  skills: ["web.md"]
}

# With options
%{
  name: :coder,
  backend: {:bwrap, %{memory_limit: "256M", presets: [:base, :code]}},
  skills: ["code.md"]
}
```

### Backend keys

Bwrap config separates backend keys (which control the sandbox) from domain keys (your application logic). The backend reads:

| Config key | Purpose | Default |
|------------|---------|---------|
| `workspace` | Host dir bound at `/workspace` | `/tmp/szc-workspace/{sandbox_id}` |
| `extra_path` | Extra dirs prepended to `PATH` inside the sandbox | `[]` |
| `extra_ro_binds` | Read-only mounts as `[{host_path, container_path}]` | `[]` |
| `extra_env` | Extra environment variables (a map) injected into the sandbox | `%{}` |
| `memory_limit` | cgroup memory cap | `"256M"` |
| `cpu_shares` | cgroup CPU shares | `100` |
| `tasks_max` | Max tasks/processes in the cgroup (`:cgroup` mode only) | `50` |
| `privilege_mode` | `:cgroup` (systemd scopes, kernel-hard limits) or `:rootless` (zero elevated capabilities - see below) | `:cgroup` |
| `nice` | CPU niceness for the sandbox in `:rootless` mode | `19` |
| `subzeroclaw_path` | Explicit binary path | resolved (see below) |
| `presets` | Sandbox base layers to overlay | `[:base]` |
| `network` | Set `:isolated` to run with no network except a forwarder pinned to the LLM endpoint (untrusted-content agents) | open network |
| `seccomp` | Apply a cBPF syscall-filter profile (deny mount, ptrace, module load, reboot, …); also enabled by `GENSWARMS_BWRAP_SECCOMP=1`. **Fails closed** — if enabled but the wrapper can't apply it, the agent aborts rather than running unfiltered | `false` |
| `store` | Nix-store bind mode: `:full` binds the whole `/nix/store`; `:closure` binds only the paths the sandbox base + `subzeroclaw` (+ `extra_store_paths`) need — tighter isolation | `:full` |
| `extra_store_paths` | Extra `/nix/store` paths to bind when `store: :closure` (whitelist additional packages) | `[]` |
| `max_turns` | Per-turn step budget passed to `subzeroclaw` (caps the tool-call loop per turn) | `subzeroclaw`'s own default |
| `request_extra` / `compact_extra` | Routing / compaction JSON forwarded to `subzeroclaw` (see the LLM-settings note above) | — |

`sandbox_id` is `{swarm}-{agent}-{timestamp_ms}`. Resource limits are enforced by wrapping the bwrap command in a `systemd-run` cgroup scope (`:cgroup` mode, the default) or in a plain-POSIX rlimit/nice launcher (`:rootless` mode - see "Privilege modes" below). Inside the sandbox, the overlay's merged directory is bound as `/`, the skills directory is bind-mounted read-only at `/root/.subzeroclaw/skills`, a sibling `logs/` directory is bound writable at `/root/.subzeroclaw/logs`, the workspace is bound at `/workspace`, and the Nix store is mounted read-only so binaries resolve. `extra_ro_binds` entries are only mounted if the host path exists. The sandbox runs with `--unshare-{user,pid,uts,ipc}` as uid/gid 1000, with `PATH` defaulting to `/bin:/usr/local/bin` (your `extra_path` dirs are prepended).

> Note: `extra_rw_binds` is listed as a bwrap backend key in the project conventions (it is accepted in agent config without error), but the current backend implements only `extra_ro_binds` (read-only) for extra mounts — `extra_rw_binds` is silently ignored. Use `workspace` for the agent's writable area.

### Binary path resolution

The bwrap backend locates the `subzeroclaw` binary in this order (first existing regular file wins):

1. Explicit `subzeroclaw_path` in config, or the `:subzeroclaw_path` application env (used directly if the file exists).
2. `../subzeroclaw/subzeroclaw` relative to the current working directory (sibling checkout).
3. `../subzeroclaw/subzeroclaw` relative to the GenSwarms source dir (when GenSwarms is used as a dependency).
4. The `SUBZEROCLAW_PATH` environment variable.
5. The system `PATH` (via `which subzeroclaw`).

### Mock and recording inside the sandbox

If `mock_script` is set in config or `SUBZEROCLAW_MOCK_SCRIPT` is set in the environment, it is passed into the sandbox as `SUBZEROCLAW_MOCK_SCRIPT`, so bwrap agents can run without LLM calls. If the `SUBZEROCLAW_RECORD_SCRIPT` environment variable is set (any value), subzeroclaw records responses to `/workspace/.recorded_responses.json` inside the sandbox.

### Privilege modes

`privilege_mode` decides how much the HOST around the swarm must grant:

- **`:cgroup`** (default) - every sandbox is a `systemd-run --user` scope with kernel-hard limits (`MemoryMax` OOM-kills a runaway tree, `CPUWeight`, `TasksMax`) and per-agent cgroup telemetry (`systemd-cgtop`, `memory.current`). The price: the host (or the container the swarm runs in) needs systemd as PID 1 and `SYS_ADMIN` for delegated cgroups. Choose it on a DEDICATED box where you own the blast radius and want the hard guarantees.

- **`:rootless`** - **zero elevated capabilities**. The systemd scope is replaced by a small launcher applying `RLIMIT_AS` (from `memory_limit`) and `nice`; tree cleanup rides the sandbox's PID namespace + `--die-with-parent`. The Nix base's small directory/symlink forest is materialized into a private per-agent root and bound as `/` - **no fuse-overlayfs process, `/dev/fuse`, or nested kernel overlay mount**. This works on managed container backing filesystems that permit unprivileged user namespaces but reject overlayfs-in-userns. Choose it on SHARED or managed infrastructure (Kubernetes) where the pod/container is the hard security boundary and bwrap is defence-in-depth inside it; the pod then runs fully unprivileged and only needs permission to create user namespaces (a seccomp profile allowing `clone(CLONE_NEWUSER)`, or `hostUsers: false`).

  The honest trade-offs: memory is a per-process address-space cap (allocations fail) rather than a cgroup OOM kill; `tasks_max` is NOT enforced per agent (RLIMIT_NPROC counts per real UID, which all agents share - bound the aggregate at the pod level instead); no per-agent cgroup telemetry.

```elixir
%{
  name: :researcher,
  backend: {:bwrap, %{privilege_mode: :rootless, memory_limit: "32M", network: :isolated}},
  skills: ["web.md"]
}
```

Requirements: bubblewrap with unprivileged user namespaces enabled (`kernel.unprivileged_userns_clone = 1`, or the equivalent seccomp/`hostUsers` arrangement on Kubernetes), `/run/swarm` available (override the agents dir with the `:bwrap_agents_dir` app env), and pre-built sandbox base layers (`nix build .#sandboxBase-*`). `:cgroup` mode additionally requires systemd and fuse-overlayfs. Base layers are resolved from `/run/swarm/sandbox-base/<preset-name>` (plus any dirs in the `:extra_preset_dirs` app env), falling back to `base` when a preset is missing. For the NixOS setup, preset/base-layer internals, and overlay/cgroup details, see [containers.md](containers.md).

## Mock

The mock backend (`lib/genswarms/backends/mock_backend.ex`) spawns no external process at all. It is a stub: it accepts input (returning `:ok` and discarding it) and produces no output. Use it to exercise swarm orchestration — topology, routing, dynamic add/remove/scale — without any agent runtime or LLM cost.

```elixir
%{name: :worker, backend: :mock}
```

It also accepts an optional `script` (`{:mock, %{script: [...]}}`), but the backend only stores that script on its ref for introspection — it does **not** match against it or generate responses (`send_input/2` and `handle_output/2` are no-ops). The bare `:mock` form is what the test suite and examples use.

> Producing canned LLM responses (with a `match`/`response` script) is a feature of **subzeroclaw**, not of the `:mock` backend. To run *real* agents (local/docker/apple_container/bwrap) without calling an LLM, point them at a subzeroclaw mock script via the `SUBZEROCLAW_MOCK_SCRIPT` environment variable, or use `mix genswarms.test --mock script.json`. See [testing.md](testing.md).

## See also

- [configuration.md](configuration.md) — the swarm config DSL and how `backend:` fits in
- [containers.md](containers.md) — NixOS containers, tool presets, and bwrap base-layer internals
- [testing.md](testing.md) — using the mock backend with `mix genswarms.test`
- [troubleshooting.md](troubleshooting.md) — diagnosing backend startup and connection failures

---

---
description: Build NixOS agent container images for GenSwarms, configure tool presets, and understand the bwrap sandbox internals.
---

# Containers and sandboxes

GenSwarms runs agents inside isolated execution environments built with Nix. Two
families of environment share the same tool presets: NixOS-based Docker images
(for the `{:docker, "name"}` backend) and Bubblewrap sandboxes (for the `:bwrap`
backend, designed for 10k+ agents on a single host). This page covers the build
targets, the preset catalogue, and the internals that assemble each environment.

Everything here is reproducible: images and sandbox bases are pinned by Nix
flake inputs (`nixpkgs` nixos-24.11), so the same tools resolve identically
across machines.

## Build targets

Container images are exposed as flake packages in `flake.nix`. Build one, then
load the result tarball into Docker. Each image is named `szc-agent-<name>:latest`.

```bash
nix build .#agentContainer-code
docker load < result
docker run --rm szc-agent-code:latest swarm-msg list
```

| Build target | Presets included | Use case |
|--------------|------------------|----------|
| `agentContainer-base` | `base` | Minimal agent with core utilities |
| `agentContainer-web` | `base`, `web` | Web research, HTTP APIs |
| `agentContainer-code` | `base`, `code` | Software development |
| `agentContainer-data` | `base`, `data` | Data processing, CSV/JSON |
| `agentContainer-full` | `base`, `web`, `code`, `data`, `python`, `node` | Full-featured agent |
| `agentContainer-python` | `base`, `python`, `data` | Python development |
| `agentContainer-node` | `base`, `node`, `web` | Node.js development |
| `agentContainer-devops` | `base`, `code`, `containers`, `cloud` | DevOps / cloud operations |

The preset-to-image mapping is defined in `nix/container.nix` (the `images`
attribute set) and wired to flake packages in `flake.nix`.

Each image also bundles, regardless of preset: `bashInteractive`, `coreutils`,
`cacert` (SSL certificates), `gnumake` and `gcc` for the startup
`subzeroclaw` build, the Nix package manager (so agents can run `nix-shell -p
...` at runtime), the `szc-wrapper` protocol script, and the `swarm-msg`
messaging CLI. Working directory is `/workspace`; `/workspace`, `/skills`, and
`/tmp` are declared as volumes. The image also sets
`SSL_CERT_FILE`/`NIX_SSL_CERT_FILE`, `NIX_PATH`, and `TMPDIR` so HTTPS and
runtime `nix-shell` both work out of the box (see the `config.Env` block in
`nix/container.nix`).

### Building on demand

You don't have to pre-build images. When a Docker agent starts, the backend
(`lib/genswarms/backends/docker_backend.ex`) maps the agent's `presets` to a
pre-built image name and, if the image is missing locally, runs
`nix build .#agentContainer-<name>` and `docker load` automatically before
launching. Only the eight preset combinations in the table above have a direct
mapping; any other combination falls back to the `full` image at build time and
the `szc-agent-base:latest` image at run time.

### Custom images

Call `mkAgentContainer` from your own flake to add domain packages. The builder
lives in `nix/container.nix` and is re-exported via
`genswarms.lib.<system>.mkAgentContainer`.

```nix
{
  inputs.genswarms.url = "github:genlayer/genswarms";

  outputs = { self, nixpkgs, genswarms, ... }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
    in {
      packages.x86_64-linux.my-agent =
        genswarms.lib.x86_64-linux.mkAgentContainer {
          name = "my-agent";
          presets = [ "base" "code" "python" ];
          tools = [ "ripgrep" "fd" "jq" ];      # names from the tools map
          extraPackages = with pkgs; [ postgresql redis ];
        };
    };
}
```

`mkAgentContainer` accepts `name`, `presets` (default `[ "base" ]`), `tools`
(individual names resolved against the tools map, then nixpkgs), `extraPackages`
(direct nixpkgs derivations), and `subzeroclawBinary` (optional path to a
subzeroclaw binary to bake in). Build with
`nix build .#my-agent && docker load < result`. The resulting image is named
`szc-agent-<name>:latest`.

### Orchestrator release

The agent container targets above are for *agents*. The Phoenix orchestrator
itself is packaged as a Nix mix release (not a Docker image — there is no
`Dockerfile` in the repo):

```bash
nix build .#orchestrator    # builds a prod mix release of the orchestrator
```

There is also a `genswarms-cli` package that builds the CLI escript as a
standalone derivation. For day-to-day use you usually run the orchestrator
directly from the dev shell (`genswarms up` / `mix phx.server`) rather than from
a built release — see [getting-started.md](getting-started.md).

## Tool presets

Presets are named groups of packages defined in `nix/tool-presets.nix`. They are
the single source of truth shared by Docker images, bwrap sandboxes, and the
NixOS agent module. Agents reference presets by name in their config.

| Preset | Tools |
|--------|-------|
| `base` | coreutils, bash, gnugrep, gnused, gawk, findutils, which, less, file, curl, jq |
| `web` | curl, wget, httpie, jq, yq, htmlq, w3m, lynx |
| `code` | git, git-lfs, gnumake, gcc, ripgrep, fd, tree, diff-so-fancy, delta, bat, tokei |
| `python` | python312, pip, virtualenv, requests, beautifulsoup4, pandas, numpy |
| `node` | nodejs_20, npm, yarn, pnpm |
| `data` | jq, yq, csvkit, miller, sqlite, duckdb, xsv |

`base` is the safe default and is included by every pre-built image. `curl` and
`jq` are in `base` because subzeroclaw needs `curl` for API calls and the
`szc-wrapper` needs `jq` for JSON protocol translation.

Additional presets also exist in `nix/tool-presets.nix` for specialized agents:
`docs` (pandoc, texlive scheme-small, poppler_utils, ghostscript, imagemagick),
`network` (curl, wget, httpie, netcat, socat, openssh, rsync, aria2), `system`
(htop, btop, lsof, strace, procps, psmisc, pciutils, usbutils), `security`
(openssl, gnupg, age, sops, pass), `containers` (docker-client, podman, skopeo,
dive), `cloud` (awscli2, google-cloud-sdk, azure-cli, kubectl, k9s, terraform),
and `ai` (openai, anthropic, tiktoken Python packages). These are not bundled
into any pre-built image except where the table above lists them; reach them via
a custom image or a custom sandbox base.

### Individual tools

For fine-grained control, `nix/tool-presets.nix` also exposes a `tools` map that
aliases friendly names to packages (for example `rg` and `ripgrep` both resolve
to ripgrep, `python3` to python312, `gh` to the GitHub CLI). Names listed in an
agent's `tools` are looked up in this map first, then fall back to a direct
nixpkgs attribute.

### Using presets in agent config

Reference presets and tools directly in the agent config. For Docker agents the
preset selection is baked into the image you build; for bwrap agents presets are
resolved at deploy time against pre-built sandbox bases (see below).

```elixir
%{
  name: :coder,
  backend: {:docker, "code"},
  presets: [:base, :code],
  skills: ["code.md"]
}
```

See [configuration.md](configuration.md) for the full agent schema and how
`presets`/`tools` are applied, and [backends.md](backends.md) for backend tuple
forms and options.

## Multi-swarm namespacing

Docker containers are namespaced by swarm name: each agent runs as a container
named `szc-{swarm}-{agent}` (set in `lib/genswarms/backends/docker_backend.ex`,
overridable per agent with the `container` config key). This lets multiple
swarms run on one host without collision.

Pause and resume operate per swarm by acting on that swarm's containers only:

```bash
docker pause szc-{swarm}-{agent}
docker unpause szc-{swarm}-{agent}
```

The orchestrator issues these for every agent in the named swarm, so pausing one
swarm never freezes another.

## Bwrap sandbox internals

The bwrap backend trades container isolation for far lower overhead, targeting
10k+ agents on a single NixOS machine. It reuses the exact same tool presets as
the Docker images but assembles them as overlay filesystems rather than images.

### Sandbox bases

`nix/bwrap-sandbox.nix` builds a read-only Nix environment per preset
combination using `pkgs.buildEnv`. Each base contains the resolved preset
packages plus the same core set as containers (`bashInteractive`, `coreutils`,
`cacert`, `nix`, the `szc-wrapper` script, and `swarm-msg`), linking `/bin`,
`/lib`, `/share`, and `/etc`. Build a base with:

```bash
nix build .#sandboxBase-code
```

The sandbox bases actually defined in `nix/bwrap-sandbox.nix` (and therefore
buildable) are:

| Flake target | Presets |
|--------------|---------|
| `sandboxBase-base` | `base` |
| `sandboxBase-web` | `base`, `web` |
| `sandboxBase-code` | `base`, `code` |
| `sandboxBase-data` | `base`, `data` |
| `sandboxBase-python` | `base`, `python`, `data` |
| `sandboxBase-node` | `base`, `node`, `web` |
| `sandboxBase-full` | `base`, `web`, `code`, `data`, `python`, `node` |
| `sandboxBase-devops` | `base`, `code`, `containers`, `cloud` |

> **Note:** `flake.nix` also declares `sandboxBase-web-code`,
> `sandboxBase-code-python`, and `sandboxBase-data-python`, but no matching
> `sandboxLib.web-code` / `code-python` / `data-python` attributes exist in
> `nix/bwrap-sandbox.nix`, so those three targets fail to evaluate. To get a
> mixed-preset base (for example `code` + `python`), build a custom base with
> `mkSandboxBase` (see *Preset resolution and custom presets* below) rather than
> relying on those declarations.

At runtime the bases are resolved by *directory name* under
`/run/swarm/sandbox-base/`, not by flake target name. The
`services.subzeroclaw-bwrap` NixOS module (`nix/bwrap-module.nix`) symlinks the
bases listed in its `sandboxPresets` option into that directory; the directory
name for a multi-preset agent is the sorted, `-`-joined preset list (see below).

### Overlay assembly

Per-agent isolation comes from a private writable root. In `:cgroup` mode it is
`fuse-overlayfs` (userspace overlay, no root required). For each agent,
`lib/genswarms/backends/bwrap/overlay_manager.ex` creates a directory tree and
mounts the union. In `:rootless` mode nothing is mounted: the small Nix base
directory/symlink forest is materialized into the per-agent `merged/`
directory, then bwrap binds that directory as `/`. This avoids both
fuse-overlayfs and nested kernel overlayfs, which managed container backing
filesystems may reject. See "Privilege modes" in [backends.md](backends.md).
Named Nix bases are cheap to materialize because their tools remain symlinks
into the read-only `/nix/store`; regular files in a custom base are copied per
agent. The directory layout is:

```
/run/swarm/
  sandbox-base/<preset>   # symlink to the pre-built Nix environment (lowerdir)
  agents/<sandbox-id>/
    upper/                # COW layer (:cgroup) or seed staging (:rootless)
    work/                 # overlayfs workdir (:cgroup only)
    merged/               # FUSE union or materialized root the agent runs in
```

```bash
fuse-overlayfs -o lowerdir=<base>,upperdir=<upper>,workdir=<work> <merged>
```

The shared sandbox base is the read-only lower layer; each agent gets a private
writable upper layer, so thousands of agents share one copy of the tools. Before
the agent starts, `/etc/resolv.conf` and `/etc/hosts` are copied into the upper
layer's `etc/` so DNS and hostname resolution work (the base `/etc` is
read-only). The agent then runs inside `merged/` via Bubblewrap, with the
`szc-wrapper` script bind-mounted at `/usr/local/bin/szc-wrapper` and the
subzeroclaw binary at `/usr/local/bin/subzeroclaw`. On shutdown the overlay is
unmounted (`fusermount -u`) and the per-agent directory tree is removed
(`File.rm_rf`).

### Resource isolation

Each agent is placed in its own systemd cgroup so a runaway agent can't starve
its neighbors. `lib/genswarms/backends/bwrap/cgroup_manager.ex` wraps the bwrap
command in a transient `systemd-run --user` unit under the `subzeroclaw.slice`,
named `szc-<sandbox-id>`. The backend config keys map to systemd properties:

| Config key | Default | systemd property |
|------------|---------|------------------|
| `memory_limit` | `"256M"` | `MemoryMax` |
| `cpu_shares` | `100` | `CPUWeight` |
| `tasks_max` | `50` | `TasksMax` |

Because every scope lives under one slice, you can monitor aggregate usage with
`systemd-cgtop` or via `CgroupManager.get_aggregate_stats/0`, and stop a single
agent by terminating its scope. Per-agent memory, CPU, and task counts are read
directly from the cgroup filesystem (`memory.current`, `cpu.stat`,
`pids.current`).

### Preset resolution and custom presets

Presets in the agent config map to a base directory name by sorting the preset
atoms and joining with `-` (so `[:code, :base]` resolves to the `base-code`
base). Resolution searches `/run/swarm/sandbox-base` plus any directories
registered by a downstream project:

```elixir
Application.put_env(:genswarms, :extra_preset_dirs, ["/my/presets"])
```

If a named preset directory is not found in any search dir, resolution falls
back to the `base` layer (logging a warning). You can also point an agent at a
fully custom base layer directly with a `{:custom, "/path/to/base"}` entry in
its `presets` list — this path is expanded and used verbatim as the overlay
lowerdir, and `{:custom, _}` entries are excluded from the sorted directory-name
computation.

To build a domain-specific base, copy `nix/preset-template.nix` into your
project as `preset.nix`, set `name`, choose `presets`, add `extraPackages`,
build it, and symlink the result into a preset search directory:

```bash
nix-build preset.nix
ln -sf $(readlink result) ./presets/solidity
```

The template uses `sandboxLib.mkSandboxBase` (also re-exported as
`genswarms.lib.<system>.mkSandboxBase`), so a custom base is byte-for-byte
compatible with the built-in ones.

### Backend config keys

The bwrap config separates backend keys from domain keys. Backend keys
recognized by the sandbox: `workspace`, `presets`, `memory_limit` (default
`"256M"`), `cpu_shares` (default `100`), `tasks_max` (default `50`),
`extra_ro_binds` (`[{host_path, container_path}]`, mounted read-only and skipped
silently if the host path is missing), `extra_path` (directories prepended to
the in-sandbox PATH, ahead of `/bin:/usr/local/bin`), `extra_env`
(`%{KEY => value}` extra environment variables), and `subzeroclaw_path` (path to
the subzeroclaw binary).

```elixir
%{
  name: :worker,
  backend: :bwrap,
  config: %{
    workspace: "/tmp/my-workspace",
    presets: [:base, :code],
    memory_limit: "256M",
    extra_path: ["/opt/tools/bin"],
    extra_ro_binds: [{"/home/user/project", "/project"}]
  }
}
```

See [backends.md](backends.md) for the complete bwrap key reference, binary path
resolution, and host requirements (the `services.subzeroclaw-bwrap` NixOS module
in `nix/bwrap-module.nix` provisions kernel limits, the `/run/swarm` tmpfs, the
`subzeroclaw.slice`, and symlinks the sandbox bases).

## See also

- [backends.md](backends.md) — backend tuple forms, bwrap config keys, binary resolution
- [configuration.md](configuration.md) — agent schema, presets and tools in config
- [architecture.md](architecture.md) — how backends fit into the supervision tree

---

---
description: The GenSwarms REST API — create and control swarms, send tasks, manage agents and topology, and query events over JSON HTTP.
---

# REST API

GenSwarms exposes a pure JSON REST API served by Phoenix (no HTML/frontend is included). The same server also hosts a WebSocket endpoint for real-time streaming — see [websocket.md](websocket.md).

All routes are defined in `lib/genswarms_web/router.ex` and implemented by the controllers in `lib/genswarms_web/controllers/`. This page documents every route, derived directly from those sources.

## Base URL and conventions

- Base URL: `http://localhost:4000` (the port is set by the `PORT` env var, default `4000`).
- The API pipeline accepts `application/json` only. Send request bodies as JSON and set `Content-Type: application/json`.
- **Authentication:** when `GENSWARMS_API_TOKEN` is set, every request must send `Authorization: Bearer <token>`; when unset, only loopback callers are accepted. The CLI attaches the token automatically. See [Security](security.md).
- **Config-scoped token:** `GENSWARMS_CONFIG_API_TOKEN` is a deliberately narrow grant that authorizes ONLY `PATCH /api/swarms/:name/objects/:object/config` and `GET /api/swarms/:name/overlay` — never the rest of the API. Hand this one to config tooling (the dashboard configurator) so a host can enable hot config edits without exposing the full control plane (create/delete swarms, add agents, route messages). The full token also works on the config routes; the config token works nowhere else. With neither token set, the loopback-only rule applies everywhere.
- CORS is restricted to an allowlist (`GENSWARMS_CORS_ORIGINS`, default local dev origins) via Corsica — see [Security](security.md#cors).
- Successful responses return a JSON object. Errors return a JSON object with an `error` (string) or `errors`/`valid` field and an appropriate HTTP status (`400`, `404`, `500`).
- Most endpoints work for both in-process swarms and daemon swarms; the controller falls back to the SQLite registry and backend CLIs (Docker / Apple `container`) when a swarm runs in a separate OS process.

## API info

| Method | Path | Description |
|--------|------|-------------|
| GET | / | API metadata: name, version, endpoint index, and a WebSocket/route summary |

The root returns a static descriptor including `name`, `version` (`"1.0.0"`), `description`, an `endpoints` map, a `websocket` section, and a `documentation` map summarizing the main route groups. The `websocket` section advertises the channel URL (`/swarm`), the channel topic pattern (`swarm:{swarm_name}`), and the client→server / server→client event lists (see [websocket.md](websocket.md) for details).

## Swarm management

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/swarms | List all swarms |
| POST | /api/swarms | Create a swarm from an inline config or a config path |
| GET | /api/swarms/:name | Get detailed swarm status |
| DELETE | /api/swarms/:name | Stop a swarm (`?purge=true` to delete all data) |
| POST | /api/swarms/:name/pause | Pause (freeze) all Docker containers for the swarm |
| POST | /api/swarms/:name/resume | Resume paused Docker containers for the swarm |
| POST | /api/swarms/:name/restart | Restart the swarm (`?delete=true` for a clean slate) |
| POST | /api/swarms/:name/message | Route a message between two agents |
| POST | /api/swarms/clean | Remove stopped/crashed swarms (`?all=true` also clears all events) |

Notes:

- `GET /api/swarms` returns `{"swarms": [ ... ]}`.
- `POST /api/swarms` accepts either `{"config": { ... }}` (an inline swarm config object) or `{"config_path": "path/to/config.exs"}`. On success it returns `201 Created` with `{"status": "created", "swarm_name": "..."}`. On a config/start error it returns `400` with `{"error": "..."}`. Missing both fields returns `400` with `{"error": "Missing 'config' or 'config_path' parameter"}`.
- `GET /api/swarms/:name` enriches the status with `topology`, per-agent `backend_type`, `skills_paths`, `container_name`, `container_status`, per-object `handler_module`/`source_file`, and a `file_paths` map (`config`, `data_dir` = `~/.subzeroclaw/swarms/<name>`, `log` = `.genswarms/logs/<name>.log`). Returns `404` with `{"error": "Swarm not found"}` if the swarm is unknown.
- `DELETE /api/swarms/:name` returns `{"status": "stopped"|"purged", "swarm_name": "...", "config_path": ...}`. With `?purge=true` it also deletes swarm files and registry rows. Returns `404` for an unknown swarm.
- `POST .../pause` and `.../resume` return a count: `{"status": "paused", "swarm_name": "...", "containers_paused": N}` / `{"status": "resumed", "swarm_name": "...", "containers_resumed": N}`. Pause/resume uses Docker `pause`/`unpause`; Apple `container` does not expose equivalent semantics, so Apple container agents are not paused by these endpoints. A `404` is returned for an unknown swarm; a backend failure returns `500`.
- `POST .../restart` reads the saved config path from the registry; `?delete=true` deletes data before restarting. Returns `{"status": "restarted", "swarm_name": "...", "delete_data": bool}`. Returns `404` if the swarm (or its config path) is unknown.
- `POST /api/swarms/:name/message` requires `{"from": "...", "to": "...", "content": "..."}` and returns `{"status": "routed", "from", "to", "swarm"}`. Missing fields return `400` with `{"error": "Missing 'from', 'to', or 'content' parameter"}`.
- `POST /api/swarms/clean` returns `{"status": "cleaned", "swarms_removed": N, "events_cleared": bool}`.

## Agent operations

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/swarms/:name/agents | List agents and their status |
| GET | /api/swarms/:name/agents/:agent | Get a single agent's status |
| POST | /api/swarms/:name/agents/:agent/task | Send a task to an agent |
| POST | /api/swarms/:name/agents/:agent/restart | Restart an agent |
| POST | /api/swarms/:name/agents/:agent/interrupt | Interrupt the active backend turn without deleting its session |
| GET | /api/swarms/:name/agents/:agent/session | Get attachable persistent-session metadata |
| GET | /api/swarms/:name/agents/:agent/history | Get the agent's message history (`?limit=`, default 100) |
| GET | /api/swarms/:name/agents/:agent/logs | Get the agent's conversation logs |
| GET | /api/swarms/:name/agents/:agent/skills | Get the agent's skill contents |
| PUT | /api/swarms/:name/agents/:agent/skills/:skill | Update one of the agent's skill files |

Notes:

- `POST .../task` requires `{"task": "..."}` and returns `{"status": "sent", "agent", "task"}`. For daemon swarms the task is queued in SQLite for the daemon to pick up. Missing `task` returns `400` with `{"error": "Missing 'task' parameter"}`.
- `GET .../agents` returns `{"agents": [ ... ]}`; `GET .../agents/:agent` returns the status map directly, or `404` with `{"error": "Agent not found"}`.
- `POST .../agents/:agent/restart` reuses the complete effective agent config and preserves a persistent tmux pane. It returns `{"status": "restarted", "agent": "..."}`, `404` if the swarm/agent is unknown, or `500` on failure.
- `POST .../agents/:agent/interrupt` returns `{"status":"interrupted","agent":"..."}` or `409` when the backend cannot interrupt the current state.
- `GET .../agents/:agent/session` returns non-secret tmux metadata, read-only/read-write attach argv, and a nested `runner` object (`kind`, isolation booleans, network, and non-secret container/sandbox identity). A non-persistent backend returns `422`; an unknown agent returns `404`.
- `GET .../history` and `GET .../logs` return `{"history": [...]}` / `{"logs": [...]}`. `GET .../skills` returns `{"skills": ...}`. A missing agent returns `404`.
- `PUT .../skills/:skill` requires `{"content": "..."}` and returns `{"status": "updated", "skill": "..."}`. A failure returns `500`.

> Adding and removing agents at runtime uses `POST /api/swarms/:name/agents` and `DELETE /api/swarms/:name/agents/:agent` — see [Dynamic topology and scaling](#dynamic-topology-and-scaling) below.

## Messages

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/swarms/:name/messages | Get the swarm's recent inter-agent message log |

Notes:

- `GET /api/swarms/:name/messages` returns `{"messages": [ ... ]}` from the router's message log. Accepts `?limit=` (default 100). This is the routed-message history (who sent what to whom); for structured observability events use the [Events](#events) endpoints instead.

## Dynamic topology and scaling

These endpoints mutate a running swarm and persist the change as an overlay (see [Overlay and snapshot](#overlay-and-snapshot)).

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/swarms/:name/topology | Get the swarm's topology (adjacency list) |
| PATCH | /api/swarms/:name/topology | Add and/or remove topology edges |
| POST | /api/swarms/:name/agents | Add a new agent to a running swarm |
| DELETE | /api/swarms/:name/agents/:agent | Remove an agent from a running swarm |
| POST | /api/swarms/:name/agents/:base/scale | Scale an agent group to a target count |

Notes:

- `GET .../topology` returns `{"topology": [{"from": ..., "targets": [...]}, ...]}` or `404` for an unknown swarm.
- `PATCH .../topology` accepts `{"add": [...], "remove": [...]}`. Each edge may be `["from", "to"]` or `{"from": "...", "to": "..."}`; both endpoints of an edge must be strings, and any edge that doesn't parse is silently ignored. Returns `{"status": "ok", "added": N, "removed": M}` (the counts reflect the parsed edges actually applied). A mutation error returns `400`.
- `POST .../agents` accepts an agent spec (`name`, `backend`, `skills`, `model`, `endpoint`, `presets`, `config`) plus optional `connections` (outgoing targets) and `incoming` (sources). `backend` may be a string (e.g. `"local"`, `"apple_container"`, `"mock"`) or an object: `{"type": "docker", "image": "coder"}`, `{"type": "apple_container", "image": "szc-agent-code:latest"}`, `{"type": "apple_container", "image": "szc-agent-code:latest", "opts": { "memory_limit": "2g" }}`, `{"type": "ssh", "host": "user@host"}`, `{"type": "bwrap", "opts": { ... }}`, `{"type":"tmux","client":"codex","opts":{"runner":"docker","image":"coding-tuis:latest","client_source":"runtime"}}`, or `{"type": "mock"}`. Known backend keys in `config` are normalized before start; unknown config keys remain domain data. Apple container and interactive tmux runners reject `"network": "isolated"` and fail closed instead of running with open network; tmux Docker/bwrap runners accept `"none"` as a complete cutoff. Returns `201 Created` with `{"status": "added", "name": "..."}`, or `400` with `{"error": "..."}` on failure.
- `DELETE .../agents/:agent` returns `{"status": "removed", "name": "..."}` or `404` with `{"error": "..."}`.
- `POST .../agents/:base/scale` requires an integer `{"count": N}` (`count >= 0`) and returns `{"status": "ok", "result": {"added": [...], "removed": [...], "failed": [{"name", "reason"}]}}` (the `added`/`removed`/`failed[].name` values are strings). A missing/non-integer/negative `count` returns `400` with `{"error": "Missing or invalid 'count'"}`; a scaling error returns `400` with `{"error": "..."}`.

## Objects

Objects are the non-agentic components of a swarm. See [objects.md](objects.md).

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/swarms/:name/objects | List objects with their lifecycle state |
| POST | /api/swarms/:name/objects | Add an object to a running swarm |
| GET | /api/swarms/:name/objects/:object | Get an object's live read-only state |
| PATCH | /api/swarms/:name/objects/:object/config | Update a running object's config (schema-gated) |
| DELETE | /api/swarms/:name/objects/:object | Remove an object from a running swarm |

Notes:

- `GET .../objects` returns `{"objects": [{"name", "state", "handler"}, ...]}`, where `name` is a string and `handler` is the inspected handler module (or `null`).
- `POST .../objects` accepts an object spec (`name`, `handler` module name, `backend`, `config`) plus optional `connections`/`incoming`. Returns `201 Created` with `{"status": "added", "name": "..."}`, or `400` with `{"error": "..."}` on failure.
- `GET .../objects/:object` returns `{"object": "...", "state": <handler domain state>}`. The framework imposes no schema on the state. An unknown object returns `404` with `{"error": "Object not found"}`.
- `PATCH .../objects/:object/config` requires `{"config": { ... }}` — a partial, string-keyed patch. It is gated by the package's `config_schema` (the `swarm-object.json` next to the handler's source, gsp design §14.2.1), **fail-closed**: a handler without a schema returns `422 {"error": "no_config_schema"}`; any key that is not `x-mutable: true` returns `422` with `immutable_keys`; host-escape backend keys (`subzeroclaw_path`, `extra_*`) are always rejected. On success the object restarts with the merged config (topology edges preserved; a rejected `init/1` rolls back to the old config), the patch persists as an `:update_config` overlay event (replayed on boot), and the response is `{"status": "updated", "object": "...", "keys": [...]}`.
- `DELETE .../objects/:object` returns `{"status": "removed", "name": "..."}` or `404`.

## Overlay and snapshot

A swarm's effective config is its seed config combined with an overlay of runtime mutations (added/removed agents, topology edits, scaling).

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/swarms/:name/overlay | Show the recorded overlay events |
| DELETE | /api/swarms/:name/overlay | Clear the overlay |
| POST | /api/swarms/:name/snapshot | Return the effective config (seed ⊕ overlay) as Elixir source |

Notes:

- `GET .../overlay` returns `{"swarm": "...", "events": [{"op": ..., "payload": ...}, ...]}`.
- `DELETE .../overlay` returns `{"status": "cleared", "swarm": "..."}`.
- `POST .../snapshot` responds with `Content-Type: text/x-elixir` and `200`, with the effective config rendered as a `.exs` source body (not JSON; produced by `Genswarms.Config.ExsWriter`). Returns `404` with a JSON `{"error": "..."}` if the swarm is unknown.

## Events

Event queries read from the durable, cross-process `EventStore` (SQLite by default), so events from daemon swarms in other BEAM nodes are visible. See [observability.md](observability.md).

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/events | Query events with filters |
| GET | /api/swarms/:name/events | Query events for one swarm |
| GET | /api/swarms/:name/agents/:agent/events | Query events for one agent |

Shared query params (all optional):

| Param | Description |
|-------|-------------|
| level | `error`, `warning`, `info`, or `debug` |
| category | `backend`, `routing`, `agent`, `swarm`, or `system` (see note) |
| swarm | Swarm name (implied on the scoped routes) |
| agent | Agent name (implied on the agent route) |
| event_type | A specific event type |
| minutes | Only events from the last N minutes |
| limit | Max events to return (default 100) |

Each response includes `events` (a list) and `count`. `GET /api/events` also echoes the normalized filters as a `query` map; the scoped routes echo `swarm` (and `agent` on the agent route). Every event is `{id, timestamp, level, category, swarm, agent, event_type, message, metadata}`; timestamps are ISO-8601 strings.

> Note: `category` is resolved with `String.to_existing_atom/1`, so only category names that already exist as atoms in the running system are accepted. The controller's documented set is `backend, routing, agent, swarm, system`, but the broader observability taxonomy and the CLI also emit an `object` category — passing `category=object` works as long as that atom has been created (e.g. after any object event has been logged). See [observability.md](observability.md) for the full taxonomy.

## Skills

See [skills.md](skills.md).

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/skills | List available skill files (`?path=` overrides the search root) |
| GET | /api/skills/:name | Get a skill's content by name |

Notes:

- `GET /api/skills` searches the configured skills directory (the `:genswarms`/`:skills_dir` app env, default `priv/skills`, expanded to an absolute path) recursively and returns `{"skills": [{"name", "path", "relative_path", "category"}, ...], "base_path", "count"}`. `category` is the relative subdirectory (or `"default"` for files at the root). Pass `?path=` to search a different root.
- `GET /api/skills/:name` returns `{"name", "path", "content", "size"}` (`size` is the byte length), or `404` with `{"error": "Skill not found"}` if the skill is not found. The `.md` extension is optional in the name, and nested skills are matched by a recursive search.

## Config validation

See [configuration.md](configuration.md).

| Method | Path | Description |
|--------|------|-------------|
| POST | /api/config/validate | Validate a swarm config |

`POST /api/config/validate` accepts one of:

- `{"config": { ... }}` — an inline config object (must be a JSON object).
- `{"config_path": "path/to/config.exs"}` — a `.exs`, `.json`, or `.yaml` file path.
- `{"content": "...", "format": "exs"|"json"|"yaml"}` — a raw config string (`"yml"` is accepted as an alias for `yaml`; an unrecognized format falls back to `exs`).

On success it returns `{"valid": true, "config": <summary>}` (the `config_path` / `format` form also echoes that input field) where the summary includes `name`, `agent_count`, `object_count`, `topology_edges`, and per-agent (`name`, `backend`, `skills`, `model`), per-object (`name`, `handler`), and `topology` (`from`/`to`) details. On a validation failure it returns `400` with `{"valid": false, "errors": [...]}`. A non-existent `config_path` returns `404` with `{"valid": false, "errors": ["File not found: ..."]}`. Sending none of the accepted fields returns `400` with `{"error": "...", "usage": { ... }}`.

## Examples

List all swarms:

```bash
curl http://localhost:4000/api/swarms
```

Create a swarm from a config file on the server:

```bash
curl -X POST http://localhost:4000/api/swarms \
  -H "Content-Type: application/json" \
  -d '{"config_path": "examples/research.exs"}'
```

Send a task to an agent:

```bash
curl -X POST http://localhost:4000/api/swarms/my-swarm/agents/researcher/task \
  -H "Content-Type: application/json" \
  -d '{"task": "Summarize the latest results."}'
```

Add an agent to a running swarm and wire it into the topology:

```bash
curl -X POST http://localhost:4000/api/swarms/my-swarm/agents \
  -H "Content-Type: application/json" \
  -d '{"name": "reviewer", "backend": {"type": "docker", "image": "code"}, "incoming": ["coder"]}'
```

Add an Apple container agent:

```bash
curl -X POST http://localhost:4000/api/swarms/my-swarm/agents \
  -H "Content-Type: application/json" \
  -d '{"name": "mac_coder", "backend": {"type": "apple_container", "image": "szc-agent-code:latest", "opts": {"memory_limit": "2g"}}, "incoming": ["researcher"]}'
```

Snapshot the effective (seed ⊕ overlay) config as runnable Elixir:

```bash
curl -X POST http://localhost:4000/api/swarms/my-swarm/snapshot -o my-swarm.exs
```

## See also

- [cli.md](cli.md) — the `swarm` CLI, which wraps many of these endpoints.
- [websocket.md](websocket.md) — real-time streaming over the WebSocket channel.
- [programmatic.md](programmatic.md) — driving swarms directly from Elixir.
- [configuration.md](configuration.md) — the swarm config DSL used by create/validate.
- [observability.md](observability.md) — the event categories and levels surfaced by the Events endpoints.

---

---
description: The GenSwarms WebSocket API — subscribe to real-time agent output, message routing, and event streams.
---

# WebSocket API

GenSwarms exposes a Phoenix WebSocket alongside its [REST API](rest-api.md) for real-time, per-swarm communication: sending tasks, fetching status, and streaming logs and events as they happen.

The implementation lives in `lib/genswarms_web/channels/swarm_socket.ex` (the socket) and `lib/genswarms_web/channels/swarm_channel.ex` (the channel). This page is derived directly from those sources.

## Connection

- Socket mount path: `/swarm` (so the URL is typically `ws://localhost:4000/swarm/websocket`, port from `PORT`, default `4000`). Only the `websocket` transport is enabled; long polling is disabled (`longpoll: false` in `endpoint.ex`).
- Channel topic: `swarm:<swarm_name>` (the socket declares `channel "swarm:*"`).
- Authentication is enforced on connect with the same fail-closed policy as the REST API (`connect/3` calls `Genswarms.Auth.authorize/3`): when `GENSWARMS_API_TOKEN` is set, a matching Bearer token is required — supplied as a `?token=<token>` query param (browsers can't set WS headers) or an `Authorization: Bearer <token>` header; when no token is configured, only loopback clients may connect. A failed check rejects the connection (`:error`). The socket has no per-connection id (`id/1` returns `nil`).
- Joining a topic verifies the swarm exists, checking the in-process `SwarmManager` first and falling back to the SQLite registry (`SwarmRegistry`). If it exists in neither, the join is rejected with `{"reason": "swarm_not_found"}`. On success the join reply is `{"swarm": "<swarm_name>"}`.

On join the channel subscribes to the swarm's internal PubSub topics (`swarm:<name>`, `:output`, `:routing`, `:status`) so that output, routing, status, and lifecycle messages are pushed to the client automatically — no extra subscribe call is needed for those. The log and event streams, by contrast, are opt-in via the `subscribe_logs` / `subscribe_events` events below.

## Inbound events (client → server)

Each of these is sent with `channel.push(event, payload)` and returns a reply.

| Event | Payload | Reply |
|-------|---------|-------|
| `send_task` | `{"agent": "...", "task": "..."}` | `ok` → `{"status": "sent"}`. `error` → `{"reason": "<inspected error>"}`. |
| `get_status` | ignored | `ok` → the swarm status map. `error` → `{"reason": "<inspected error>"}`. |
| `subscribe_logs` | `{"agent": "..."}`, or `{}` for all agents | `ok` → `{"subscribed": true, "agent": <agent-or-null>, "recent_logs": [...]}` (last 50, oldest first). |
| `unsubscribe_logs` | `{"agent": "..."}` or `{}` (must match the agent used to subscribe) | `ok` → `{"unsubscribed": true, "agent": <agent-or-null>}`. |
| `subscribe_events` | `{"filters": {"level": ..., "category": ..., "event_type": ...}}` (any subset; `{}` or omitted = no filtering) | `ok` → `{"subscribed": true, "filters": {...}, "recent_events": [...]}` (last 50, oldest first). |
| `unsubscribe_events` | ignored | `ok` → `{"unsubscribed": true}`. Clears **all** event subscriptions on the socket. |

Notes on the inbound events:

- `send_task` and `get_status` delegate to `SwarmManager`; on failure the reason is the Elixir term rendered with `inspect/1` (e.g. `":not_found"`), so treat it as an opaque diagnostic string, not a stable machine-readable code.
- `subscribe_logs` is keyed by agent: subscribing with `{"agent": "researcher"}` and then `{}` registers two independent subscriptions. `unsubscribe_logs` removes only the subscription whose `agent` matches (use `{}` to remove the all-agents subscription).
- `subscribe_events` accumulates filter sets: calling it twice adds a second subscription, and a `log_event` is pushed as `event` if it matches **any** registered filter set. `unsubscribe_events` discards every event subscription at once (it does not take an `agent`/`filters` argument).
- `recent_logs` / `recent_events` are returned synchronously in the reply, sourced from the durable `EventStore` (SQLite-backed by default). Because that store is shared across BEAM nodes, history from daemon swarms running in other processes is visible. The live stream then arrives as `log_entry` / `event` pushes.

## Outbound pushes (server → client)

Subscribe to these with `channel.on(event, callback)`. The lifecycle and output pushes start flowing on join; `log_entry` and `event` require an active subscription.

| Event | Payload | When |
|-------|---------|------|
| `agent_output` | `{"agent": ..., "content": ...}` | Raw agent output. |
| `message_routed` | routing data map | A directed message was routed between components. |
| `message_broadcast` | broadcast data map | A broadcast message (`@all:`) was routed. |
| `agent_status` | `{"agent": ..., "state": ...}` | An agent changed state. |
| `swarm_started` | `{"status": "<string>"}` | The swarm started (status is stringified). |
| `swarm_stopped` | `{}` | The swarm stopped. |
| `agent_added` | `{"name": ..., "spec": {...}}` | An agent was added at runtime; `spec` is the serialized agent spec. |
| `agent_removed` | `{"name": ...}` | An agent was removed at runtime. |
| `topology_changed` | `{}` | The topology was modified at runtime. |
| `log_entry` | see below | A streamed log line, pushed only while a matching `subscribe_logs` subscription is active. |
| `event` | see below | A streamed event, pushed only while a matching `subscribe_events` subscription is active. |

### Filtering semantics

- `log_entry` is pushed only while a `subscribe_logs` subscription is active **and** the event's agent matches. A `subscribe_logs` with no `agent` (`{}`) matches every agent.
- `event` is pushed only while a `subscribe_events` subscription is active **and** the event matches the subscribed `filters`. Each of `level`, `category`, and `event_type` present in the filter must equal the event's corresponding field (compared as atoms). An empty filter set (`{}`) matches every event.

### Payload shapes

A `log_entry` payload contains:

```json
{
  "id": "...",
  "timestamp": "2026-06-05T12:00:00Z",
  "level": "info",
  "agent": "researcher",
  "event_type": "agent_output",
  "message": "...",
  "metadata": {}
}
```

An `event` payload carries the same fields **plus** `category` and `swarm`:

```json
{
  "id": "...",
  "timestamp": "2026-06-05T12:00:00Z",
  "level": "info",
  "category": "routing",
  "swarm": "my-swarm",
  "agent": "researcher",
  "event_type": "message_routed",
  "message": "...",
  "metadata": {}
}
```

`timestamp` is rendered as an ISO 8601 string. The same field shapes are used for the `recent_logs` / `recent_events` entries returned by the subscribe replies.

## JavaScript example

Using the Phoenix JS client (`phoenix` npm package). Note that the client appends `/websocket` to the socket URL automatically, so pass `ws://localhost:4000/swarm`:

```javascript
import { Socket } from "phoenix"

const socket = new Socket("ws://localhost:4000/swarm")
socket.connect()

const channel = socket.channel("swarm:my-swarm", {})

channel.join()
  .receive("ok", resp => console.log("joined", resp))          // { swarm: "my-swarm" }
  .receive("error", resp => console.error("join failed", resp)) // { reason: "swarm_not_found" }

// Lifecycle/output pushes start automatically on join
channel.on("agent_output", o => console.log(o.agent, o.content))
channel.on("agent_status", s => console.log(s.agent, "→", s.state))

// Stream events (e.g. only errors); recent history comes back in the reply
channel.push("subscribe_events", { filters: { level: "error" } })
  .receive("ok", ({ recent_events }) => console.log("recent", recent_events))

channel.on("event", e => console.log("event", e))

// Send a task to an agent
channel.push("send_task", { agent: "researcher", task: "Summarize results." })
  .receive("ok", () => console.log("task sent"))
  .receive("error", ({ reason }) => console.error("send failed", reason))
```

## See also

- [rest-api.md](rest-api.md) — the JSON REST API on the same server.
- [observability.md](observability.md) — events, logs, and the `EventStore`.
- [cli.md](cli.md) — the `swarm` CLI, including `swarm logs` and `swarm events --follow`.

---

---
description: Drive GenSwarms directly as an Elixir library — start swarms, send tasks, and manage agents from code.
---

# Programmatic API

GenSwarms is an OTP application (`:genswarms`) and can be driven directly from
Elixir. The public surface lives in the `Genswarms` module
(`lib/genswarms.ex`), which delegates to `Genswarms.SwarmManager`. This guide
covers starting and managing swarms in process, sending tasks, and subscribing
to live events over `Phoenix.PubSub`.

Add `:genswarms` as a dependency (or work inside an `iex -S mix` session in the
project) and make sure the application is started so the supervision tree,
registries, and `Genswarms.PubSub` are running.

## Public functions

| Function | Signature | Returns |
|----------|-----------|---------|
| `start_swarm/1` | `start_swarm(config_path)` | `{:ok, swarm_name}` \| `{:error, reason}` |
| `start_swarm_from_config/1` | `start_swarm_from_config(config_map)` | `{:ok, swarm_name}` \| `{:error, reason}` |
| `status/1` | `status(swarm_name)` | `{:ok, map}` \| `{:error, :not_found}` |
| `send_task/3` | `send_task(swarm_name, agent_name, task)` | `:ok` \| `{:error, reason}` |
| `list_swarms/0` | `list_swarms()` | `[map]` |
| `get_topology/1` | `get_topology(swarm_name)` | `{:ok, map}` \| `{:error, reason}` |
| `stop_swarm/1` | `stop_swarm(swarm_name)` | `{:ok, config_path}` \| `{:error, :not_found}` |

`start_swarm_from_config/1`, `list_swarms/0`, and `stop_swarm/1` are convenience
delegates to `SwarmManager.start_from_config/1`, `SwarmManager.list/0`, and
`SwarmManager.stop/1` respectively. Note that this in-process API talks to the
local `SwarmManager` GenServer; it is independent of the daemon/CLI lifecycle
that goes through SQLite.

> **Return shape note:** `stop_swarm/1` returns `{:ok, config_path}` (the path
> the swarm was started from, or `nil` if it was started from a config map), not
> a bare `:ok`. It returns `{:error, :not_found}` when the swarm isn't running.

## Starting a swarm from a file

`start_swarm/1` loads a configuration file (`.exs`, `.json`, or `.yaml`/`.yml`)
and starts the swarm. It returns the swarm name on success.

```elixir
{:ok, swarm_name} = Genswarms.start_swarm("examples/tic-tac-toe/tic_tac_toe_swarm.exs")
```

Failure modes worth handling:

- `{:error, reason}` — the config file failed to load or parse.
- `{:error, :already_exists}` — a swarm with that name is already running.
- `{:error, {:partial_start, errors}}` — the swarm record was created but one or
  more agents/objects failed to start. `errors` is a list of `{:error, reason}`
  tuples. The swarm is left in `:error` status; inspect it with `status/1` and
  stop it with `stop_swarm/1` if you want a clean restart.

## Starting a swarm from a config map

`start_swarm_from_config/1` skips file loading and takes the configuration map
directly — useful when you build configs programmatically.

```elixir
config = %{
  name: "example-swarm",
  agents: [
    %{name: :researcher, backend: :local, skills: ["web.md"]},
    %{name: :coder, backend: {:docker, "agent-coder"}, skills: ["code.md"]}
  ],
  topology: [
    {:researcher, :coder},
    {:coder, :researcher}
  ]
}

{:ok, swarm_name} = Genswarms.start_swarm_from_config(config)
```

See [configuration.md](configuration.md) for the full set of config keys.

## Inspecting and managing swarms

```elixir
# All running swarms (list of maps)
Genswarms.list_swarms()

# Detailed status for one swarm
{:ok, status} = Genswarms.status("example-swarm")

# Topology adjacency map
{:ok, topology} = Genswarms.get_topology("example-swarm")

# Stop a swarm (returns the config path it was started from, or nil)
{:ok, _config_path} = Genswarms.stop_swarm("example-swarm")
```

`status/1` returns a map with `:name`, `:status`, `:started_at`, `:config_path`,
`:agents`, `:objects`, `:agent_counts`, and a `:config` summary
(`:agent_count`, `:object_count`, `:topology_edges`). Each entry from
`list_swarms/0` carries `:name`, `:status`, `:started_at`, `:agent_count`, and
`:object_count`.

## Sending tasks to agents

`send_task/3` delivers a task string to a named agent. The agent name may be an
atom or a string; strings are converted to atoms internally before the task is
forwarded to the agent's `AgentServer`.

```elixir
Genswarms.send_task("example-swarm", :researcher, "find papers on transformers")

# A string agent name works too
Genswarms.send_task("example-swarm", "coder", "implement the parser")
```

## Runtime mutation (SwarmManager)

Beyond the `Genswarms` facade, `Genswarms.SwarmManager` exposes functions for
mutating a running swarm in place. These are not delegated through `Genswarms`,
so call them on `SwarmManager` directly. Most accept a `persist: true` option to
append the change to the swarm's overlay log so it survives a restart (default
is `false` — the change is in-memory only).

| Function | Purpose |
|----------|---------|
| `add_agent/3` | Add an agent at runtime. `opts`: `connections: [atom]`, `incoming: [atom]`, `persist: boolean`. Returns `{:ok, name}`. |
| `remove_agent/3` | Remove an agent (and its topology edges). Returns `:ok`. |
| `add_object/3` | Add a non-agentic object. Same opts as `add_agent/3`. |
| `remove_object/3` | Remove an object. |
| `add_topology_edges/3` | Add `[{from, to}]` edges. |
| `remove_topology_edges/3` | Remove `[{from, to}]` edges. |
| `scale_agent_group/4` | Scale a group `base`, `base_1`, `base_2`… to a target count. Returns `{:ok, %{added: [...], removed: [...], failed: [...]}}`. |
| `pause/1`, `resume/1`, `paused?/1` | Freeze/unfreeze the swarm's Docker containers. `pause`/`resume` return `{:ok, count}`. |
| `get_full_config/1` | Return the effective in-memory `SwarmConfig` (seed config merged with overlay). |

```elixir
alias Genswarms.SwarmManager

# Add an agent connected to :coder, persisted across restarts
{:ok, :reviewer} =
  SwarmManager.add_agent("example-swarm",
    %{name: :reviewer, backend: :local, skills: ["review.md"]},
    connections: [:coder], incoming: [:coder], persist: true)

# Scale a "fixer" pool up to 5 replicas (fixer_1 .. fixer_5)
{:ok, %{added: added, removed: removed, failed: failed}} =
  SwarmManager.scale_agent_group("example-swarm", :fixer, 5)
```

Each of these mutations broadcasts `{:topology_changed, swarm_name}` on the
`"swarm:<name>"` topic (see below).

## Subscribing to events via PubSub

GenSwarms broadcasts live activity on `Phoenix.PubSub` under the
`Genswarms.PubSub` name. Subscribe from any process and handle the messages in
`handle_info/2` (or receive them in an IEx session). Each broadcast is a plain
Erlang tuple — there is no JSON envelope at this layer.

### Per-swarm topics

The `SwarmManager`, `AgentServer`, and `Router` broadcast on swarm-scoped
topics:

| Topic | Message | Meaning |
|-------|---------|---------|
| `"swarm:<name>"` | `{:swarm_started, swarm_name, status}` | The swarm finished starting. `status` is `:running` or `:error`. |
| `"swarm:<name>"` | `{:swarm_stopped, swarm_name}` | The swarm was stopped. |
| `"swarm:<name>"` | `{:agent_added, swarm_name, name, spec}` | An agent was added at runtime. |
| `"swarm:<name>"` | `{:agent_removed, swarm_name, name}` | An agent was removed at runtime. |
| `"swarm:<name>"` | `{:topology_changed, swarm_name}` | The topology changed (agent/object/edge mutation or scaling). |
| `"swarm:<name>:output"` | `{:agent_output, agent_name, content}` | Raw agent output. |
| `"swarm:<name>:status"` | `{:agent_status, agent_name, agent_state}` | An agent changed state (`agent_state` is a string, e.g. `"idle"`). |
| `"swarm:<name>:routing"` | `{:message_routed, log_entry}` | A point-to-point message was routed. |
| `"swarm:<name>:routing"` | `{:message_broadcast, log_entry}` | A broadcast was routed. |

The `log_entry` on the `:routing` topic is a map of the form:

```elixir
%{
  timestamp: ~U[...],
  swarm: "example-swarm",
  from: :researcher,
  to: :coder,                 # an atom for :message_routed,
                              # a list of atoms for :message_broadcast
  type: :direct,              # :direct or :broadcast
  content_preview: "first 100 chars of the message"
}
```

```elixir
Phoenix.PubSub.subscribe(Genswarms.PubSub, "swarm:example-swarm:routing")

receive do
  {:message_routed, entry} -> IO.inspect(entry, label: "routed")
  {:message_broadcast, entry} -> IO.inspect(entry, label: "broadcast")
end
```

### Observability event stream

The centralized event log exposes a helper API on
`Genswarms.Observability.LogStore` so you do not have to hardcode topic strings.
Events are broadcast as `{:log_event, event}`.

```elixir
alias Genswarms.Observability.LogStore

# All events
LogStore.subscribe()

# Only events for one swarm
LogStore.subscribe("example-swarm")

# Later
LogStore.unsubscribe()
```

> **Note:** `LogStore.unsubscribe/0` only unsubscribes from the global
> `"log_store:events"` topic. If you subscribed to a swarm-specific stream with
> `LogStore.subscribe("example-swarm")`, unsubscribe from it directly with
> `Phoenix.PubSub.unsubscribe(Genswarms.PubSub, "log_store:events:example-swarm")`.

A subscriber process then receives:

```elixir
def handle_info({:log_event, event}, state) do
  IO.inspect(event, label: "event")
  {:noreply, state}
end
```

Each `event` is a map with `:id`, `:timestamp`, `:level`
(`:debug | :info | :warning | :error`), `:category`
(`:backend | :routing | :agent | :object | :swarm | :system`), `:swarm`,
`:agent`, `:event_type`, `:message`, and `:metadata`.

Under the hood, `LogStore.subscribe/0` subscribes to the `"log_store:events"`
topic and `LogStore.subscribe/1` to `"log_store:events:<swarm>"`. Prefer the
helper functions over subscribing to the raw topics. See
[observability.md](observability.md) for querying historical events.

### Worked example: a GenServer subscriber

```elixir
defmodule ExampleSwarm.Watcher do
  use GenServer
  alias Genswarms.Observability.LogStore

  def start_link(swarm), do: GenServer.start_link(__MODULE__, swarm)

  @impl true
  def init(swarm) do
    LogStore.subscribe(swarm)
    Phoenix.PubSub.subscribe(Genswarms.PubSub, "swarm:#{swarm}:output")
    {:ok, %{swarm: swarm}}
  end

  @impl true
  def handle_info({:log_event, event}, state) do
    IO.inspect(event, label: "event")
    {:noreply, state}
  end

  def handle_info({:agent_output, agent, content}, state) do
    IO.puts("#{agent}: #{content}")
    {:noreply, state}
  end
end
```

## See also

- [objects.md](objects.md) — building deterministic non-agentic components
- [rest-api.md](rest-api.md) — the HTTP equivalent of these operations
- [observability.md](observability.md) — querying and streaming events
- [configuration.md](configuration.md) — the swarm configuration DSL
---

---
description: Observe GenSwarms swarms — stream logs, query events, and track metrics across agents and the runtime.
---

# Observability

GenSwarms exposes everything that happens in a swarm through a single event spine.
Every observable state transition emits a `:telemetry` event, a bridge funnels
those into a centralized store, and the store persists, streams, and serves them.
Understanding that spine is the key to building any dashboard, monitor, or alerting
on top of the framework.

## The single spine

There is one rule: every observable state transition emits a `:telemetry` event,
and nothing else logs it. A telemetry bridge funnels those events into `LogStore`,
which both persists them (ETS + durable store) and streams them over PubSub /
WebSocket.

A transition is logged in exactly one place: its `emit_telemetry/2,3` call. The
emitter never also calls `LogStore.log` for the same moment. `LogStore.log` is
reserved for diagnostics and I/O that have no transition event: backend container
ops, raw agent stdout, received messages, config-load failures. Those are
single-source, so they never double up with the bridge.

```
emit_telemetry(:agent_started, ...)              # emitters (swarm_manager, agent_server, ...)
        |  [:genswarms, :agent, :agent_started]
        v
Genswarms.Observability.TelemetryBridge            # single :telemetry handler
        |  LogStore.log(:info, :agent, :agent_started, "agent fixer_1 started", ...)
        v
Genswarms.Observability.LogStore
        |-- ETS ring buffer        -> LogStore.query / GET /api/events (fast, in-node)
        |-- EventStore (durable)   -> cross-process / `genswarms events` CLI
        +-- PubSub {:log_event, e} -> SwarmChannel "event" / "log_entry" push (live)
```

To make a new transition observable, emit a telemetry event under
`[:genswarms, <domain>, <event>]` and add it to
`Genswarms.Observability.TelemetryBridge` `known_events/0`. Nothing else: no
controller, no broadcast, no `LogStore` call at the call site.

The bridge attaches **once at application start**
(`Genswarms.Observability.TelemetryBridge.attach/0`, called by
`Genswarms.Application.start/2` after the supervision tree is up). It attaches to
the concrete `[:genswarms, domain, event]` triples in `known_events/0` — not a
prefix — so unrelated `:genswarms` telemetry is never swept in. The handler is
wrapped in a rescue: if a translation fails it logs a warning and drops the event
rather than taking down the emitting process.

The bridge derives the log `level` from the event name. When the level depends on
the outcome (a partial swarm start, an unexpected agent exit), the emitter passes
`level:` in the telemetry metadata to set it explicitly; the bridge strips that key
before persisting, so it never leaks into the payload.

## Event taxonomy

`level` is derived from the event name. The exact rules
(`TelemetryBridge.level_for/1`), checked in order against the event-name string:

| Substring in event name | Level |
|---|---|
| contains `error` | `:error` |
| contains `failed` | `:error` |
| contains `invalid` | `:warning` |
| contains `not_found` | `:warning` |
| contains `full` | `:warning` |
| otherwise | `:info` |

A `level:` key in the telemetry metadata overrides this derivation for that event.

`category` is the telemetry domain, with one normalization: the `:router` domain
is mapped to the `:routing` category. All other domains pass through unchanged
(`:swarm`, `:agent`, `:object`).

The full vocabulary the bridge knows
(`Genswarms.Observability.TelemetryBridge.known_events/0`):

| Domain (category) | Event | Level | Meaning |
|---|---|---|---|
| `swarm` | `swarm_started` | info | swarm finished starting (emitters may pass `level:` for a partial start) |
| `swarm` | `swarm_stopped` | info | swarm torn down |
| `agent` | `agent_started` | info | agent process up |
| `agent` | `agent_ready` | info | event-driven backend reached a recognized safe prompt |
| `agent` | `agent_blocked` | info | backend is waiting for human trust/permission input |
| `agent` | `agent_needs_attention` | info | durable turn/session recovery needs operator review |
| `agent` | `agent_interrupted` | info | active backend turn was interrupted without deleting its session |
| `agent` | `agent_send_failed` | error | backend refused or failed a turn delivery |
| `agent` | `agent_stopped` | info | agent process exited |
| `agent` | `agent_error` | error | agent backend/runtime error |
| `agent` | `agent_added` | info | agent added to a running swarm |
| `agent` | `agent_removed` | info | agent removed from a running swarm |
| `agent` | `task_sent` | info | task delivered to an agent |
| `agent` | `message_delivered` | info | message delivered to target inbox |
| `object` | `object_started` | info | object handler initialized |
| `object` | `object_stopped` | info | object handler stopped |
| `object` | `object_error` | error | object handler crashed/errored |
| `object` | `object_added` | info | object added to a running swarm |
| `object` | `object_removed` | info | object removed from a running swarm |
| `routing` | `message_routed` | info | direct message routed (`:from`, `:to`) |
| `routing` | `message_broadcast` | info | broadcast routed (`:from`) |
| `routing` | `invalid_route` | warning | message rejected by topology |

Every event carries `:swarm` in its metadata. Agent events also carry `:agent`,
and object events carry `:object`. The bridge lifts the swarm name into the event's
`:swarm` field and lifts **either** the agent name **or** the object name into the
event's `:agent` field (`agent: metadata[:agent] || metadata[:object]`) — there is
no separate object column, so object events appear under the same `agent` field /
`-a` / `?agent=` filter as agents. The bridge then drops `:swarm`, `:agent`, and
`:object` from the remaining metadata (and the `:level` override key) and keeps the
rest as a JSON-friendly `metadata` blob.

> **Category naming note.** The `:object` category is real — the bridge emits it
> and `genswarms events --category object` filters on it. For historical reasons
> the `LogStore` `@type category` typespec and the `EventsController` /
> [rest-api.md](rest-api.md) `category` docs enumerate only
> `backend | routing | agent | swarm | system` and omit `object`. The omission is
> documentation/typespec drift, not a runtime restriction: object-category events
> are persisted and queryable through every path. Use the taxonomy above as the
> authoritative list.

## Telemetry events and metrics

Raw telemetry events are emitted under `[:genswarms, <domain>, <event>]` with
metadata that always includes `:swarm` (and `:agent` for agent events). The
`Genswarms.Telemetry` supervisor declares a set of `Telemetry.Metrics` definitions
(consumable by LiveDashboard or any reporter). Metric names follow
`genswarms.<domain>.<event>`:

| Metric | Type | Tags | Source event |
|---|---|---|---|
| `genswarms.swarm.swarm_started.count` | counter | `:swarm` | `[:genswarms, :swarm, :swarm_started]` |
| `genswarms.swarm.swarm_stopped.count` | counter | `:swarm` | `[:genswarms, :swarm, :swarm_stopped]` |
| `genswarms.swarm.agent_count` | last_value | `:swarm` | `[:genswarms, :swarm, :agent_count]` |
| `genswarms.agent.agent_started.count` | counter | `:swarm`, `:agent` | `[:genswarms, :agent, :agent_started]` |
| `genswarms.agent.agent_stopped.count` | counter | `:swarm`, `:agent` | `[:genswarms, :agent, :agent_stopped]` |
| `genswarms.agent.agent_error.count` | counter | `:swarm`, `:agent` | `[:genswarms, :agent, :agent_error]` |
| `genswarms.agent.task_sent.count` | counter | `:swarm`, `:agent` | `[:genswarms, :agent, :task_sent]` |
| `genswarms.agent.message_delivered.count` | counter | `:swarm`, `:agent` | `[:genswarms, :agent, :message_delivered]` |
| `genswarms.router.message_routed.count` | counter | `:swarm` | `[:genswarms, :router, :message_routed]` |
| `genswarms.router.message_broadcast.count` | counter | `:swarm` | `[:genswarms, :router, :message_broadcast]` |
| `genswarms.router.invalid_route.count` | counter | `:swarm` | `[:genswarms, :router, :invalid_route]` |

The `genswarms.swarm.agent_count` last-value is produced by a periodic
`:telemetry_poller` measurement (period 10s) that calls
`Genswarms.Telemetry.measure_swarms/0`, which polls `Genswarms.SwarmManager.list/0`
and emits `[:genswarms, :swarm, :agent_count]` per swarm. (`agent_count` is a
metrics-only event — it is **not** in `known_events/0`, so the bridge does not turn
it into a `LogStore` event; it never appears in the queryable event stream.)
Standard Phoenix and BEAM VM metrics (`phoenix.endpoint.*`,
`phoenix.router_dispatch.*`, `phoenix.live_view.mount.*`, `vm.memory.total`,
`vm.total_run_queue_lengths.*`) are also registered.

## The EventStore behaviour

The durable, cross-process log sits behind one swappable interface,
`Genswarms.Observability.EventStore` (a behaviour plus a facade). Everything that
persists, reads, or tails events goes through it (`LogStore`, `EventRelay`, the
controllers, the channel, and the CLI), never a concrete backend. The backend is a
single config knob.

The callbacks are: `persist/1`, `query/1`, `events_since/2`, `max_event_id/0`, an
optional `persist_many/1` (bulk write), and an optional `child_specs/0` (processes
the backend needs supervised; the app splices them into its tree at boot via
`EventStore.child_specs/0`).

```elixir
@callback persist(event()) :: :ok
@callback persist_many([event()]) :: :ok                              # optional
@callback query(keyword()) :: [event()]
@callback events_since(since_id :: non_neg_integer(), limit :: pos_integer()) :: [event()]
@callback max_event_id() :: non_neg_integer()
@callback child_specs() :: [Supervisor.child_spec()]                  # optional

@optional_callbacks child_specs: 0, persist_many: 1
```

The facade provides safe defaults for the optional callbacks: if a backend does not
export `persist_many/1`, `EventStore.persist_many/1` falls back to N× `persist/1`;
if it does not export `child_specs/0`, `EventStore.child_specs/0` returns `[]`. The
event shape passed to `persist/1` is a map with `:level`, `:category`,
`:event_type`, `:message` and optional `:swarm`, `:agent`, `:metadata`; reads
return the same maps additionally carrying `:id` and `:timestamp` (assigned by the
backend).

### Backends

| Backend | Role |
|---|---|
| `EventStore.Sqlite` | Thin adapter over the `events` table in `.genswarms/swarms.db` (managed by `SwarmRegistry`). Stateless and synchronous — each call opens a short-lived connection — so it declares no `child_specs/0` (no supervised process). |
| `EventStore.Buffered` | Engine-independent write-batching decorator wrapping any inner backend. The default. |

The default in `config/config.exs` batches writes on top of SQLite:

```elixir
config :genswarms, :event_store, Genswarms.Observability.EventStore.Buffered

config :genswarms, Genswarms.Observability.EventStore.Buffered,
  inner: Genswarms.Observability.EventStore.Sqlite,
  interval_ms: 100,
  max_buffer: 1_000
```

`EventStore.Buffered` enqueues each `persist/1` (and `persist_many/1`) into a
`Writer` GenServer and flushes the batch via the inner backend's `persist_many/1`
on a 100ms timer (or sooner when `max_buffer` is reached). It also declares the
`Writer` (plus any of the inner backend's own children) through its `child_specs/0`,
so the app supervises it. This keeps disk writes off `LogStore`'s critical path and
lets the inner backend amortize them: with `EventStore.Sqlite`, one
`open -> BEGIN -> inserts -> COMMIT -> close` per flush instead of a connection per
event. The tradeoff is a small durability window (at most one flush interval) on a
hard crash; the live in-node path (ETS + PubSub) is synchronous and unaffected. The
buffer is flushed on graceful shutdown via the `Writer`'s `terminate/2`.

Tests run with the plain synchronous `EventStore.Sqlite` backend so that
`persist -> query` is deterministic with no buffering (`config/test.exs` sets
`config :genswarms, :event_store, Genswarms.Observability.EventStore.Sqlite`).

To raise throughput under load, tune the buffer (fewer, larger commits in exchange
for a slightly larger latency/durability window):

```elixir
config :genswarms, Genswarms.Observability.EventStore.Buffered,
  inner: Genswarms.Observability.EventStore.Sqlite,
  interval_ms: 250,
  max_buffer: 5_000
```

Because every caller goes through the facade, a future `EventStore.Postgres` or a
Redis/streaming backend can be swapped in transparently: only the backend module
changes, not the emitters, the bridge, the channel, or `EventRelay`.

## Cross-process event stream

A BEAM's in-memory machinery (PubSub, the process `Registry`, the ETS `LogStore`)
is node-local and invisible from another OS process. GenSwarms supports two
deployment shapes.

Co-located: the swarm runs in the same BEAM as the Phoenix endpoint (for example,
started in-process via `POST /api/swarms`). Live PubSub, the WebSocket stream, and
the live snapshot endpoints all work directly. Nothing special is needed.

Monitor + daemons: the usual shape at scale. Each swarm runs as its own daemon
(`genswarms start`, its own BEAM), and a separate monitor/API node observes all of
them. The only thing the processes share is the SQLite `events` table in
`.genswarms/swarms.db`.

```
daemon swarm A --+  emit_telemetry -> LogStore -> SQLite events  --+
daemon swarm B --+                                                 +--> shared .genswarms/swarms.db
daemon swarm C --+                                                 --+
                                                                   |
                  monitor / API node ---- EventRelay polls events_since --+
                                                                   |       |
                  REST /api/events  -- reads SQLite ---------------+       v
                  WS swarm:<name>   <- EventRelay re-broadcasts {:log_event} onto
                                       the same LogStore PubSub topics -> SwarmChannel push
```

- `GET /api/events` (and the swarm/agent variants) read SQLite, so they surface
  every swarm, daemon or in-process.
- `Genswarms.Observability.EventRelay` runs on the monitor node. It tails new
  SQLite rows every 500ms (the `:interval` default; batches of 500 via
  `EventStore.events_since/2`) and re-broadcasts them onto the in-node PubSub
  topics (`log_store:events` and `log_store:events:<swarm>`), mirroring
  `LogStore.broadcast_event/1` exactly, so the existing `SwarmChannel` pushes them
  to WebSocket clients live, with no clustering required. It starts from the
  current tip on boot (relays only new events going forward; history comes from the
  subscribe-time snapshot). Latency is approximately the poll interval.
- A WebSocket client gets recent history from the snapshot on subscribe (also read
  from SQLite) and the live tail from the relay.

The relay is started by `Genswarms.Application.start_web_server/1`
(`maybe_start_event_relay/0`) — and only there, so daemons (which never start the
web server) do not run it. Run it only on a monitor/API node that does not host
swarms in-process: there the in-node `LogStore` never broadcasts swarm events (they
happen in the daemons), so the relay is the sole live source and there is no
double-delivery. To disable it explicitly, set:

```elixir
config :genswarms, :event_relay, false
```

### Still node-local (known limits)

- Live process-state pulls (`GET /objects/:name`, `/agents/:name`) call into the
  in-node `Registry`, so they only reach swarms in the same BEAM. For daemon
  swarms, rely on the event stream (and `GET /swarms/:name`, which has a SQLite
  fallback) instead of synchronous state pulls.
- Object internal state changes do not emit events (only object lifecycle does), so
  a live object-state feed is not available: poll `GET /objects/:name` in a
  co-located setup, or have the object emit on change.
- For true sub-second push or cross-host fan-out without polling, swap
  `Phoenix.PubSub` to its Redis adapter; the single spine means only the transport
  changes, not the emitters, bridge, or channel.

## Querying events

### CLI

`genswarms events` reads the durable spine, so it surfaces daemon swarms running in
other BEAMs by reading the `events` table in `.genswarms/swarms.db`.

```bash
genswarms events                      # recent events across all swarms
genswarms events -s my-swarm          # one swarm
genswarms events --category routing   # filter by category (backend|routing|agent|object|swarm|system)
genswarms events --errors             # errors only
genswarms events --follow             # stream in real time
```

The `--category` values map 1:1 to the taxonomy above (and include `object`). See
[cli.md](cli.md) for the full command reference.

### REST API

History endpoints are backed by the same spine and read from SQLite, so they cover
every swarm:

| Endpoint | Returns |
|---|---|
| `GET /api/events` | recent events, filterable by `level`/`category`/`swarm`/`agent`/`event_type`/`minutes`/`limit` |
| `GET /api/swarms/:name/events` | events for one swarm |
| `GET /api/swarms/:name/agents/:agent_name/events` | events for one agent |

Snapshot endpoints (current state) complement the history:

| Endpoint | Returns |
|---|---|
| `GET /api/swarms/:name` | swarm status, agents, objects, counts |
| `GET /api/swarms/:name/topology` | topology adjacency |
| `GET /api/swarms/:name/objects` | objects + lifecycle state |
| `GET /api/swarms/:name/objects/:object_name` | one object's live domain state |
| `GET /api/swarms/:name/agents/:agent_name` | one agent's status |

See [rest-api.md](rest-api.md) for full details.

### Real-time (WebSocket / PubSub)

On the `swarm:<name>` channel, subscribe and patch a view from the push stream.
The channel subscribes to the per-swarm PubSub topics on join (`swarm:<name>`,
plus `:output`, `:routing`, and `:status` sub-topics) and pushes these
server→client messages (all confirmed in `SwarmChannel.handle_info/2`):

| Push | Source |
|---|---|
| `event`, `log_entry` | `LogStore` (the whole taxonomy, after `subscribe_events` / `subscribe_logs`) |
| `agent_output` | agent stdout |
| `agent_status` | agent state transition |
| `message_routed`, `message_broadcast` | router |
| `swarm_started`, `swarm_stopped` | swarm lifecycle |
| `agent_added`, `agent_removed`, `topology_changed` | dynamic mutations |

Within a single BEAM, code can subscribe directly with
`Genswarms.Observability.LogStore.subscribe/0` (all events as `{:log_event, event}`)
or `LogStore.subscribe/1` (one swarm). See [websocket.md](websocket.md) for the
channel protocol.

## Building a dashboard

A dashboard is a consumer, not framework code (the project is API-first and
headless by design; no HTML ships here):

1. Bootstrap from the snapshot endpoints (status + topology + objects).
2. Open the `swarm:<name>` channel, `subscribe_events` for the taxonomy, and patch
   the view from the push stream.
3. Domain-specific concepts (for example user "sessions" or conversation
   transcripts) live in the consumer and read from the consumer's own store; the
   framework stays generic and exposes only generic object state via the
   introspection endpoint above.

## See also

- [cli.md](cli.md) — `genswarms events` and the full CLI reference
- [rest-api.md](rest-api.md) — `/api/events` and snapshot endpoints
- [websocket.md](websocket.md) — real-time channel protocol

---

---
description: Test GenSwarms swarms — validate configs and run examples with mix genswarms.test and the mock backend.
---

# Testing and development

GenSwarms ships two layers of tests: fast ExUnit unit tests for individual modules
and an end-to-end harness (`mix genswarms.test`) that validates and runs the example
swarm configurations. A mock backend lets you exercise topologies and routing
without any LLM API calls. This page covers both, plus formatting and the
development server.

## Unit tests

Run the ExUnit suite with `mix test`:

```bash
mix test                                            # run all tests
mix test --cover                                    # run with coverage report
mix test test/genswarms/routing/router_test.exs     # a single file
mix test test/genswarms/routing/router_test.exs:42  # a single test by line
mix test --only tag_name                            # only tests with a given tag
```

Tests run with the synchronous `EventStore.Sqlite` backend (set in `config/test.exs`
via `config :genswarms, :event_store, Genswarms.Observability.EventStore.Sqlite`) so
that persist→query is deterministic, rather than the buffered default. The same
config disables the Phoenix endpoint (`server: false`) and lowers the log level to
`:warning`. Key test files include:

| File | Covers |
|---|---|
| `test/genswarms/agents/agent_protocol_test.exs` | `@agent:` message parsing |
| `test/genswarms/routing/router_test.exs` | message routing (incl. system objects) |
| `test/genswarms/config/loader_test.exs` | config loading (`.exs`/`.json`/`.yaml`/`.yml`) |
| `test/genswarms/config/swarm_config_test.exs` | config validation |
| `test/genswarms/agents/inbox_test.exs` | the message queue |

## Formatting

```bash
mix format    # format all source per .formatter.exs
```

Run `mix format` before committing; CI and reviewers expect formatted code.

## End-to-end harness

`mix genswarms.test` discovers, validates, and runs every example. It:

1. Discovers all swarm configs (`.exs`) and sim files (`.sim`) recursively under `examples/`.
2. Validates each one.
3. Runs each (starts the swarm, waits the full timeout, then stops it).
4. Captures a per-example log.
5. Reports pass/fail/skip for each, with a combined summary.

```bash
mix genswarms.test                           # validate + run all examples
mix genswarms.test --validate-only           # only validate configs, don't run
mix genswarms.test --example tic-tac-toe     # test a specific example
mix genswarms.test --mock script.json        # run real agents with the mock script (no LLM)
mix genswarms.test --timeout 60000           # custom timeout per swarm (ms)
mix genswarms.test --steps 3                 # steps for .sim examples
mix genswarms.test --logs-dir /tmp/logs      # custom logs directory
mix genswarms.test --quiet                   # suppress per-example info lines
```

### Flags

All flags use `--flag value` (single dash, hyphenated) form and are parsed in
strict mode — unknown flags are dropped silently.

| Flag | Type | Default | Purpose |
|---|---|---|---|
| `--validate-only` | boolean | off | Validate configs only; skip running. No logs directory or `summary.log` is written. |
| `--example <name>` | string | all | Keep only files whose path contains the literal substring `/<name>/` |
| `--timeout <ms>` | integer | `60000` | Per-swarm/per-sim run timeout in milliseconds |
| `--steps <n>` | integer | `3` | Number of steps for `.sim` examples (ignored by `.exs` configs) |
| `--mock <path>` | string | none | Expand `<path>` and export it as `SUBZEROCLAW_MOCK_SCRIPT` for LLM-free runs |
| `--logs-dir <path>` | string | `.test-logs` | Directory for captured run logs (expanded with `Path.expand/1`) |
| `--quiet` | boolean | off | Suppress per-example info output (failures are still printed) |

The `--example` filter matches a path *segment*, so it works against the
directory name. For instance, `--example tic-tac-toe` selects
`examples/tic-tac-toe/tic_tac_toe_swarm.exs` because the path contains
`/tic-tac-toe/`. The bundled example directories are: `bridge`, `bwrap-skills`,
`dynamic-swarm`, `massive-swarm`, `party`, and `tic-tac-toe`.

### Output

Unless `--validate-only` is set, each example writes a `<name>.log` to the logs
directory (default `.test-logs/`), plus a combined `summary.log`. The log
filename is derived from the swarm **name** (the config `:name`, not the file
path), sanitized by replacing every character outside `[a-zA-Z0-9_-]` with `_`
(so hyphens and underscores are preserved). For the bundled examples, whose
config names are `tic-tac-toe` and `party-test`, the resulting files are:

```
.test-logs/
├── tic-tac-toe.log
├── party-test.log
└── summary.log
```

A per-example `.log` records the relative path, swarm name, agent/object counts,
topology, timeout, and the final status (or `TIMEOUT` / `ERROR`). The
`summary.log` contains the `N passed, N failed, N skipped` header followed by one
`✓`/`✗`/`⊘` line per example.

Exit codes:

- `0` — all examples passed (or were skipped).
- `1` — at least one example failed, **or** no `.exs`/`.sim` files were found under `examples/`.

With `--validate-only`, no logs directory is created and no `summary.log` is
written — results are printed to the console only.

A run is a **skip** (`⊘`) when an `.exs` file evaluates to something that is **not
a map** (the `(not a swarm config)` case), or when a `.sim` file is found but
`SubzeroSim` is not available in the project. Note that an `.exs` file evaluating
to a map *without* a `:name` key is **not** a skip — it counts as a **pass**
(reported as `(valid map config)`); only a swarm config (a map *with* a `:name`
key) is actually started and run.

## Testing without an LLM

There are two distinct ways to avoid real LLM calls; they serve different goals.

### The `:mock` backend — test orchestration

`backend: :mock` is a stub that spawns no external process and produces no agent
output. It is for testing swarm **orchestration** — topology, routing, and
dynamic add/remove/scale — deterministically and instantly:

```elixir
%{
  name: "test-swarm",
  agents: [
    %{name: :researcher, backend: :mock},
    %{name: :coder, backend: :mock}
  ],
  topology: [{:researcher, :coder}]
}
```

`MockBackend.send_input/2` and `deploy_skills/2` are no-ops, and `handle_output/2`
always returns empty output, so the backend does not exercise agent reasoning —
only the machinery around agents. An optional `%{script: [...]}` (via
`{:mock, %{script: [...]}}`) is stored on the backend struct for introspection
but is never used to generate responses. See [backends.md](backends.md).

### `--mock` / `SUBZEROCLAW_MOCK_SCRIPT` — run real agents with canned responses

To run *real* agents (local/docker/apple_container/bwrap) end to end without calling an LLM, give
subzeroclaw a mock script:

```bash
mix genswarms.test --mock path/to/script.json
```

The task expands the path with `Path.expand/1` and exports it as
`SUBZEROCLAW_MOCK_SCRIPT`, which is passed through to the agents (including bwrap
sandboxes). The `subzeroclaw` runtime — not GenSwarms — reads the script and
returns canned responses instead of calling the API. The script format is
defined by subzeroclaw.

You can also set `SUBZEROCLAW_MOCK_SCRIPT` directly in the environment to get the
same behavior outside the test harness.

## Development server

Start the Phoenix API server (REST + WebSocket, no HTML) for local development:

```bash
mix phx.server
```

The server defaults to port `4000` (override with the `PORT` environment variable).
A typical loop is to start the server, then drive it from the CLI or HTTP client:

```bash
genswarms start examples/tic-tac-toe/tic_tac_toe_swarm.exs   # start a swarm as a daemon
genswarms events --follow                                     # watch the event stream live
```

## See also

- [backends.md](backends.md) — backend types including the mock backend
- [cli.md](cli.md) — `genswarms` command reference
- [configuration.md](configuration.md) — swarm config DSL and validation

---

---
description: Troubleshoot GenSwarms — fixes for agents that won't start, messages not routing, and common backend issues.
---

# Troubleshooting

Common problems running GenSwarms and how to fix them. Most issues fall into agent startup, message routing, backend setup, task delivery, or the API server.

Before digging in, two commands surface most problems:

```bash
genswarms status [name]          # Swarm/agent lifecycle state
genswarms events --errors        # Recent error events across all swarms
```

## Agent not starting

1. Confirm the `subzeroclaw` binary is reachable. The bwrap backend searches in this order: explicit config (`subzeroclaw_path`), `../subzeroclaw/subzeroclaw` (a sibling checkout), the `SUBZEROCLAW_PATH` env var, then `PATH`. If none resolve to a regular file, the agent fails to start.
2. Verify your LLM provider key is set (`SUBZEROCLAW_API_KEY`), since agents need it to call the model. (If you are running without an LLM for testing, set `SUBZEROCLAW_MOCK_SCRIPT` instead so subzeroclaw returns canned responses.)
3. Inspect the swarm and agent state:

```bash
genswarms status example-swarm
genswarms logs example-swarm researcher
```

## Messages not routing

1. Make sure the topology allows the edge `source -> target`. The `Router` only routes along configured topology edges (system objects `:metrics`, `:tick`, and `:gateway` are always allowed without an explicit edge).
2. Check the agent is emitting the correct `@agent:` syntax, for example `@coder: please implement this`. Use `@all:` to broadcast to all connected agents.
3. Review the message log (the `limit` query param defaults to 100):

```bash
curl http://localhost:4000/api/swarms/example-swarm/messages
curl "http://localhost:4000/api/swarms/example-swarm/messages?limit=20"
```

4. As an alternative to `@agent:` syntax, agents can drop a JSON file (`{"to":"target","content":"msg"}`) into `{workspace}/.outbox/`; the LogWatcher polls that directory and routes it. Inside a container, the `swarm-msg send <target> <msg>` helper writes these files for you (it JSON-encodes the message and writes it into `/workspace/.outbox/`).

## SSH backend fails

1. Confirm key-based SSH works first: `ssh user@host` should connect without a password prompt.
2. Verify the remote `subzeroclaw` path is correct on the target host. On NixOS machines the backend defaults to skills at `/var/lib/subzeroclaw/skills` and runs the agent as the `subzeroclaw` user (via `sudo -u`); for non-NixOS hosts set `nixos: false` in the backend opts so it uses `~/.subzeroclaw/skills` and runs as the login user.
3. Ensure the remote skills/workspace directory is writable for the SSH user — skills are copied over via SFTP at startup.

## Docker backend fails

1. Check the Docker daemon is up: `docker ps`.
2. Confirm the agent image exists: `docker images`. Build images with `nix build .#agentContainer-<preset>` and `docker load < result` (presets: `base`, `web`, `code`, `data`, `python`, `node`, `full`). If the expected image is missing, the backend tries to build it via `nix` and otherwise falls back to `szc-agent-base:latest`.
3. Inspect a container's logs directly. GenSwarms names containers `szc-{swarm}-{agent}`:

```bash
docker logs szc-example-swarm-coder
```

4. Containers are run with `--rm`, so a crashed agent leaves no container behind. Catch the failure in the event log instead:

```bash
genswarms events --category backend
```

## Tmux agent is stuck at `starting`, `blocked`, or `needs_attention`

1. Confirm host-side tmux is installed with `tmux -V`. For `runner: :host`, also
   run `codex --version` (or `claude --version` / `opencode --version`) on the
   host. For an isolated runner, inspect the `runner` object returned by the
   session endpoint and use the checks below.
2. Fetch the exact session metadata and attach command:

```bash
curl http://localhost:4000/api/swarms/<swarm>/agents/<agent>/session
tmux -L genswarms attach-session -r -t genswarms-<swarm>:<agent>
```

   Remove `-r` only when you intend to answer a trust/permission prompt or steer
   the client. Detach with `Ctrl-b d`; the pane continues running.
3. `blocked` means the visible tail resembles a trust or permission prompt.
   Resolve it manually or interrupt the current turn with:

```bash
curl -X POST http://localhost:4000/api/swarms/<swarm>/agents/<agent>/interrupt
```

4. `needs_attention` means GenSwarms found an unacknowledged turn, an invalid
   completion receipt, or uncertain `send-keys` delivery. Inspect
   `<workspace>/.genswarms/turns/<swarm>/<agent>/<turn-id>/`. Preserve the turn
   directory while diagnosing it: `task.md` is the durable request, `reply.md`
   plus `done.json` is the completion, and `ack.json` proves GenSwarms handled
   it. A completed but unacknowledged turn may be delivered again after restart.
   `attention_reason: "nudge_not_submitted"` means the exact nudge remained at
   the active cursor after the backend's one Enter-only retry; the task text was
   not duplicated.
5. If the TUI prompt is not recognized, update its adapter pattern. Use
   `quiet_ready_fallback: true` only for a trusted, known client screen; a quiet
   terminal is not proof that a TUI is ready.
6. A bwrap pane uses a short `xargs` parent command while the actual client runs
   below it. If startup reports an argv-manifest or `xargs` error, confirm GNU
   `xargs` is available on the host and that
   `<state_dir>/.genswarms/host-launch.argv0` is a regular mode-0600 file. Do not
   print that manifest into a shared log.

For `runner: :docker`:

```bash
docker inspect gstui-<swarm>-<agent>
docker exec gstui-<swarm>-<agent> codex --version  # client_source: runtime
docker exec gstui-<swarm>-<agent> "$(readlink -f "$(command -v codex)")" --version  # host_nix
```

The container image must contain the client when `client_source: :runtime`.
With `client_source: :host_nix`, the host client must resolve into `/nix/store`
and `nix-store --query --requisites <store-root>` must succeed. A
`container_identity_mismatch` means a persistent same-named container was
created with a different image/mount/network/resource contract; explicitly stop
the agent to destroy it, then start with the new config. An environment mismatch
reports only the variable name, never its value.

For `runner: :bwrap`, confirm `bwrap` and `/run/swarm/sandbox-base/base` exist.
Rootless TUI panes have no virtual-address limit by default; an explicit small
`memory_limit` becomes `RLIMIT_AS` and may crash JS clients such as OpenCode even
when their resident memory is modest. Explicit cgroup mode defaults to a `2G`
hard limit. The sandbox lives at
`/run/swarm/agents/gstui-<swarm>-<agent>` and is removed after an explicit
destroy or a failed fresh preparation.

`network: :none` is a full cutoff and will also break cloud model calls.
`network: :isolated` intentionally fails for interactive TUI runners because
the existing LLM-only forwarder is specific to subzeroclaw. Docker supports
`:open`, `:none`, or a named network; bwrap supports `:open` or `:none`.

## Apple container backend fails

1. Confirm the `container` CLI is installed and the service is running:

```bash
container system status --format json
container system start
```

2. Confirm the image exists in Apple's local image store:

```bash
container image inspect szc-agent-code:latest
```

Build preset images with `nix build .#agentContainer-<preset> -o result`. Current Nix container outputs are Docker archives; Apple `container image load` expects an OCI archive. Convert or publish the image before loading it into Apple's image store, for example:

```bash
docker load -i result
skopeo copy docker-daemon:szc-agent-base:latest oci-archive:szc-agent-base-oci.tar:szc-agent-base:latest
container image load --input szc-agent-base-oci.tar
```

The backend tries the build/load path when an image is missing, but a failed build, missing Nix, or incompatible archive leaves the final image error to `container run`.

3. Inspect a container directly. GenSwarms names Apple containers `szc-{swarm}-{agent}` unless `container_name` is set:

```bash
container inspect szc-example-swarm-coder
container logs -n 50 szc-example-swarm-coder
```

4. If the error is `{:unsupported_network, :isolated}`, the backend is refusing to run with open network. Apple `container` does not currently expose the egress-forwarding semantics GenSwarms uses for Docker/bwrap isolation; use Docker or bwrap for agents that require `network: :isolated`.

5. Pause/resume is Docker-only. Apple `container` agents keep running when you call the pause/resume endpoints or Mix tasks.

## Tasks not delivered to daemon swarms

Daemon swarms (started with `genswarms start`) receive tasks through a SQLite-backed queue, not directly. The daemon polls the queue every 500ms.

1. Confirm the daemon is actually running: `genswarms status`.
2. Look for queued/processed task activity in the event log:

```bash
genswarms events --category agent
```

3. Inspect the queue itself in `.genswarms/swarms.db` (the `tasks` table) to confirm rows are inserted with status `pending` and later flipped to `processed`.
4. Check for errors:

```bash
genswarms events --errors
```

> Valid `--category` values: `backend`, `routing`, `agent`, `object`, `swarm`, `system`. Add `-s <swarm>` to scope to one swarm. `genswarms events` performs a one-shot query and prints the matching events (default limit 50); it does not continuously tail.

## API returns errors

1. Confirm the API server is up (the root path returns API info):

```bash
curl http://localhost:4000/
```

2. If a browser frontend is failing, CORS is already permissive on the API server (`origins: "*"`, all methods and headers allowed), so a CORS rejection usually points to a wrong URL or the server being down rather than a CORS policy.
3. Read the server output for the detailed error; start it in the foreground with `mix phx.server` while debugging.

## Cleaning up stuck state

If swarms are left in a `stopped` or `crashed` state, or the database accumulates stale rows, clean them up via the mix task:

```bash
mix genswarms.clean          # Remove stopped/crashed swarm entries and their files
mix genswarms.clean --all    # Also clear the event log
```

> The `clean` operation is not exposed as an escript subcommand — `genswarms clean` is not a recognized command and will error. Use the `mix genswarms.clean` task or the API route below.

Via the API, `POST /api/swarms/clean` removes stopped/crashed swarms (add `?all=true` to also clear the event log). To remove a single swarm and all of its data, `DELETE /api/swarms/:name?purge=true` stops the swarm and deletes its files, events, and queued tasks.

## See also

- [CLI reference](cli.md)
- [Backends](backends.md)
- [Observability](observability.md)