# HoBo — full documentation > The complete HoBo docs as one markdown document, in reading order. > The condensed briefing lives at https://tyler-r-kendrick.github.io/HoBo/llms.txt. --- # What is HoBo? HoBo is an **AI-native meta-cloud** built on one guiding principle: **every app deserves a home — preferably a cheap one.** You describe your app once, HoBo places it on real infrastructure that passes your policies, and then — this is the part nobody else does — **it keeps looking**. When a better-value home opens up, continuous migration moves your app there, without a rewrite, a war room, or a weekend. > **Honest status:** HoBo is **pre-release**, and the core loop already runs: pack a Bindle → `hobo plan` / `hobo up` / `hobo dev` against the default local lane, plus the `hobo-mcp` agent surface — with live deploy lanes credential-gated. See [the trail map](/HoBo/guide/status/) for what runs now. ## The pitch *"Write your app once. HoBo finds it the cheapest home that meets your standards — and moves it when somewhere better opens up. Built for humans and the agents that work for them."* A developer (or an agent) packs the app into a [**Bindle**](/HoBo/primitives/bindles/), runs `hobo up`, and the app lands in a [**Camp**](/HoBo/primitives/camps/) — inferred on the local lane by default, or pinned by topology you author. [**Yards**](/HoBo/primitives/yards/) select providers; camps describe environments. From then on, placement is a *standing decision*, not a historical one. ## Why "HoBo"? The name is the philosophy. A hobo travels light, works where the work is, and moves on when a town stops being worth the rent: - **Travel light.** Everything the app owns fits in its bindle. - **Work anywhere.** Ports and adapters mean any conforming provider is a valid home. - **Move when it's worth it.** Continuous migration re-plans placement for as long as the app lives. ## Where to go next | Path | Start here | |---|---| | **Guide** | [Getting started](/HoBo/guide/getting-started/) · [Core concepts](/HoBo/guide/concepts/) · [Trail map](/HoBo/guide/status/) · [CLI intro](/HoBo/guide/cli/) | | **Architecture** | [Overview](/HoBo/architecture/) · data / control planes · WASM · air-gap | | **Security** | [Identity](/HoBo/security/identity/) · multi-tenant · credentials | | **CI/CD** | [Continuous migration](/HoBo/cicd/migration/) · progressive delivery | | **Primitives** | Bindle, Camp, Yard, Junction, Station, Trail | | **Operations** | Built-in Yard operators (SRE, GitOps, …) | | **The Crew** | Custom ACP agents | | **Testing / Reference** | Full pattern and lookup manuals | --- # Getting started Time to leave home. This guide takes you from an empty directory to an app that has a home — and a standing ticket to a cheaper one. You'll meet HoBo's two foundational primitives, in the order you need them: 1. **[Bindle](/HoBo/primitives/bindles/)** — *pack.* Everything your app owns, rolled into one provider-agnostic bundle. The one thing you must write: no bindle, nothing to host. 2. **[Camp](/HoBo/primitives/camps/)** — *settle.* The environment shape your app occupies (prod/staging/…). Every app lives in a camp, but **writing one is optional** — leave it out and HoBo infers a camp. **Provider selection lives on the [Yard](/HoBo/primitives/yards/)**, not inside the camp. **[Yards](/HoBo/primitives/yards/)** and **[Junctions](/HoBo/primitives/junctions/)** are advanced scenarios — pin a provider, multi-region, or multi-cloud. Short section [at the end](#advanced--yards-and-junctions). > **Pre-release note:** `@hobo/*` is **not on npm**. Public `npx hobo` is a *different* package. The local lane (`collect` / `plan` / `up` / `dev`) runs **after** you clone this repo, install with pnpm, and build the workspace. Default IaC is still an **in-memory Pulumi stand-in**. Live Cloudflare/Vercel/Pulumi lanes are credential-gated. ## Prerequisites - **Node.js** ≥ 22 (`node:sqlite`; CI uses 24). TypeScript is the first reference implementation. - **pnpm** 10 (`packageManager` in the repo root). A GitHub zip without `external/ts-autocode` is incomplete — clone with submodules / the full tree. - No cloud account for the local lane. ## Run the local lane (clone — not npm) ```sh git clone --recurse-submodules https://github.com/Tyler-R-Kendrick/HoBo.git cd HoBo pnpm install pnpm hobo doctor --cwd examples/hello-bindle pnpm hobo plan --cwd examples/hello-bindle pnpm hobo up --cwd examples/hello-bindle pnpm hobo dev --cwd examples/hello-bindle # serves a live local URL ``` `pnpm hobo` is the supported invocation — it builds `@hobo/hobo` and `@hobo/adapter-hono` on first use, so there is no separate build step to forget. (`pnpm --filter @hobo/hobo exec hobo …` does **not** work: pnpm never links a package's own bin into its own `node_modules/.bin`.) Do **not** run `npx hobo`. Use the composed `@hobo/hobo` binary from this workspace via `pnpm hobo`. `hobo new` exists on that binary only; it is not a public npm scaffold. The same commands deploy the apps the [Examples console](/HoBo/examples/) shows — they share one catalog: ```sh pnpm hobo up --cwd examples/try-site/apps/hello-world pnpm hobo plan --cwd examples/try-site/apps/todo-api ``` ## Step 1 — Pack your bindle The bindle is the first thing HoBo looks for and the only thing it ships. It's plain code — no YAML sidecar, no config dialect: ```ts // src/bindle.ts import { bindle } from "@hobo/bindles"; import { app as api } from "./api"; import { app as web } from "./webapp"; export default bindle("my-app", (b) => { const db = b.data().allow("read", "write"); // a SQL-ish data port, least-privilege verbs inline const jobs = b.queue().allow("enqueue"); // background work const apiSvc = b .api(() => api.fetch) // HTTP API entrypoint .dependsOn(db, jobs); // typed bindings, injected at runtime const webSvc = b.web(() => web.fetch).dependsOn(apiSvc); return { db, jobs, api: apiSvc, web: webSvc }; }); ``` Three things to internalize — they're what make everything later cheap: - **Resource names come from the returned object keys.** `{ db, jobs, api, web }` is your app's topology; the object *is* the manifest. - **`.dependsOn(...)` is the wiring.** It declares edges *and* injects typed bindings at runtime. Nothing reaches a backend it didn't declare — deny by default. - **`.allow(...)` is inline least privilege.** Verbs are scoped to the resource's kind — `"read"`/`"write"` on a data port, `"enqueue"` on a queue, `"get"`/`"set"` on a kv store. Permissions travel *in* the bindle, so they move when the app moves. That's the trick: a bindle carries everything needed to unpack it anywhere, which is exactly what makes migrating to a cheaper home a routine event instead of a rewrite. That's a complete description. Against a **built** `@hobo/hobo` binary, `hobo up --cwd examples/hello-bindle` converges on the default **local lane** (offline, durable SQLite journal, in-memory Pulumi stand-in — not live multi-cloud). If a committed `hobo.bindle.json` is present it wins; otherwise the manifest is compiled from `src/bindle.ts`. > **Starting from nothing?** You don't need existing code to pack a bindle. Declare a `@component` class whose `@operation` methods carry an intent, a typed contract shape, and an optional rubric — a `stub()` body declares *code to be generated* — and hand it straight to `b.api(...)`. See [Greenfield: declare the bindle, generate the code](/HoBo/primitives/bindles/#greenfield-declare-the-bindle-generate-the-code). ## Step 2 — Camps (optional) and providers (on the Yard) A **camp** answers *what environment is this?* — not *which cloud account*. Camps declare env type, labels, and runtime **requirements**. They do **not** own the Bindle and do **not** select providers. ```ts // src/camps.ts — optional; omit entirely for inferred local camps import { camp } from "@hobo/camps"; export const prod = camp("prod", { envType: "prod", runtimeRequirements: ["runtime-local"], }); ``` **Provider selection belongs on the Yard** (advanced — day one can skip this and use the local lane): ```ts // src/yard.ts import { yard } from "@hobo/yards"; import { prod } from "./camps"; export default yard("aws-east", { provider: "aws", regions: ["us-east-1"], camps: [prod], }); ``` Converge: ```sh hobo plan --camp prod # dry run: placement, cost, warnings — read-only hobo up --camp prod # build, plan, gate, provision, deploy hobo logs --camp prod # live-tail through the camp runtime ``` Useful truths: - **The camp is a *current address*, not a vow.** The bindle stays provider-agnostic so the home can change. Continuous migration reuses plan → gate → converge. - **Writing a camp is optional.** No camp file → inferred local camp on the default lane. Explicit camps pin env semantics or satisfy locked-down placement policy. - **Zero-setup first deploy:** after the workspace build above, `hobo up` / `hobo dev` need no cloud credentials. `hobo login` mints a local dev session; `hobo claim` binds an anonymous deployment to an account later. **Live** lanes are credential-gated: Cloudflare (`CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID`), Vercel (`VERCEL_TOKEN`), or live IaC credentials. - **Provider-native power stays in the Bindle** as additive annotations (not a rewrite): ```ts import { aws } from "@hobo/provider-aws"; const apiSvc = b .api(() => api.fetch) .with(aws.with.fargate({ cpu: 512, memoryMiB: 1024 })); ``` **You now know everything required to run HoBo.** A bindle is the whole entry fee. Everything below is for apps with wanderlust. ## Advanced — Yards and Junctions Skip this on day one. Come back when you pin a provider or go multi-cloud. ### Yard: one provider, many camps ```ts import { camp } from "@hobo/camps"; import { yard } from "@hobo/yards"; export default yard("aws", { provider: "aws", regions: ["us-east-1", "us-west-2"], camps: [ camp("prod-east", { envType: "prod" }), camp("prod-west", { envType: "prod" }), ], }); ``` One yard per provider. Multi-region = several camps under one yard. Multi-cloud = several yards under a junction. ### Junction: many yards, many clouds > **Early-stage:** `junction()` authoring and plan derivation ship; each yard still realizes its own half. Full multi-cloud realization is on the [trail map](/HoBo/guide/status/). ```ts import { junction } from "@hobo/junctions"; import awsYard from "./yards/aws"; import azureYard from "./yards/azure"; export default junction("my-app", { yards: [awsYard, azureYard] }); ``` Requires **at least two** member yards. Continuous migration then has a bigger map; residency and sovereignty policy decide what may cross which border. ## Where to next - [Bindles](/HoBo/primitives/bindles/) and [Camps](/HoBo/primitives/camps/) — the full reference for the two primitives you just used. - Worked examples ship with the product docs and starter templates. - [Core concepts](/HoBo/guide/concepts/) — the vocabulary in one sitting. - [The hobo CLI](/HoBo/guide/cli/) — the whole verb surface (it's deliberately small). - [hobo-code](/HoBo/guide/hobo-code/) — talk to Bo after `hobo-code` starts the session. - [HoBo for agents](/HoBo/agents/) — pointing an AI agent at HoBo? It gets its own credentials, not yours. --- # Core concepts HoBo's metaphor is the itinerant worker who travels light, and the names lean into it — but every term has a precise technical meaning. Here's the whole vocabulary in one sitting. ## The four scopes Scopes nest by breadth: **`Bindle ⊂ Camp ⊂ Yard ⊂ Junction`**. The bindle is the one thing you must write; the camp is where your app always lives (declare one to pin placement, or let HoBo infer it from the code you've configured); the last two are for apps that go places. | Scope | What it is | |---|---| | **[Bindle](/HoBo/primitives/bindles/)** | The app, packed: services, resources, wiring, policies, and deployment intent. Provider-agnostic by construction — which is what makes it cheap to move. | | **[Camp](/HoBo/primitives/camps/)** | Environment shape (prod/staging/…) and runtime **requirements**. Does **not** own the Bindle or select the cloud provider. Durable runtime ports (EventStore, Queue, ActorHost, Clock) are provided by the Camp substrate. | | **[Yard](/HoBo/primitives/yards/)** | **Provider selection point** plus zero or more camps: cross-camp policy, identity, reconciliation, provider-global networking. One yard per provider. | | **[Junction](/HoBo/primitives/junctions/)** | Where yards meet: cross-yard **federation** for hybrid/multi-cloud. Peer yards, no shared mutable state — the final edge is federation, not containment. | Multi-region within a provider = several camps under one yard. Multi-cloud = several yards federated into a junction. ## App Kits (inside a Bindle) Inside a Bindle, resources split into two families: | Family | What it is | Examples | |---|---|---| | **[App Kit](/HoBo/primitives/app-kits/)** | Application surface that hosts entrypoints or interactive runtime behavior | `api`, `web`, `service`, `ui`, `websocket`, `job`, `wasm-component` | | **Infra Kit** | Port / infrastructure dependency — not an entrypoint host | `kv`, `bucket`, `queue`, `topic`, `secret`, `data`, … | **[Session Replay](/HoBo/reference/session-replay/)** is default-supplied on every App Kit (not only `web`). ## Continuous migration The guiding principle behind everything above: **placement is a standing decision, not a historical one.** Because the bindle is provider-agnostic and the camp is just its current address, HoBo's Dispatcher can keep re-planning placement for as long as the app lives — watching price, credits, reliability, and policy. When a better home clears the gates, the move runs through the same plan → gate → converge loop as any deploy. Migration isn't an emergency procedure in HoBo; it's a lifestyle. - **Policy-gated:** an inferred or migrated placement is approved by policy, not automatically — your platform decides how much wanderlust to allow. - **Reversible:** every move is a sequence of events; rolling back a migration is replaying the decision, not archaeology. - **Cost-directed:** the default objective is simple and honest — the cheapest home that meets your standards. ## WIT/WASM ports The portable contract between app code, runtime ports, and provider adapters is **WIT** (the WebAssembly Component Model's interface language, hosted on wasmCloud by default). App Kits sit above that contract: - **Ports are WIT interfaces** — storage, messaging, HTTP, identity, and more. - **Apps are wasm components** (with an OCI-container fallback for anything wasm can't host yet). - **Providers are capability providers** linked to component imports — deny-by-default: a component reaches a backend only via a link the manifest declared. - **Any Component-Model language** can author against the same interfaces. TypeScript is the first reference implementation, not a requirement. That thin, typed port contract is what makes continuous migration economical: moving homes means re-linking ports to a new provider's adapters, not rewriting the app. App Kits *compose* those ports into Bindle-declared surfaces you author day to day. ## Ports and adapters, everywhere Cloud providers, IaC engines (Pulumi, Crossplane, Terraform/OpenTofu), build/VM tools (Dagger, Packer, Firecracker), identity systems, and AI optimizers all sit behind replaceable ports. Two consequences: - **Conformance is the contract.** Every port ships a contract test suite; any adapter — human- or AI-written — must pass it to be certified as a valid home. - **Patchable, not just portable.** You can swap any engine, add any adapter, fork the Yard, and persist AI/human patches, with conformance as the trust bridge. ## The replayable event log Control-plane state, projections, SRE workflows, and audit trails all derive from an **append-only event log**. Commands mutate state only by appending events (CQRS/event sourcing), which buys: - **Recovery:** a crash anywhere — mid-deploy, mid-migration — is recovered by replaying the stream. - **Audit:** every state transition, including *agent token issuance* and *every migration decision*, is a durable, attributable fact. - **Time travel:** projections can be rebuilt at any point in history. "Where did this app live in March, and why did it move?" is a query, not a forensic investigation. The default durable Camp EventStore is **SQLite behind ports** (one partition, one active writer; scale by partitions). Actors append under optimistic sequence **and** a fencing token so a stale holder cannot win after lease transfer. Catch-up projections use database-local positions; live callbacks are an optimization, not the only delivery path. Queue/timer/outbox side effects can stage in the **same** durable transaction as domain facts, while network dispatch happens outside that transaction. ## Contract effects (what a boundary can do) Contracts are not only request/response shapes. A boundary can declare an **EffectSpec**: purity (pure vs varies vs transacts…), capability tags (may fail, may suspend…), and domain I/O (network, storage, identity, policy eval, agent tools…). Pure contracts isolate for unit tests with zero infrastructure; domain effects drive automatic harness mocks. When a revision **widens** effects (for example pure → network), can-deploy treats that as a potential breaking change for consumers that assumed a narrower set. ## Visible resiliency (secondary layer) Operations that face flaky I/O may carry a **ResilienceProfile** — deterministic retry schedules, timeouts, circuit-breakers, and related strategies — as **contract-visible metadata**, not a new top-level control-plane noun. Application emits attempt/terminal facts for replay. Hard rules stay fixed: `policy.Decided` denials and typed terminal failures **never** retry; cancellation stops pending work; inspection is reconstructed from the event log, never process memory. ## Agent-native identity AI agents are **first-class principals**, not scripts holding a human's API key: - Workload identity (SVID/OIDC) with **brokered short-lived tokens** — secretless-first, no static credentials. - A machine-readable **capability standard** so agents can self-provision exactly the credentials a task needs. - Policy checks at every phase, and traceable actions in the same event log as everything else. - An MCP control surface alongside the SDK and CLI, so the same capabilities are reachable by humans and agents alike. ## Where the details live - [Bindles](/HoBo/primitives/bindles/) · [App Kits](/HoBo/primitives/app-kits/) · [Camps](/HoBo/primitives/camps/) · [Yards](/HoBo/primitives/yards/) · [Junctions](/HoBo/primitives/junctions/) — the primitive references. - [Session Replay](/HoBo/reference/session-replay/) — default on every App Kit. - [Architecture](/HoBo/architecture/) — the machinery that does the describing, placing, and moving. - [HoBo for agents](/HoBo/agents/) — the agent identity model and MCP surface. --- # The `hobo` CLI The CLI is deliberately small — a hobo carries one stick, not a toolbox. **`up` and `down` are the only lifecycle verbs**: ship, promote, roll back, migrate, and even self-host bootstrap are flags on the same convergence loop, not separate commands. Every documented example on this site sticks to this canonical surface. ## Lifecycle ```bash hobo up hobo up --camp prod hobo up --camp prod --to hobo up --rollback hobo up ./platform --seed hobo down --camp pr-123 ``` `up` realizes the current object graph: build, plan, gate, provision, deploy. `down` tears it back in reverse dependency order. `up` always reruns the applicable gate before mutation — a stale preflight can never authorize a deploy. Continuous migrations run this same loop; there is no separate `migrate` verb. ## The full verb surface | Verb | Role | Semantics | |---|---|---| | `up` / `down` | lifecycle | Realize / tear down the object graph. | | `dev` | local loop | Offline local-runtime camp; serve api/web handlers in-process. | | `new` | local loop | Scaffold a runnable SDK program. | | `login` / `claim` | identity | Dev identity (OIDC/SVID path); bind an anonymous deployment to an account. | | `connection` | identity | Durable Connections (create / authorize / list / get / revoke / rotate). | | `identity` | identity | Continuous identity migration status / inspect (zero Bindle config). | | `access` | identity | Elevated / cross-env access or break-glass (short-lived, audited). | | `feature` | delivery | OpenFeature flags / canary / experiment (`list` \| `eval` \| `set` \| `kill` …). | | `deploy` | delivery | Attested continuous-delivery **status / logs / rollback** (not a second lifecycle). | | `incident` | delivery | CD failure disposition: A2H handoff or sandboxed self-heal. | | `plan` | read-only | Dry-run the `up` decision: plan, matching, cost, warnings. | | `verify` | read-only | Unified Gatekeeper preflight. | | `can-deploy` | read-only | Producer/consumer compatibility matrix. | | `inspect` | read-only | Projections and time-travel views (`run`, `deploy`, `topology`, …). | | `logs` | read-only | Live-tail Camp streams. | | `doctor` | diagnostics | Environment, credential, connectivity, and portability hazards. | | `messaging` | diagnostics | Messaging capability + delivery-guarantee readiness. | | `codegen` / `build` / `collect` / `stub` | authoring | Bindings, wasm components, manifest IR, contract-backed stubs. | | `drift` | diagnostics | Drift report + governed remediation paths. | | `add` / `registry` | distribution | Install providers after sig/provenance/conformance; registry admin. | | `import` | migration | Adopt an existing estate in place — never destroy/recreate on import. | | `generate` | generation | Request the provider-generation loop for a missing backend. | | `sandbox` / `test` | test loop | Isolated lanes; out-of-scope boundaries served from contracts. | | `attest` / `erase` | governance | Signed evidence exports; crypto-shred with projection rebuild. | | `broker` | governance | Conductor resolve / config / matrix (placement plane). | | `yard` | control plane | Install and inspect the default bare-metal Yard. | | `station` | control plane | Station ops/setup (`init` \| `up` \| `down` \| `status` \| `rails *`) — not lifecycle. | | `serve` | control plane | Local control-plane HTTP+SSE for the Yard game UI. | | `replay` | observability | Session Replay list/get/open/summarize/retention/doctor. | | `derivative` | extensibility | Manage derived HoBo implementations. | | `runbook` | SRE | Execute or replay incident runbooks under policy. | ## Not lifecycle (and not missing) Where other tools invent ceremony commands, HoBo folds them into the model: - **Lifecycle shipping** is only `up` / `down` (plus flags). A separate “deploy command” as *lifecycle* is rejected. - **`deploy` as a verb still exists** — it is the continuous-delivery *observe* surface (`status` / `logs` / `rollback`), not a second ship path. - `destroy`, `teardown`, `rollback`, and `promote` as standalone commands are rejected; use `down` or `up --rollback` / promotion via convergence. - `train`, `crew`, and `fleet` are SDK code and projections, not CLI verbs. ## Status The **canonical verb set** is on the composed `hobo` surface today and maps **1:1** to [`hobo-mcp`](/HoBo/agents/mcp/) tools (parity is contract-tested). Lifecycle is `up` / `down`; `hobo deploy` is **observe-only** delivery status, not the ship verb. Interactive authoring is [`hobo-code`](/HoBo/guide/hobo-code/). The default path is the **local lane**: offline Yard runtime, durable SQLite journal, no cloud account. Live deploy lanes are credential-gated (Cloudflare, Vercel, or live IaC credentials). Some advanced flags (`--to`, `--rollback`, `--seed`) and deeper multi-cloud depth are still maturing — see [the trail map](/HoBo/guide/status/). Global flags that work everywhere: `--json`, `--camp`, `--cwd`, `--verbose`, `-q`, `--help`. --- # hobo-code `hobo-code` is the **interactive coding session** (Bo). `hobo` is the **control-plane CLI** (`up` / `down` / `plan` / `dev` and the rest of the canonical verb set). They are different binaries. `@hobo/*` is **not on npm**. Do not run `npx hobo` or invent `npx hobo-code`. After you clone this repo: ```sh pnpm install pnpm exec turbo run build --filter @hobo/adapter-hono --filter @hobo/code --filter @hobo/hobo pnpm --filter @hobo/code exec hobo-code ``` That starts the session. From there, **talk** for hello-world / run / debug. Zero-LLM pattern lookup is explicit: `hobo-code template …` or `/template web frontend backed by an api` (ADR-0153). Ordinary chat is not silently intercepted for arbitrary queries. ## What belongs where | You want | Use | |---|---| | Plan / converge / tear down an existing Bindle | `hobo plan` / `hobo up` / `hobo down` / `hobo dev` | | Author or extend an app by talking | `hobo-code` session | | Search/render a pattern with no model | `hobo-code template` or `/template` | | Agent tool calls over the same verbs | `hobo-mcp` — see [the MCP surface](/HoBo/agents/mcp/) | ## Honest limits - Workspace build first. A half-built tree cannot import `@hobo/adapter-hono`. - Default apply is still the in-memory Pulumi stand-in. Live lanes are credential-gated. - Known intents fill templates. Implementation holes marked `"use training"` stay holes until training exists. Next: [Getting started](/HoBo/guide/getting-started/) · [The hobo CLI](/HoBo/guide/cli/) · [Try HoBo](/HoBo/examples/). --- # The trail map Where is HoBo on the road? Here's the honest itinerary. | Area | State | |---|---| | **Product** | **Pre-release.** Not accepting tenants; `@hobo/*` packages are **not published to npm** yet. | | **Developer surface** | SDK primitives, the **canonical `hobo` verb set** (source: `packages/cli/src/shell/verbs.ts`), and the MCP surface documented here are **contract-pinned**: specified, validated, and stable to learn against. Product binary: `@hobo/hobo`; shell/factories: `@hobo/cli`. Interactive authoring is [`hobo-code`](/HoBo/guide/hobo-code/), a different binary. | | **Implementation** | Progressing through staged build gates (below). Default deploy is an **in-memory Pulumi stand-in**, not live multi-cloud apply. See "What runs today" and ship targets. | | **This site** | The canonical public developer documentation — for humans and agents. | Ship targets stay **pre-release**: OSS public docs (this site), local-first developer tooling after a workspace build, hosted beta later. Default apply is an in-memory Pulumi stand-in. See [Getting started](/HoBo/guide/getting-started/) and [Architecture](/HoBo/architecture/) for what a visitor can run without a private repo checkout. ## What runs today - **The TypeScript bindle flow.** After a workspace build of `@hobo/hobo` and `@hobo/adapter-hono`, a project whose `src/bindle.ts` default-exports a `bindle()` runs `hobo collect`, `hobo plan`, `hobo up`, `hobo down`, and `hobo dev` on the default **local lane** — no cloud account. `@hobo/*` is not on npm (`npx hobo` is the wrong package). Default IaC is an **in-memory Pulumi stand-in** (`HONEST SCOPE` in `@hobo/hobo` compose); a successful `up` does **not** imply real infrastructure was provisioned. A half-built tree used to look like “no Bindle”; import errors now surface. `hobo dev` serves the project's own api/web entrypoint handler in-process; a committed `hobo.bindle.json` manifest artifact wins when present. - **Durable local EventStore.** The Camp EventStore defaults to a **single-owner SQLite partition** (`@hobo/storage-sqlite` + `@hobo/runtime-local`): stream-head CAS, actor fencing, positioned catch-up, durable subscription checkpoints, and atomic domain events with staged queue/timer/outbox effects. Browser hosts use `@hobo/runtime-browser` with OPFS when available (fail-closed when durable is required but OPFS is not). - **Contract effects + visible resiliency.** `@hobo/contracts` BoundaryContracts may declare **EffectSpec** and **ResilienceProfile**; can-deploy treats effect widening as breaking when declared; Gatekeeper surfaces effect summaries; `@hobo/contract-harness` isolates pure contracts and runs never-retry fault injection. No Effect-TS/Polly/XState public dependency. - **Station + pure Conductor.** `@hobo/stations` + `@hobo/broker` — control-plane isolation, Rails, receipts pin `stationId` + `railHash`. - **Trails + continuous identity.** `@hobo/trails` recipes and hands-off dual-path identity continuous (`@hobo/identity`); pure identity continuous needs zero Trail authoring. **`hobo login` defaults to fixture/local session material**, not production OIDC/SPIRE. - **Live migration & upgrade executors.** `@hobo/trails` ships upgrade preflight/compatibility evidence, a real expand→backfill→contract executor over SQLite (snapshot-before-mutation, compat-gated destructive contract), pooled→siloed zero-loss tenant moves (fence → drain → export/import → checksum-gated reap), and CDC data-dual ports. The `migration.*` event bodies are pinned in `@hobo/events`' append-only golden-replay corpus with an upcaster-totality gate. - **Budget-driven routing.** `@hobo/dispatcher` (deterministic credit/budget selection, `dispatch.Decided`), `@hobo/broker` budget threshold signals, and the `@hobo/stations` closed routing control loop + progressive health-gated cutover. `pnpm run staging:cutover` rehearses a zero-downtime backend swap on a loopback lane (archived evidence); Vercel/Cloudflare lanes are credential-gated and fail closed. - **Jurisdiction & residency.** Typed legal geography in `@hobo/core` (region→jurisdiction catalog, per-tenant residency constraints, fail-closed `evaluateResidency`) with a residency Gatekeeper gate, a residency policy pack, and OSCAL-aligned control→evidence packs in `@hobo/policy`. - **Thin SaaS providers.** `@hobo/provider-convex` — honestly-scoped capability classes, typed impedance limits, credential-gated Management API (fails closed without a broker-minted key). - **Harness progressive delivery.** OpenFeature-shaped flag evaluation, delivery, and experiments in `@hobo/identity`; CD apply plane in `@hobo/cloud`. - **Connectivity foundations.** `@hobo/connectivity` — hierarchical naming (never bare `.hobo`), overlay/DNS, secretless connectors, env isolation, PAM hooks. - **Messaging.** Unified `topic()` / messaging plane with Kafka and optional Dapr adapters; default remains the local in-process path. - **Component host scaffolds.** `@hobo/runtime-wasm` (Wasmtime), `@hobo/runtime-oci` (kubectl transport), `@hobo/runtime-cloud-native` (mock default + a **real credential-gated Cloudflare provider** proving the remote FetchHandler contract; aws/gcp/azure/vercel stay declared-unimplemented fail-closed). See [Component hosts](/HoBo/reference/component-hosts/). - **Componentize honesty.** `@hobo/componentize` has two lanes: the offline default emits **synthetic wasm contract bytes** (`buildWasmContractBytes`) for digest/boundary proofs, and `HOBO_COMPONENTIZE_REAL=1` drives the **real jco/`componentize-js` pipeline** (toolchain-gated, fail-closed — never a silent synthetic fallback) producing genuine component-model binaries. - **Session Replay on every App Kit.** `@hobo/replay` + default annotation on all App Kits (`api`, `web`, `service`, `ui`, `websocket`, `job`, `wasm-component`) — not only `web` (ADR-0137). Prefer **App Kit** over "membrane type" at Bindle authoring; WIT membrane stays ADR-0069. Local dual fixtures default-on; `hobo dev` wraps HTTP Kits with bootstrap + `/__hobo/replay/*`; production flag/sample gated; `hobo replay *`. See [Session Replay](/HoBo/reference/session-replay/) · [App Kits](/HoBo/primitives/app-kits/). - **The MCP control surface.** The `hobo-mcp` binary (`@hobo/hobo`) serves the **canonical verb set** as MCP tools over stdio, dispatching through the same command registry as the CLI, with verb/tool parity contract-tested, deny-by-default auth, and audited decisions. See [the MCP surface](/HoBo/agents/mcp/). - **Login.** `hobo login` runs a real RFC 8628 device-code grant when an issuer is configured (`HOBO_OIDC_ISSUER` → OIDC discovery → device authorization → token polling → JWKS verification), fail-closed with typed remediation; the offline fixture session remains the no-issuer default. `hobo verify` runs the ADR-0146 residency dimension from environment-supplied tenant constraints (`HOBO_RESIDENCY_ALLOW` et al) through the policy layer. - **Credential-gated live lanes.** Cloudflare and Vercel deploys unlock via env tokens; the real pinned Pulumi engine unlocks via `HOBO_PULUMI_LIVE=1`. These are opt-in; they are not the default offline path. - **Examples in the repo:** `hello-bindle` (`hobo up` after adapter build), `multi-cloud-shop` (local Yard kv+queue), `durable-agent` (**tests only**, no Bindle), `self-host-yard` (in-process Yard). Pages [Examples](/HoBo/examples/) ship local-bridge apps; browser-WIT hello-world needs local `astro dev`. - **Not yet shipped / design-stage:** npm registry publish, full certified multi-cloud provider adapters (fly/aws/azure/gcp production apply paths), Junction package maturity toward S4, production OIDC/SPIRE defaults, and alternate EventStore engines (`runtime-postgres` / Temporal / Restate / Cloudflare DO). ## How the build is staged One leg at a time: - **Local toolchain** — hello Bindle through CI with guards green. - **Local fidelity** — offline `hobo dev`, replay equals live. - **First cloud + Yard** — first provider and drift policy. - **Agent autonomy** — scoped agent auth and operator hooks. - **Multi-cloud** — overlays, placement across providers, Junctions. - **Extensibility** — provider generation and forge hooks. - **Production hardening** — HA/DR, canary, FinOps. ## What “contract-pinned” means Public surfaces — SDK signatures, CLI verbs, event families — are pinned by executable contracts and conformance suites. Documented examples track the canonical CLI. When this site changes, it is because the **product contract** changed, not because marketing drifted. ## Following along - [Getting started](/HoBo/guide/getting-started/) - [Core concepts](/HoBo/guide/concepts/) - [Agent docs](/HoBo/agents/) and [`llms.txt`](/HoBo/llms.txt) --- # Architecture ## In this section | Page | Topic | |---|---| | [Data plane](/HoBo/architecture/data-plane/) | Event log, aggregates, projections, SQLite default | | [Control plane](/HoBo/architecture/control-plane/) | Yard: plan → gate → converge | | [WASM & WIT surface](/HoBo/architecture/wasm/) | Component Model ports & hosts | | [Air-gapped & sovereign](/HoBo/architecture/air-gapped/) | Zero-egress profiles | A condensed tour of the machinery that packs, places, and — crucially — *keeps re-placing* your app. ## The one-paragraph model A developer (or agent) writes a **Bindle** (app topology) and optionally `camp()`/`yard()`/`junction()` scope objects mapping ports to providers. The **Yard** — HoBo's provider-specific control plane — receives a deploy request; the **Planner** turns Bindle + mapping into a **DeploymentPlan** (a DAG of resource plans); the **Orchestrator** drives each resource to reality through **ResourceActors** that call IaC/build adapters inside sandboxed **Executors**, brokering short-lived credentials from **Identity**. Every state transition is an event in an append-only log, so the whole system is replayable, auditable, and recoverable. The app lands in a **Camp**, which also supplies the durable runtime the app's actors and workflows run on. And then the **Dispatcher** keeps watching: when a cheaper valid home appears, the same pipeline runs again as a migration. ## Control plane vs data plane Keeping these separate is a core architectural commitment: - **Control plane (the Yard)** decides *what infrastructure should exist* and makes it exist. Event-sourced, actor-based, multi-tenant — and **never on the critical path** of a running app's request. If the Yard is down, deployed apps keep serving. (They just can't move that day.) - **Data plane (the running Bindle)** is your actual app plus the Camp's durable runtime (event store, queues, streams, actors, clock). It uses HoBo's runtime SDK primitives but does not depend on the Yard at request time. ## Control-plane components | Component | Responsibility | Emits | |---|---|---| | **Gateway** | Authn/z; SDK RPC + REST + MCP; accept commands; serve read models | `DeployRequested`, … | | **Identity** | Workload identities (SVIDs); OIDC; broker provider tokens | `IdentityIssued`, `TokenGranted` | | **Registry** | Catalog of providers + their Capability Manifests | `ProviderRegistered` | | **Planner** | Resolve mappings → `DeploymentPlan` DAG; capability/region/cost constraints | `Planned`, `PlanRejected` | | **Broker** (branded "the Conductor") | Policy-governed placement decisions across providers; wraps the policy/contract/billing/delivery/connector planes ([details](/HoBo/primitives/stations/#the-conductor)) | `broker.Resolved`, `broker.Rejected` | | **Dispatcher** | Continuous placement: watch cost/policy/reliability signals; propose policy-gated migrations to better homes | `MigrationPlanned`, … | | **Orchestrator** | One ResourceActor per resource reconciling desired↔actual; sagas for ordering & compensation | `Provisioning`, `Provisioned`, `Failed`, `Deployed` | | **Executors** | Run IaC apply/preview/destroy + builds in sandboxes with scoped, short-lived credentials | `ExecStarted`, `ExecSucceeded`, `ExecFailed` | | **Event Store** | Append-only source of truth; durable subscriptions; snapshots (default engine: single-owner SQLite partition behind ports) | *(is the log)* | | **Read models** | Materialized projections for queries and dashboards | *(query API)* | Two runtime/IfC additions: the **wasm host** (runs components; holds the link table binding each import to a capability provider) and the **Collector** (runs components in resource-collection mode to produce the manifest IR the Planner consumes). ## The deploy path (and the migration path — same path) ```text hobo up → Gateway authenticates the principal → DeployRequested → Planner solves the plan → Planned → Orchestrator spawns ResourceActors → Provisioning … Provisioned → (failure? sagas compensate) → Failed → Compensating → DAG completes → Deployed …and later, unprompted: Dispatcher spots a cheaper valid camp → MigrationPlanned → policy gate approves (or a human does) → the exact same convergence loop → Migrated ``` A crash anywhere is recovered by replaying the deployment's event stream. Inspection (`hobo inspect run `) reads projections and never touches providers. There is deliberately no separate "migration engine" to trust — if you trust `hobo up`, you already trust a move. ## Key invariants 1. **Commands mutate only via events.** No component writes resource state except by appending an event. 2. **Executors are the only thing that touches a cloud** — always sandboxed, always with a short-lived scoped token. 3. **The Yard is itself a Bindle**, deployed by an earlier Yard or bootstrapped locally (self-host = hosted). 4. **Same code, many Camps.** Swapping a Camp's runtime backend changes durability backing, never app logic — that's what keeps moves cheap. 5. **Deny-by-default linking.** A deployed component reaches a backend only via a link the manifest declared. ## Capability layers Public surfaces stack like this — adapters plug in without rewriting apps: | Layer | What you work with | |---|---| | **Authoring** | Bindles, App Kits, Camps, Yards, Junctions | | **Contracts & policy** | Boundary contracts, Gatekeeper, policy packs, effects/resiliency intent | | **Runtime ports** | EventStore, Queue, Messaging, Streamer, ActorHost, Clock | | **Hosts & engines** | Local/browser journals, Wasmtime, OCI/Kubernetes, optional cloud-native hosts | | **Control** | Stations, Conductor, Trails, progressive delivery (Harness) | | **Surfaces** | `hobo` CLI, MCP, identity, connectivity | See the [capability map](/HoBo/reference/packages/) for the product-facing overview and the [trail map](/HoBo/guide/status/) for what ships today. ### Durable event log default The Camp durable EventStore defaults to **SQLite**, with one database file (**partition**) owned by one active writer. Scale by partitions — not multi-writer access over shared network storage. SQLite is an **engine**, not the portability boundary: the same EventStore ports and storage capability can be implemented by other stores later. Stream revision uses a stream head (not a full scan), actor **fencing tokens** reach storage, and domain events can commit atomically with staged queue/timer/outbox work. Store-local **positions** support catch-up without becoming domain identity. ### Contract effects + visible resiliency Boundary contracts may declare optional **effects** (pure vs network/storage/…) and optional **resiliency profiles** (deterministic retry, timeouts, circuit-breakers). Effect widening can break can-deploy when consumers assumed purity. Resiliency stays **secondary** — it is not a top-level control-plane noun — and policy denials / terminal failures **never** enter a retry schedule. --- # Data plane The data plane is where **facts live**. Control plane (Yard) decides infrastructure; data plane runs your app and the Camp’s durable runtime. If the Yard is down, camps keep serving — they just cannot re-place that day. ## Why event sourcing is the foundation | Benefit | What you get | |---|---| | **Recovery** | Crash mid-deploy or mid-workflow → replay rebuilds exact state | | **Audit** | Every infra change and credential issuance is an ordered fact | | **Time-travel** | Reconstruct as-of; diff two points (`hobo inspect`) | | **Determinism** | Durable workflows replay over cached step results | ## One log, two folds | Fold | Purpose | |---|---| | **Write model (aggregate)** | Authoritative state for invariant checks inside a single-writer actor | | **Read model (projection)** | Rebuildable views for queries, dashboards, reactive UI | Both fold the **same append-only event log**. Snapshots are **caches**, never sources of truth. Snapshot ahead of head → hard error (`SNAPSHOT_AHEAD`). ## Commands → past-tense facts 1. Commands enter at a boundary (HTTP, MCP, workflow step) — they are **not** stored as events. 2. Aggregate `decide` runs pure business rules against folded state. 3. Aggregate `emit` past-tense facts. 4. Append uses **`expectedSeq`** (optimistic concurrency) and optional **`fenceToken`** (lease fencing). 5. Optional **`stageEffect`** stages queue/timer/outbox work for the same durable transaction. 6. After commit, live subscribers wake; network dispatch of staged work is at-least-once. ## Default durable store Camp EventStore defaults to a **single-owner SQLite partition** behind ports: | Property | Detail | |---|---| | Stream head | Revision + `owner_epoch` on a head row — not `COUNT(*)` | | Positions | Database-local `bigint` for catch-up (`EventPosition`) | | Checkpoints | Durable subscription positions | | Atomic effects | Domain events + staged work in one commit | | Scale model | Add partitions; not multi-writer NFS WAL | Details: [EventStore reference](/HoBo/reference/eventstore/) · [Durability tests](/HoBo/testing/durability/). ## Reactivity Default reads are **subscriptions** over materializations (**replay-then-live**). GraphQL facades and point lookups are escape hatches over the same models. Session consistency is the usual default; stronger/weaker levels are entity-configurable. ## Upcasters & quarantine Event bodies evolve via forward-only upcasters. Poison events produce `*.Quarantined` facts so folds skip deterministically instead of wedging forever. Crypto-shred / erase rebuilds projections without rewriting history bytes in place. ## What is *not* the event log | Data | Home | |---|---| | Streamer **frame bodies** | Streamer journal (tokens, logs, progressive UI) | | Authoritative-external provider DBs | Explicit escape hatch with staleness labels | | Blob/object payloads | Bucket resources — store refs in events | ## Multi-aggregate work Cross-aggregate operations are **eventually consistent** by design. Use sagas/choreography (events library) and correlation ids — do not invent a distributed transaction across aggregate streams. ## Related [Architecture](/HoBo/architecture/) · [Control plane](/HoBo/architecture/control-plane/) · [Aggregates testing](/HoBo/testing/aggregates/) · [Runtime ports](/HoBo/reference/runtime-ports/) --- # Control plane The **Yard** is HoBo’s provider-specific control plane: it decides *what infrastructure should exist* and drives it to reality. It is **never** on the critical path of a live user request to your app. ## Components | Component | Job | |---|---| | **Gateway** | Authn/z; SDK / REST / MCP ingress | | **Identity** | Workload SVIDs, OIDC, brokered short-lived provider tokens | | **Registry** | Provider + capability catalog; certification records | | **Planner** | Bindle + mapping → DeploymentPlan DAG; Matching | | **Conductor (Broker)** | Policy-governed multi-provider placement | | **Dispatcher** | Continuous re-placement (cost/policy/reliability) | | **Orchestrator** | ResourceActors reconcile desired ↔ actual; sagas compensate | | **Executors** | Sandboxed IaC/build with scoped credentials | | **Event Store** | Append-only control-plane truth | | **Read models** | Dashboards, inspect, status | ## Same path for deploy and migrate `hobo up` and continuous migration share **plan → gate → converge**. There is no separate “migration engine” to trust. Dispatcher re-enters the same loop when a better home opens. ## Gates before reality Nothing mutates cloud state until Gatekeeper (and friends) allow: 1. Contract compatibility (can-deploy) 2. Policy packs + obligations 3. Cost / quota 4. Golden suite (when enabled) 5. Optional secretless-config / can-migrate / can-trail / effects visibility Machine-readable reasons ship on `gatekeeper.Decided` for agents and humans. ## ResourceActors & choreography One ResourceActor per resource owns reconciliation. Independent subtrees provision concurrently. Failure past threshold choreographs rollback via events — not a single brittle coordinator thread. See [Matching & ResourceActors](/HoBo/reference/matching/). ## Executor sandbox - Firecracker-class isolation; **no ambient cloud creds** - Short-lived capability-scoped tokens per resource + action - Stdout via Streamer; IaC state per resource backend - Concurrency bounded per tenant and provider ## Self-host = hosted The Yard is itself a Bindle. Bootstrapping a self-host Yard is the same lifecycle as any other app. Air-gapped profiles use signed offline bundles and in-boundary substitutes — see [Air-gapped](/HoBo/architecture/air-gapped/). ## Stations A Conductor occupies a **Station** (hard isolation boundary). App topology (Bindle ⊂ Camp ⊂ Yard ⊂ Junction) is separate. Missing Station fails closed for control-plane work. ## Related [Conductor](/HoBo/primitives/stations/#the-conductor) · [Identity](/HoBo/security/identity/) · [Stations](/HoBo/primitives/stations/) · [Matching](/HoBo/reference/matching/) --- # WASM & WIT surface HoBo’s portability contract is **WIT/WASM** (Component Model / wasmCloud-shaped). TypeScript is the first reference implementation language, not a requirement for the architecture. **[App Kits](/HoBo/primitives/app-kits/)** (Bindle-declared `api`/`web`/`service`/… surfaces) compose those ports at authoring time. WIT interfaces are the capability boundary the hosts and providers implement. ## Why WIT | Property | Benefit | |---|---| | Typed capability imports | Deny-by-default sandbox is structural | | Host-supplied storage | Engines swap without guest SQL | | OCI fallback | Pragmatic packaging where components are not ready | | Same Bindle, N runtimes | Conformance-tested portability | ## Storage capability Guests use `hobo:storage@0.1.0` — **not** filesystem paths. Hosts bind SQLite partitions or other engines. See [WIT & storage](/HoBo/reference/wit-storage/). ## Messaging capability Publish/subscribe is **the messaging port** (`topic()`). Hosts may bind: - Kafka / Redpanda / MSK via Kafka provider - Dapr pub/sub via optional optional Dapr runtime (Camp opt-in) - Local in-memory for offline tests App code never imports Kafka or Dapr clients. See [Messaging, Kafka & Dapr](/HoBo/reference/messaging/). ## Build `hobo build` compiles services to components; generated drift fails closed. `hobo stub` keeps contract-backed stand-ins while implementation is generated. ## Isolation dimension WASM capability security sits beside data / compute / credentials / network isolation. Compromised guest code should not inherit ambient cloud credentials. ## Related [Architecture](/HoBo/architecture/) · [Messaging, Kafka & Dapr](/HoBo/reference/messaging/) · [Capability map](/HoBo/reference/packages/) · [Security](/HoBo/security/) --- # Air-gapped & sovereign HoBo supports a **zero-egress** Yard profile for sovereign and regulated deployments. Deploy / match / migrate / route must work **without phone-home**. ## Capability substitutes | Capability | Connected | Air-gapped substitute | |---|---|---| | Registry | Crawl capability manifests | **Signed offline bundle import** | | Images | Public pull | Mirrored in-boundary OCI | | TLS | Public ACME | Internal CA / private ACME | | Identity / SVID | Yard-issued | Unchanged (already in-boundary) | | Secrets | Provider KMS | In-boundary KMS/HSM | | Embeddings | Hosted API | Local model or BYO vectors | | AI generation | Hosted LLM | Local LLM or human-authored only | | Telemetry | 3p sinks | In-boundary OTel collector | ## Rules 1. Registry refuses unsigned/unproven plugins even offline — trust moves to the **bundle signer**, not to “files on a USB.” 2. No feature may require egress for core deploy/migrate. 3. Metering and audit stay in-boundary; export is operator-controlled. 4. Provider free-tier games still forbidden — sovereign is not a loophole for multi-account free tiers. ## CLI `hobo registry` offline bundle import; `hobo doctor` should report egress assumptions. ## Related [Security](/HoBo/security/) · [Control plane](/HoBo/architecture/control-plane/) · [CLI](/HoBo/reference/cli/) --- # Security HoBo is **secure-by-default, zero-trust, defense-in-depth**. This section is the home for identity, access, tenancy, and network isolation — not scattered under Guide. ## In this section | Page | Topic | |---|---| | [Identity & auth](/HoBo/security/identity/) | Workload SVIDs, brokered tokens, continuous identity, secretless-first | | [Multi-tenant & policy](/HoBo/security/multi-tenant/) | Deny-by-default, lanes, guardrails, provider-terms hygiene | | [Connectivity & networking](/HoBo/security/connectivity/) | Named endpoints, overlay, secretless connectors, PAM | | [Agent credentials](/HoBo/security/credentials/) | Scoped short-lived agent credentials — no ambient admin | | [Config & secrets](/HoBo/reference/config-secrets/) | Secretless resolution ladder (reference) | | [Policy plane](/HoBo/reference/policy/) | Packs, obligations, `policy.Decided` (reference) | ## Related [Architecture](/HoBo/architecture/) · [Roles / DevSecOps](/HoBo/roles/devsecops/) · [Troubleshooting](/HoBo/troubleshooting/) --- # Identity & auth HoBo is **secretless-first**. Long-lived static cloud keys are the residual path, not the default. ## Principals | Principal | Notes | |---|---| | Humans | Org RBAC roles; OIDC/SSO | | Workloads | SVID/OIDC per Camp/service | | Agents (users) | First-class principals with scoped, short-lived credentials | | Crew operators | Built-in Yard agents under the same model | ## Brokered credentials 1. Task declares needed capabilities. 2. Policy decides allow / deny / mutate / require-approval. 3. Identity brokers a **short-lived, scoped** token to the provider. 4. `CredentialAudit` records access **by reference**, never plaintext secrets in the log. Confused-deputy and runaway spend are mitigated by scoped SVIDs, policy gates, spend caps, and full replay. ## Continuous identity migration When identity providers or principal shapes change: - Dual-path continuity is Matching/policy-driven. - **Zero required Bindle config** for pure identity continuous. - Dual fixtures extend offline fixtures. - Observe with `hobo identity status` / `hobo doctor`. - Do not invent a parallel IdP DSL or Bindle-only migration product. Trails may *compose* `identity.continuous`; pure identity continuous needs no Trail authoring. ## Secrets residual path `secret(name)` / `hobo:secrets` remain for legacy-compatible and residual classical credentials. Prefer `fromBinding`, bound resources, and federation. Config resolution ranks secretless sources first — [Config & secrets](/HoBo/reference/config-secrets/). ## Multi-tenant context `HOBO_TENANT` / `HOBO_PROJECT` / `HOBO_BRANCH` / `HOBO_ENV` carry soft multi-tenancy. Hard isolation is phased; PolicyEngine is the baseline enforcer. Extreme isolation uses separate Yards/Stations. ## Agents Agents authenticate like any principal, get tool scopes via MCP, and never receive ambient admin. Details: [Agents](/HoBo/agents/) · [Credentials](/HoBo/security/credentials/) · [Audit](/HoBo/observability/agent-audit/). ## Related [Security](/HoBo/security/) · [Identity reference](/HoBo/reference/identity/) · [Connections via Station](/HoBo/primitives/stations/) --- # Multi-tenant & policy Posture: **secure-by-default, zero-trust, defense-in-depth**. Every principal (human, workload, agent) is authenticated and authorized per action — not trusted because it is “inside the VPC.” ## Threat model (summary) | Asset | Primary mitigations | |---|---| | Provider credentials | Keyless OIDC federation; short-lived resource-scoped tokens | | Tenant data & streams | Per-tenant isolation; claim enforcement on envelopes | | Executors | MicroVM isolation; no ambient creds; no egress by default | | Generation plugins | Signed + conformance; sandboxed runs | | Capability Manifest prose | Treat discovery text as untrusted (prompt-injection aware) | | Yard itself | Append-only log + fencing; least privilege; immutable history | | Agents | Scoped SVIDs; policy gates; spend caps; full replay | ## Deny by default Unlisted `(role, action, resource)` tuples are denied. Control-plane CLI/MCP verbs must map to `hobo::` actions or conformance fails. ## Policy as code Policy packs attach to Yard/Junction. **Mandatory** rules are deny-ceilings (children may only further restrict). **Default** rules are overridable. Decisions always emit `policy.Decided` — even allows. See [Policy](/HoBo/reference/policy/). ## Lanes & sandboxes Sandbox lanes isolate experimental traffic and contract observations so they cannot poison baseline contracts or training trajectories. Baseline subscribers ignore sandbox-lane events by default. ## Guardrails AI operators and generation paths run under enforced guardrails (PII, jailbreak, unsafe tool use). Blocks are events: `guardrail.Blocked` + `policy.Decided`. ## Network isolation Hard environment isolation and optional Privileged Access Gateway bound who can reach what. Overlay (Tailscale/Netbird-class) and hierarchical DNS names live in connectivity — never bare `.hobo` TLD. Details: [Connectivity](/HoBo/reference/connectivity/) · [Naming](/HoBo/reference/naming/). ## Supply chain - Plugins: signed, SLSA-style provenance, conformance report required - Images: reproducible builds, digest-pinned, scanned - Dependencies: SBOM + vuln gates - Air-gap: signed offline bundles — [Air-gapped](/HoBo/guide/air-gapped/) ## Audit & compliance The event log *is* the audit log. Security-relevant families export as **OCSF** to SIEM. Audit streams are never-compact by policy. Data residency constraints are Planner-enforced. ## Provider terms Registered providers are **the operator’s own accounts**. Never pool free-tier headroom across tenants; never rotate accounts to reset free tiers. Rotation is **across providers**, not across accounts within a provider. ## Runtime isolation (WASM) WASM capability security is a **structural** isolation dimension (deny-by-default imports), alongside data / compute / credentials / network. ## Related [Identity](/HoBo/security/identity/) · [Policy reference](/HoBo/reference/policy/) · [DevSecOps role](/HoBo/roles/devsecops/) · [Agent audit](/HoBo/observability/agent-audit/) --- # Connectivity & networking Networking in HoBo is **identity-aware and name-first**. Apps dial hierarchical named endpoints; material stays brokered; isolation tiers fail closed. ## Named endpoints - Pure hierarchical naming (DNS-safe labels, never bare `.hobo`) - Local: `*.hobo.localhost` - Overlay: `*.hobo.test` - **Never** bare `.hobo` See [Naming reference](/HoBo/reference/naming/). ## Overlay & DNS Connectivity hosts overlay adapters (Tailscale/Netbird-class), pluggable DNS providers, and Junction dynamic DNS for federation without shared mutable control state. ## Secretless connectors Workloads use connector plugins (postgres/tcp/mysql/redis, …) that dial **names**. Classical URL+password is residual and audited. ## Isolation & PAM Environment isolation tiers + optional Privileged Access Gateway sessions. Cross-Station dials need federation grants. ## Local DX `hobo dev` registers named endpoints; optional portless mode; `HOBO_PORTLESS=0` forces ports. `hobo doctor` reports connectivity hazards. ## Related [Connectivity reference](/HoBo/reference/connectivity/) · [Security](/HoBo/security/) · [Stations](/HoBo/primitives/stations/) --- # Agent credentials Agents are first-class principals. They receive **scoped, short-lived** credentials — never ambient org-admin by default. ## Issuance loop 1. Agent authenticates (SVIDs / OIDC device flows as applicable). 2. Declares needed capabilities for the task. 3. Policy allow / deny / require-approval. 4. Broker issues time-boxed token/scopes. 5. Audit refs only in the log. ## Rules | Rule | Detail | |---|---| | Least privilege | Scope to tools + resources needed | | Short TTL | Rotate; no long-lived agent PATs as default | | No plaintext in context | Schema-only views and fixtures for secrets | | Spend caps | Policy breakers on runaway tool use | | MCP auth | Same policy plane as CLI | ## Related [Agents overview](/HoBo/agents/) · [MCP](/HoBo/agents/mcp/) · [Identity](/HoBo/security/identity/) · [Audit](/HoBo/observability/agent-audit/) --- # CI/CD & continuous delivery Placement and delivery are **standing decisions**. Deploy, migrate, and progressive rollout share one gated path. ## In this section | Page | Topic | |---|---| | [Continuous migration](/HoBo/cicd/migration/) | Dispatcher re-placement; dual windows; Trails | | [Progressive delivery](/HoBo/cicd/progressive-delivery/) | Harness flags, experiments, gateway weights | | [Generation pipeline](/HoBo/cicd/generation/) | Governed AI generation under conformance | | [GitOps operator](/HoBo/operations/gitops/) | Promote only when can-deploy is green | | [Deployment rehearsal](/HoBo/testing/deployment/) | Offline plan / local lane / Gatekeeper tests | ## Related [Control plane](/HoBo/architecture/control-plane/) · [Trails](/HoBo/primitives/trails/) · [Harness reference](/HoBo/reference/harness/) · [Operations](/HoBo/operations/) --- # Continuous migration HoBo’s differentiator: placement is a **standing decision**, not a historical one. When a cheaper or better-policy home opens, the platform plans the move, gates it, and converges — without a rewrite or a weekend war room. ## Same path as deploy | Step | Deploy (`hobo up`) | Continuous migrate | |---|---|---| | Plan | DeploymentPlan DAG | Same | | Gate | Gatekeeper | Same (+ can-migrate / can-trail as needed) | | Converge | ResourceActors | Same | | Observe | Events + inspect | Same | There is **no** separate migration engine with different trust properties. ## What triggers re-placement - Price / credit / free-tier ranking updates (as **ranking input**, dated sources) - Policy or residency changes - Reliability / SLO signals - Explicit human or Crew proposal - Trail recipe readiness Dispatcher owns continuous re-entry; FinOps Crew **advises**. ## Dual windows Schema/data duals, identity dual-path, and traffic duals (Harness weights) let you run old and new in parallel until cutover gates pass. Fail closed on integrity — never “best effort drop events.” ## Trails App-owned recipes compose platform steps (schema dual, junction rebind, `identity.continuous`, progressive cutover). Pure identity continuous still requires **zero** Trail authoring. See [Trails](/HoBo/primitives/trails/). ## What migration is not - Not downtime-required by default (progressive + dual) - Not a free pass around can-deploy - Not silent rebinding of external customer domains without policy - Not account-hopping within a provider to reset free tiers ## Testing [Migration fixtures](/HoBo/testing/migration/) · [Dual fixtures](/HoBo/testing/dual-fixtures/) · [Deployment rehearsal](/HoBo/testing/deployment/) ## Related [Progressive delivery](/HoBo/cicd/progressive-delivery/) · [Control plane](/HoBo/architecture/control-plane/) · [Matching](/HoBo/reference/matching/) --- # Progressive delivery Progressive delivery is how HoBo shifts **traffic, flags, and contract observation** without treating every change as a big-bang deploy. ## Building blocks | Block | Role | |---|---| | **Harness flags** | Ranked evaluation with audit facts | | **Experiments** | Bounded exposure | | **Gateway weights** | Split traffic across revisions/camps | | **Sandbox lanes** | Isolate experimental observations | | **Trails** | Multi-step cutover recipes when infra + data move together | | **Gatekeeper** | Still admits infra mutations | ## Safe defaults 1. Baseline contracts ignore sandbox-lane events. 2. Flag and weight changes are **events** — replayable. 3. Auto-rollback hooks pair with canary live-fix (Crew SRE path). 4. Prod writes often `require-approval` under policy packs. ## CLI - `hobo feature …` — flags / progressive ops - `hobo trails plan|status|inspect` — recipe observation - `hobo inspect` — projections and time-travel ## Testing Shadow Bridge mode, dual fixtures (flag on/off), and chaos partial shifts. See [Harness reference](/HoBo/reference/harness/) · [Bridge modes](/HoBo/testing/bridge-modes/). ## Related [Continuous migration](/HoBo/cicd/migration/) · [Trails](/HoBo/primitives/trails/) · [Crew SRE](/HoBo/operations/sre/) --- # Generation pipeline HoBo treats AI generation as a **governed factory**, not a paste bin. Generated providers, patches, and app assistance run under the same contract, policy, and audit spine as human work. ## Loop (conceptual) 1. Request generation (`hobo generate` / MCP / Crew SWE·UX) 2. Produce candidate artifacts in a **sandbox lane** 3. Conformance + tests + SBOM/vuln gates 4. Human or policy approval when required 5. Registry certification / promote out of sandbox 6. Patchability preserved — generated code is not a black box seal ## Guardrails - Prompt/tool safety → `guardrail.Blocked` + `policy.Decided` - Capability Manifest prose is untrusted input (injection-aware) - No ambient credentials in generation executors - Training trajectories exclude sandbox poison by default ## What generation must not claim - Silent production mutate without Gatekeeper - Skipping conformance because “the model was confident” - Embedding secrets into generated IaC or docs ## Related [Progressive delivery](/HoBo/cicd/progressive-delivery/) · [SWE Crew](/HoBo/operations/swe/) · [Security](/HoBo/security/) · [Conformance testing](/HoBo/testing/conformance/) --- # Meet the primitives HoBo’s vocabulary is small on purpose. Learn these nouns once; every guide and reference page reuses them. ## Topology & packaging | Primitive | One-liner | Page | |---|---|---| | **Bindle** | Provider-agnostic packed app | [Bindles](/HoBo/primitives/bindles/) | | **App Kit** | Bindle-declared app surface (`api`/`web`/…) | [App Kits](/HoBo/primitives/app-kits/) | | **Camp** | Environment shape + runtime requirements | [Camps](/HoBo/primitives/camps/) | | **Yard** | Provider selection + camps | [Yards](/HoBo/primitives/yards/) | | **Junction** | Cross-yard federation | [Junctions](/HoBo/primitives/junctions/) | ## Control isolation & migration | Primitive | One-liner | Page | |---|---|---| | **Station** | Hard isolation for a Conductor | [Stations](/HoBo/primitives/stations/) | | **Trail** | App-owned continuous migration recipe | [Trails](/HoBo/primitives/trails/) | | **Resources** | Catalog factories (App Kits + Infra Kits) | [Resource catalog](/HoBo/primitives/resources/) | ## Not primitives (but nearby) | Term | Where it lives | |---|---| | Conductor / Broker | [Guide](/HoBo/primitives/stations/#the-conductor) | | Gatekeeper | [Reference](/HoBo/reference/contracts/) | | Harness | [Reference](/HoBo/reference/harness/) | | Crew | [Crew](/HoBo/crew/) | ## Related [Core concepts](/HoBo/guide/concepts/) · [Architecture](/HoBo/architecture/) · [Glossary](/HoBo/reference/glossary/) --- # Bindles > *A hobo's bindle is everything they own, tied to a stick. If it doesn't fit in the bindle, it doesn't come along.* The **Bindle** is HoBo's first primitive and the only thing HoBo ships. It's your whole application — services, resources, wiring, policies, deployment intent — packed into one **provider-agnostic** bundle. You cannot run HoBo without one, and the discipline it imposes is the whole business model: an app that fits in its bindle is an app that's cheap to move, and an app that's cheap to move is an app whose rent goes down. ## Declaring a bindle A bindle is plain code, and this surface ships today in `hobo/bindles`: the default export of your project's `src/bindle.ts` is exactly what `hobo collect`, `hobo plan`, `hobo up`, and `hobo dev` load (a committed `hobo.bindle.json` manifest artifact wins when present). `bindle(name, build)` hands your build function a resource builder; the object you return *is* the manifest: ```ts import { bindle } from "hobo/bindles"; import { app as api } from "./api"; import { app as web } from "./webapp"; export default bindle("my-app", (b) => { const db = b.data().allow("read", "write"); const jobs = b.queue().allow("enqueue"); const cache = b.kv().allow("get", "set"); const uploads = b.bucket(); const apiKey = b.secret().allow("access"); const apiSvc = b .api(() => api.fetch) .dependsOn(db, jobs, cache, apiKey); const webSvc = b.web(() => web.fetch).dependsOn(apiSvc); return { db, jobs, cache, uploads, apiKey, api: apiSvc, web: webSvc }; }); ``` ## App Kits vs Infra Kits Resources fall into two authoring families: | Family | Kinds | Session Replay | |---|---|---| | **[App Kits](/HoBo/primitives/app-kits/)** | `api`, `web`, `service`, `ui`, `websocket`, `job`, `wasm-component` | **Default-on** (privacy-first) | | **Infra Kits** | `data`, `kv`, `bucket`, `queue`, `topic`, `secret`, … | Off unless explicitly annotated | WIT/WASM is how ports and providers link at runtime; App Kits are the Bindle-facing surfaces you declare. ## The builder surface | Builder | Kind | Family | What it declares | |---|---|---|---| | `b.api(handler?)` | `api` | App Kit | HTTP API entrypoint. **Session Replay default-on.** | | `b.web(handler?)` | `web` | App Kit | Web app entrypoint. **Session Replay default-on.** | | `b.service(handler?)` | `service` | App Kit | Long-running service. **Session Replay default-on.** | | `b.data()` | `data` | Infra Kit | A SQL-ish managed data port. | | `b.kv()` | `kv` | Infra Kit | Key-value storage. | | `b.bucket()` | `bucket` | Infra Kit | Object storage. | | `b.queue()` | `queue` | Infra Kit | Point-to-point work queue. | | `b.topic()` | `topic` | Infra Kit | Pub/sub topic. | | `b.secret()` | `secret` | Infra Kit | A secret reference — never a secret value. | Every resource supports three chainable refinements: | Method | Meaning | |---|---| | `.dependsOn(...resources)` | Declares wiring edges **and** injects typed bindings at runtime. Deny-by-default: no declared edge, no access. | | `.allow(...verbs)` | Inline least privilege, with verbs typed per resource kind (`"read"`/`"write"`/`"query"` on `data`, `"enqueue"`/`"dequeue"` on `queue`, `"get"`/`"set"` on `kv`, …). Permissions travel in the bindle. | | `.with(...annotations)` | Typed provider-specific options — the escape hatch. Includes `sessionReplay(...)` to opt a Kit out of Session Replay or set sample rate. | ## Session Replay on every App Kit Every App Kit is annotated with platform option `session-replay` by default — **api, web, service, ui, websocket, job, wasm-component**, not only `web`. Local `hobo dev` wraps HTTP App Kits: HTML gets a privacy-first bootstrap; `/__hobo/replay/*` serves ingest/list — **no app tracker code required**. ```ts import { sessionReplay } from "hobo/bindles"; // opt out one Kit b.api(() => handler).with(sessionReplay({ enabled: false })); ``` Details: [Session Replay](/HoBo/reference/session-replay/) · [App Kits](/HoBo/primitives/app-kits/). ## Greenfield: declare the bindle, generate the code Packing an existing app — the brownfield path above — is only half the story. A component can also be **declared**: a `@component` class whose `@operation` methods carry the groundings HoBo generates code from — an **intent** (what it's for, in prose), a **contract declaring the shape** (typed input and output, plus invariants and forbidden outputs — the boundary generated code may never cross; the shape is never optional), and an optional **rubric** (named scorers with promotion thresholds, enforced as policy at the gate — omit it and the gate defaults to exact-match over your contract's examples). An operation with a `stub()` body is **code to be generated**; a hand-written body is a baseline the optimizer improves. ```ts import { bindle } from "hobo/bindles"; import { component, operation, param, stub } from "hobo/runtime"; import { z } from "zod"; import { Cart, Order, Photo, RankInput, submitOrder } from "./domain"; @component({ intent: "Public storefront API: ranked feed, cart, checkout." }) class Storefront { @operation({ intent: "Order photos to maximize dwell without burying brand-new sellers.", contract: { input: RankInput, // the shape — required output: z.array(Photo), forbiddenOutputs: ["output[0] is nsfw"], }, params: { input: param("Candidate photos with engagement signals") }, rubric: { dwellLift: 0.8, newSellerShare: 0.75 }, // optional — sharpens the gate }) rankFeed = stub(); // no implementation: generate it @operation({ intent: "Turn a priced cart into an order exactly once.", contract: { input: Cart, output: Order }, }) async checkout(cart: Cart): Promise { "use training"; // hand-written baseline the optimizer keeps improving return submitOrder(cart); } } /** The bindle hands the declared component straight to the builder. */ export default bindle("snapshop", (b) => { const photos = b.bucket(); const catalog = b.data().allow("read", "write"); const api = b.api(Storefront).dependsOn(catalog, photos); return { photos, catalog, api }; }); ``` `rankFeed` has zero implementation, and the loop from declaration to running code is the same convergence loop as everything else in HoBo: 1. **Declare.** The bindle above is a complete, plannable manifest — `hobo plan` and `hobo up` work on it unchanged. Under the hood each `@operation` lowers to a **trainable** bound to its resource — signature-only for stubbed operations. That's the same machinery as `"use training"`, but you never touch it directly. 2. **Stub.** `hobo stub` emits the generated registrations and a typed client into your project's generated output. Every declared operation is servable from day one by a **contract-derived stub**, so the whole app runs end-to-end before any implementation exists — and every stub call is recorded as a **specification-by-example** trace. 3. **Generate.** The same **generation pipeline** that writes [provider adapters](/HoBo/customizations/providers/) synthesizes each operation's implementation from its interface + contract + recorded traces, in a capability-scoped sandbox with no egress. 4. **Gate.** Promotion needs all three lenses: **conformance green** (the contract must never break), **eval thresholds met** (the rubric — scorer by scorer), and **policy allows** (prod promotions can require human approval). 5. **Promote.** A passing candidate is written back into the operation's generated region as an AST-aware edit with signed provenance — auto-applied when low-risk, raised as a PR when not. Prior candidates live in the event log, so promotion is reversible. Prefer declaring one grounding at a time? Every decorator is optional and the options object decomposes into **granular decorators** — `@intent`, `@returns` on methods, `@description` on parameters (in legacy-decorator projects; as a `params:` value under standard decorators), with a bare `@trainable` composing the class. The declaration can even be **ambient** — a `declare class` with only signatures, no bodies at all — which `hobo stub` scans statically, taking the TypeScript signature as the declared shape: ```ts @trainable export declare class Program { @intent("Produce a simple hello-world program") @returns("Hello World! or Hello, ! when supplied") trainableMethod( @description("Optional person to greet") name?: string, ): string; } ``` Anything you don't declare is inferred: intent from the method name, a synthesized `decl://` contract ref, the default exact-match rubric. Escalation is **per operation, not per app** — `Storefront` above already shows all three rungs. Swap `rankFeed`'s `stub()` for a real body when you're ready to own it; keep `"use training"` on it (like `checkout`) and the optimizer keeps improving the body against the same intent + contract + rubric, under the same gates; drop the directive and it's plain hand-maintained code. The manifest never changes shape along the way. Brownfield and greenfield converge on the same artifact: one manifest, one `hobo up`, one event log. Pack code you have, declare code you don't, and mix freely — a half-implemented bindle is a normal bindle. ## Rules the bindle enforces - **Names come from the returned object keys.** `{ db, api }` yields stable IDs like `data:db` and `api:api`. Duplicate stable IDs are a hard error. - **The manifest is pure data.** Serializing a bindle yields exactly `{ name, resources, edges }` — handlers are reachable for tooling via `definition.entrypoint(name)` but never leak into the graph. - **Everything is frozen.** A built bindle is deeply immutable. The manifest you planned against is the manifest that ships. - **Provider-agnostic by construction.** Nothing in the builder surface names a cloud. Provider choices live in [Camps](/HoBo/primitives/camps/); provider *hints* live in `.with(...)` annotations, which any other provider is free to ignore. ## Why this shape matters Every classic lock-in mechanism is a thing that lives *outside* your app description: the console-clicked queue, the IAM policy in a wiki, the env var that only exists in one region. The bindle's job is to leave nothing outside. When permissions (`.allow`), wiring (`.dependsOn`), and intent are all in the bundle, **unpacking it somewhere cheaper is a mechanical operation** — which is what lets [continuous migration](/HoBo/guide/concepts/#continuous-migration) be routine rather than heroic. ## Where bindles go next - A bindle needs a home: [Camps](/HoBo/primitives/camps/) — pitch one to pin placement, or let HoBo infer it from the code you've configured. - Multiple homes under one provider: [Yards](/HoBo/primitives/yards/). - Homes across clouds: [Junctions](/HoBo/primitives/junctions/). --- # App Kits An **App Kit** (short: **Kit**) is a Bindle-declared **application surface** — the kinds that host entrypoints or interactive runtime behavior. Kits compose runtime ports into something you pack in a Bindle; providers implement those ports behind WIT interfaces. ## App Kit kinds | Label | Kind | Role | |---|---|---| | API Kit | `api` | HTTP API entrypoint | | Web Kit | `web` | Web / HTML entrypoint | | Service Kit | `service` | Long-running HTTP service | | UI Kit | `ui` | UI render surface | | Realtime Kit | `websocket` | Realtime rooms / sockets | | Job Kit | `job` | Batch / async work | | Component Kit | `wasm-component` | Wasm component dependency as surface | Helpers: app kit kinds and labels in the Bindles SDK. ## Infra Kits (not App Kits) Infrastructure/port resources that are **not** entrypoint hosts: `kv` · `bucket` · `queue` · `topic` · `secret` · `data` · `schedule` · `external` · `container` · `flag` · `auth` · `crypto` · `notify` · `guardrail` · `golden` · `experiment` · `rollout` See infra kit kinds in the Bindles SDK. ## HTTP App Kits Subset of App Kits that local Yard hosts serve as fetch handlers: `api` · `web` · `service` · `ui` — `HTTP_app kit kinds` / `isHttpAppKitKind`. ## Session Replay **Every App Kit** is Session-Replay capable by default: privacy-first capture, dual fixtures locally, progressive gating in production. Infra Kits are not auto-annotated. Opt out per Kit: ```ts import { sessionReplay } from "hobo/bindles"; b.api(handler).with(sessionReplay({ enabled: false })); ``` Kill switch: `HOBO_REPLAY=0`. Full surface: [Session Replay](/HoBo/reference/session-replay/). ## Naming split | Term | Layer | Meaning | |---|---|---| | **App Kit** | Bindle authoring | Entrypoint-bearing resource kinds | | **Infra Kit** | Bindle authoring | Port / infrastructure resources | | **WIT port** | Runtime capability interface | Host/provider link for a Kit | ## Related [Bindles](/HoBo/primitives/bindles/) · [Resource catalog](/HoBo/primitives/resources/) · [Session Replay](/HoBo/reference/session-replay/) · [WASM & WIT](/HoBo/architecture/wasm/) · [Core concepts](/HoBo/guide/concepts/) --- # Camps > *A camp is where you stay, not who you are. Good campsites are worth appreciating — and leaving, when the rent goes up.* The **Camp** is HoBo's second foundational primitive: the answer to *what kind of home is this?* — environment type, requirements, and preferences — **not** which cloud account or which Bindle owns the slot. Every running app occupies a camp, but **writing one is optional**: leave it out and HoBo infers a camp from policy and approved providers. You write one to pin environment semantics yourself, or because placement is locked down to explicit camps. ## What a camp is (and is not) | Camps **do** | Camps **do not** | |---|---| | Name an environment (`prod`, `staging`, `pr-123`, …) | Own a Bindle | | Carry `envType`, labels, require/prefer constraints | Select a cloud provider | | Declare runtime **requirements** (substrate needs) | Embed long-lived credentials | | Attach under a [Yard](/HoBo/primitives/yards/) | Mix providers in one camp | **Provider selection belongs on the Yard.** **Bindles stay provider-agnostic** and are placed by deployment request — not nested inside `camp(...)`. ## Authoring ```ts import { camp } from "hobo/camps"; // Environment shape only — no bindle, no provider. export const prod = camp("prod", { envType: "prod", require: ["durable-storage"], prefer: ["low-latency"], runtimeRequirements: ["runtime-local"], labels: { tier: "gold" }, }); export const staging = camp("staging", { envType: "staging", }); ``` Attach camps when you author a yard: ```ts import { yard } from "hobo/yards"; import { prod, staging } from "./camps"; export default yard("aws-east", { provider: "aws", regions: ["us-east-1"], camps: [prod, staging], }); ``` Then converge with the CLI: ```sh hobo plan --camp prod # dry run: placement, cost, warnings hobo up --camp prod # converge ``` > **Day-one path:** you often never write a camp file. `hobo up` / `hobo dev` against the default **local lane** infers a local camp and durable SQLite journal with no cloud account. Explicit camps matter when you pin multi-env topology or platform policy requires them. ## Rules camps enforce - **One home at a time.** Mixing providers in one camp is invalid — that's what [Yards](/HoBo/primitives/yards/) (one provider) and [Junctions](/HoBo/primitives/junctions/) (many providers) are for. - **Runtime is a requirement, not a vendor SDK.** Camps declare substrate needs (`runtimeRequirements`); engines that satisfy them plug in behind Camp ports (EventStore, Queue, ActorHost, Clock). The **local runtime** ships today; alternate engines (Postgres-backed, Temporal, Restate, …) are on the roadmap. - **Environment comes from the name.** `prod`, `staging`, `preview`/`pr-*`, and `dev` get sensible `envType` inference; override when your naming is more creative. - **Rejected at parse:** `bindle`, `defaultProvider`, `tiers`, and `bindings` on `camp(...)` — those belonged to an older model. Prefer Yard provider + Bindle deploy. ## Writing a camp is optional - **Inferred camps (the default).** With no camp declared, the Dispatcher infers one from configured code and approved providers, optimizing for price under policy. - **Locked-down placement.** A junction broker admin can require explicit, approved camps. - **Local camp.** `hobo dev` runs your bindle against the local runtime — offline on your laptop, with honest fidelity labels where local behavior differs from cloud. ## Camps and continuous migration **The camp is your app's current address, not its identity.** Because the bindle is provider-agnostic and every camp exposes the same runtime contract, the Dispatcher can propose a cheaper camp at any time — and the move is the same plan → gate → converge loop as `hobo up`. ## Where camps go next - Provider scope and multi-region: [Yards](/HoBo/primitives/yards/). - Multi-cloud federation: [Junctions](/HoBo/primitives/junctions/). - Bindle authoring: [Bindles](/HoBo/primitives/bindles/). - Trail map: [Status](/HoBo/guide/status/). --- # Yards > *The rail yard is where a line's traffic gets sorted. One company's yard, one company's rules.* The **Yard** is where **provider selection** lives. It composes **one provider** with zero or more [Camps](/HoBo/primitives/camps/) and carries what is shared across those camps: policy defaults, identity scope, reconciliation, and provider-global networking. It is also, not coincidentally, where HoBo's control-plane scope shows up for that provider. **Day one? You may not need this page.** A Bindle plus the default local lane is enough — [Getting started](/HoBo/guide/getting-started/). Come back when you pin a real provider or multi-region topology. ## Building a yard ```ts import { camp } from "hobo/camps"; import { yard } from "hobo/yards"; const prod = camp("prod", { envType: "prod" }); const staging = camp("staging", { envType: "staging" }); // Provider is required on the Yard — never on the Camp. export default yard("aws-east", { provider: "aws", regions: ["us-east-1"], camps: [prod, staging], }); ``` That's the authoring shape: a name, a **provider**, optional regions/labels/requirements, and member camps (may be empty for a requested shape before live camps exist). ## Rules yards enforce - **One provider, strictly.** A yard cannot mix providers. Multi-region within a provider = several camps under one yard. Multi-cloud = several yards under a [Junction](/HoBo/primitives/junctions/). - **Provider is required.** Camps do not carry `defaultProvider`; the yard is the only direct selection point. - **Camps are shapes, not Bindle owners.** Attach env camps; place Bindles via deploy/converge, not by nesting `bindle:` inside `camp(...)`. - **Provider-native properties are refs only** — no credentials in yard properties. ## When to reach for a yard - **Pin a provider** for prod/staging fleets. - **Multi-region:** `prod-east` and `prod-west` camps behind one policy scope. - **Environment fleets:** prod, staging, and ephemeral `pr-*` camps that share identity and promotion rules. - **Governance:** one place to enforce conformant provider builds across camps. - **Self-host control:** run a local/bare-metal yard (`hobo yard …`) when you own the control plane substrate. ## The yard is itself a bindle HoBo's control plane ships *as a bindle*. Self-hosting is converging that bindle into a yard you operate — same tooling app teams use. No enterprise-edition fork, no second product. ## Where yards go next - Crossing clouds: [Junctions](/HoBo/primitives/junctions/). - Environment shapes: [Camps](/HoBo/primitives/camps/). - Machinery: [Architecture](/HoBo/architecture/). --- # Junctions > *Junctions are where lines cross. Nobody owns a junction — that's the point.* The **Junction** is HoBo's most advanced primitive: the federation layer that connects [Yards](/HoBo/primitives/yards/) **across clouds** for hybrid and multi-cloud operation. Yards remain peers — there is no super-controller and **no shared mutable state**. The final edge of HoBo's scope model is federation, not containment. **This is graduate-level wandering.** Most apps live happily in one camp for a long time; many never need a second yard. If you're just arriving, start with [Getting started](/HoBo/guide/getting-started/). > **Early-stage:** Junction authoring now ships the authoring and inference surface — `junction()`, connectivity-tier selection, the federated DNS plan, per-yard plan extraction, and the trust/policy/registry federation records. It's a local derivation engine: each yard still realizes its own half, and the realization machinery lands with the multi-cloud stage (see [the trail map](/HoBo/guide/status/)). ## Wiring a junction ```ts import { junction } from "hobo/junctions"; import awsYard from "./yards/aws"; import azureYard from "./yards/azure"; export default junction("my-app", { yards: [awsYard, azureYard] }); ``` Just the members — connectivity is inferred. The junction reads the member yards' manifests and derives the cross-cloud connectivity plane (peering/tiering plus DNS), and **each yard realizes its own half**. Trust federation, signed policy packs, and signed registry bundles ride the same mechanism. ## Rules junctions enforce - **Peer yards only.** A junction adds yards; it does not reach into them. Each yard keeps its own event log, its own control plane, its own blast radius. - **No shared mutable state.** What crosses a junction is signed, versioned artifacts — policy packs, registry bundles, trust roots — not live coordination. - **Policy at the border.** Data-residency, sovereignty, and egress rules attach at the junction scope: what may cross which border is an explicit, auditable decision. - **The broker admin holds the keys.** Junction governance is also where camp inference can be switched off: a junction broker admin can require explicit, approved [camps](/HoBo/primitives/camps/), so placement stays a deliberate act instead of an inferred one. The lockdown is enforced by [the Conductor](/HoBo/primitives/stations/#the-conductor), and no narrower policy can switch inference back on. ## When to reach for a junction - **Hybrid:** a cloud yard federated with the yard running on your own metal. - **Multi-cloud by policy:** EU data in an EU yard, everything else wherever it's cheapest. - **Credit surfing:** when your app's cheapest home this quarter is on the cloud that just handed you startup credits — and next quarter it isn't. - **Exit insurance:** the practical difference between "we could leave AWS" and "we have left AWS twice this year, briefly, and nobody noticed." ## Junctions and continuous migration A junction is where the guiding principle gets its full range. Within one camp, migration means rebalancing resources; within a yard, moving between regions; **across a junction, the Dispatcher weighs whole clouds against each other** — price, credits, egress, residency, reliability — and proposes policy-gated moves across the border. The same plan → gate → converge loop as everything else. The map got bigger; the walking stayed the same. ## Where junctions go next - The vocabulary in one sitting: [Core concepts](/HoBo/guide/concepts/). - The machinery that plans cross-cloud moves: [Architecture](/HoBo/architecture/). --- # Stations & the Conductor A **Station** is the hard control-plane isolation boundary a **Conductor** occupies. It is **not** Bindle ⊂ Camp ⊂ Yard ⊂ Junction containment. The **Conductor** (sometimes called the broker in lower-level surfaces) is a **pure runtime** that mounts a Station client + active Rail(s), runs resolve/scale/deliver, and emits receipts that pin `stationId` + `railHash` (+ catalog/policy hashes). It owns no durable Connections or policy packs — the Station does. ## Station owns | Asset | Role | |---|---| | Connections | Authorized dial/control relationships | | Rails | Content-addressed profiles / pins | | Bootstrap Rail | Initial trusted least-privilege profile | | Local client state | Under `HOBO_HOME/stations/` | Authoring and ops surfaces: `station()` helpers, Rails, local client, and Privileged Access Gateway hooks. ## Station CLI (ops/setup) ```text hobo station init|up|down|status|rails * ``` These are **ops**, not app lifecycle verbs. Missing Station **fails closed** for control-plane work — run `hobo station init`. ### Profile requirements | Field | Role | |---|---| | `defaultStation` | Which Station is active | | `activeRail` | Which Rail pins policy/profile | | `defaultBrokerUri` | Remote Conductor HTTP endpoint only — **not** a setup path | Cross-Station Connection use fails closed without federation grant. ## PAM Privileged Access Gateway sessions can be Station-aware — break-glass paths still audit. --- ## The Conductor Somebody has to decide which provider hosts your bindle, whether a deploy is allowed, whether it fits the budget, whether it breaks a downstream contract, whether the app scales up or down to zero. That somebody is **the Conductor** — the one who decides which train you're riding. In code: **broker** (Conductor). Same component. The Conductor decides *only* from the policies it is configured to maintain, while **occupying a Station**. ### What it does It takes a brokering request — "place these resources for this [bindle](/HoBo/primitives/bindles/), in this scope" — and answers with exactly one of: - a **Resolution Receipt**: per-resource placements (provider, region, SKU, price), pinned to content hashes of the provider catalog and the policy set — optionally **Ed25519-signed**; or - a **Resolution Rejection**: machine-readable reasons from a **closed rejection catalog**, each naming the enforcement plane that refused (`policy`, `contracts`, `billing`, `placement`, `connectors`) with a remediation agents can act on. Placement is brokered **per resource, across providers**. Spin-up/spin-down (`up` / `down` / `zero`) are Conductor commands; accepted receipts go to delivery (Argo CD Application rendering / live submission when wired). ### The five planes it wraps | Plane | What it enforces | Default | Also shipped | |---|---|---|---| | **Policy** | placement, approvals, junction lockdowns | TypeScript policy packs | OPA/Rego fail-closed | | **Contracts** | can-deploy compatibility | Gatekeeper / PACT | external Pact broker feed | | **Billing** | budget, spend caps, quotas | in-proc billing policy | metered plane over usage facts | | **Delivery** | rolling out accepted receipts | rendered Argo Applications | live Argo CD API, fail-closed | | **Connectors** | external systems the Conductor may act through | connector registry | external catalog | Unreachable external policy engines **fail closed** — no engine, no placement. ### Policies are layered — lockdowns win Merge outside-in: **junction → yard → camp → bindle**. Choice-shaped fields can be overridden by narrower scopes; **restrictions accumulate** (denied providers stay denied). Junction broker-admin lockdowns cannot be relaxed by bindle policy. ### Connector registry Connectors (GitHub App, Pulumi, Azure, Vercel, Cloudflare, …) are registered with **credentialRef pointers only** — never secrets in the registry. The Conductor brokers short-lived access through identity/Station connections. ### CLI ```text hobo broker config [--init] hobo broker up ``` Requires Station context when available (`HOBO_STATION` / profile / `--station`). With no cloud config, the local machine can register as a zero-cost local provider host so the loop is testable offline. ### Related [Control plane](/HoBo/architecture/control-plane/) · [Security](/HoBo/security/) · [Matching](/HoBo/reference/matching/) · [CLI](/HoBo/reference/cli/) · [Connectivity](/HoBo/reference/connectivity/) --- # Trails **Trails** are app-owned, versioned continuous migration recipes (Trails package). They compose platform capabilities — schema duals, junction rebinds, progressive cutover, `identity.continuous` — without forcing Trail authoring for pure identity continuous. ## Step kinds (conceptual) | Kind | Role | |---|---| | Schema/data dual | Open dual window; cut over when safe | | Junction rebind | Peer routing change | | Progressive | Compose Harness weights/flags | | `identity.continuous` | Dual-path principal continuity | | Custom recipe | PlatformRecipe catalog entries | ## Gates - **can-trail** — readiness/evidence before run - **Gatekeeper** — still admits infra mutations - **Golden oracles** — pin expected plan/run verdicts in tests ## CLI ```text hobo trails status|plan|inspect ``` Observe and plan; do not invent parallel progressive/ETL planes. ## Rules 1. Pure identity continuous still requires **zero** Trail authoring when Matching/policy drives dual-path continuity. 2. Trails do not replace can-deploy. 3. History and run state are facts — replayable. 4. Progressive delivery prefers existing Harness. ## Related [Continuous migration](/HoBo/cicd/migration/) · [Harness](/HoBo/reference/harness/) · [Identity](/HoBo/security/identity/) · [Migration fixtures](/HoBo/testing/migration/) --- # Resource catalog App resources are declared in **`hobo/bindles`** — the app authoring surface — not in lower-level runtime ports. Names come from record keys the builder returns; explicit Bindle v1 methods do not take resource name arguments. Resources group as **App Kits** (entrypoint surfaces) and **Infra Kits** (platform resources). Ports and providers link via WIT/WASM ([WASM & WIT](/HoBo/architecture/wasm/)). ## App Kits Entrypoint-bearing surfaces — [App Kits](/HoBo/primitives/app-kits/). **Session Replay default-on** for each. | Factory | Role | |---|---| | `api` / `web` / `service` / `ui` | HTTP App Kits | | `websocket` | Realtime Kit | | `job` | Job Kit | | `wasm-component` | Component Kit | ## Infra Kits Port / infrastructure resources — no default Session Replay. | Factory | Role | |---|---| | `kv` / `bucket` | Key-value and object storage | | `queue` / `topic` | Messaging | | `secret` | Managed secret resource (residual; prefer secretless) | | `data` | Portable data dependency (paradigm chosen by Matching) | | `schedule` | Time-based work | | `flag` / `auth` / `crypto` / `notify` / `guardrail` / `golden` / `experiment` / … | Cross-cutting catalog | ## Wiring | Method | Role | |---|---| | `.allow()` | Least-privilege capability grants | | `.with(...)` | Provider-specific optimization escape hatch | | `connector` | External dependency protocols (e.g. A2A/ACP for agents) | | `config` / `env` / `secretRef` / `fromBinding` | Config references | ## Lowering Collection produces provider-agnostic **manifest IR**. Planner lowers nodes to **ResourceIntent** for Matching, cost routing, and DeploymentPlan DAGs. ## What is not a factory - `search` is not a top-level factory (search extensions are opt-in). - Topology (`camp`, `yard`, `junction`) is **not** re-exported from bindles. - Stations/Trails are not Bindle resource factories. ## Related [Bindles](/HoBo/primitives/bindles/) · [Matching](/HoBo/reference/matching/) · [Config & secrets](/HoBo/reference/config-secrets/) · [Capability map](/HoBo/reference/packages/) --- # Operations Built-in **Yard operator agents** (the historical “Crew roster”) that detect, diagnose, and **policy-gated** remediate. Custom third-party agents live under [The Crew](/HoBo/crew/) (ACP). ## Roster | Operator | Mission | Deep dive | |---|---|---| | **SRE** | Detect → diagnose → gated remediate | [SRE](/HoBo/operations/sre/) | | **Endpoint Monitor** | Synthetics, SLO burn, drift | [Endpoint Monitor](/HoBo/operations/endpoint-monitor/) | | **GitOps** | When/what deploys; promotion | [GitOps](/HoBo/operations/gitops/) | | **Platform Engineer** | Definitions, packs, policy proposals | [Platform Engineer](/HoBo/operations/platform-engineer/) | | **SWE** | Implement / refactor / repair app code | [SWE](/HoBo/operations/swe/) | | **UX Designer** | UI against design.md | [UX Designer](/HoBo/operations/ux-designer/) | | **FinOps** | Advisory cost & anomalies | [FinOps](/HoBo/operations/finops/) | Also: [Shared model](/HoBo/operations/shared-model/) · [Handoffs & A2H](/HoBo/operations/handoffs/) · [Runbooks](/HoBo/operations/runbooks/) · [Engines & ports](/HoBo/operations/engines/) ## Guarantees - Org RBAC — cannot exceed role - Event-sourced observations and tool calls - MCP tools where applicable - Engine behind a port (swappable) - Writes policy-gated (often require-approval in prod) ## Related [The Crew (custom ACP)](/HoBo/crew/) · [Observability](/HoBo/observability/) · [CI/CD](/HoBo/cicd/) --- # Crew shared model ## Identity & policy - Each operator is a **principal** with workload identity and short-lived brokered credentials. - Actions are deny-by-default RBAC on `hobo:*` control-plane verbs. - Mandatory org policy is a **deny ceiling** — operators cannot widen what policy denies. - Credential audit answers: *what did this operator access, with which token, when?* ## Event families | Family | Use | |---|---| | `crew.*` | Roster-level: dispatched, proposed, applied/denied, handoff | | `run.*` | Durable agent run lifecycle | | Domain families | `incident.*`, `pipeline.*`, `deployment.*`, `gen.*`, … | **No per-agent event silos.** Remediation facts stay on domain streams. ## Engines behind ports | Concern | Default | Optional swap | |---|---|---| | SRE reasoning | Built-in | HolmesGPT adapter | | Runbooks | Pipelines engine | StackStorm-class engines | | Coding | Built-in / configured coding agent | External coding engine port | | UI generation | `ui-render` adapters | OpenUI / json-render / mcp-apps | | Cost advice | Built-in FinX projections | External recommender port | Swapping an engine is per-Yard config; the deterministic core (events, gates, RBAC) stays. ## Decision pattern 1. Observe signal (projection, alert, burn, drift, gate fail). 2. Dispatch operator (`crew.AgentDispatched`). 3. Propose action + blast radius (`crew.ActionProposed`). 4. Policy decides (`policy.Decided`) → apply or deny (`crew.ActionApplied` / `ActionDenied`). 5. Escalate via `crew.HandoffRequested` (agent→agent or agent→human / A2H). ## Integration ports Crew drives: - **`issue-tracker`** — JIRA / GitHub Issues / Linear - **`knowledge`** — Confluence / Obsidian / Wikis (runbooks, postmortems-by-replay) Writes ride `crew.ActionApplied` + `policy.Decided`. ## Guardrails Jailbreak/PII-laden prompts → `guardrail.Blocked` + `policy.Decided`, replayable. Operators stay sandboxed under AI governance. ## Related operator pages [SRE](/HoBo/operations/sre/) · [Endpoint Monitor](/HoBo/operations/endpoint-monitor/) · [GitOps](/HoBo/operations/gitops/) · [Platform Engineer](/HoBo/operations/platform-engineer/) · [SWE](/HoBo/operations/swe/) · [UX Designer](/HoBo/operations/ux-designer/) · [FinOps](/HoBo/operations/finops/) --- # SRE operator **Mission:** incident detection → root-cause → **gated remediation**. The SRE is the Crew member with the deepest product model: it binds alerting/SLO, live-fix, drift + auto-rollback, chaos findings, and pipelines into a governed incident lifecycle. ## Signals in - `alert.Fired`, `slo.BudgetBurned` - `resource.DriftDetected`, `compat.GateFailed` - `spend.CircuitTripped` / cost degradation signals - golden suite continuous failures ## Incident aggregate Stream `incident:` — single-writer, rebuildable from the log. Lifecycle (simplified): `Opened → Triaged → RootCauseIdentified → RemediationProposed → RemediationApplied|Failed → Resolved → PostmortemFiled` (+ `SLABreached` on MTTD/MTTR overrun). Multiple alerts fold into **one** incident via correlation/causation IDs — not N pages. ## Decision tree ```text signal → open Incident → diagnose (projections; optional HolmesGPT engine) → classify (runtime + contract taxonomy) → decide: low-risk reversible → auto-fix (canary live-fix + auto-rollback) needs review → propose gated Delta/PR high-impact/unknown → escalate (on-call / A2H handoff) → policy gate always ``` ### A2H (agent-to-human) Escalate with a structured handoff: failing check, proposed fixes, blast radius, decision required. Routed over notify + obligation deadlines. Human apply / re-bless / abort is event-sourced. Security or money-touching goldens marked `requiresHuman` skip auto-fix. ## Runbooks Trigger → diagnosis steps (MCP queries) → gated fix actions. Runbooks are versioned/signed, event-sourced (`runbook.*`), and built on the pipelines engine. Unmatched fix class → escalate. ## Failure taxonomy (examples) | Class | Direction | |---|---| | Runtime regression | live-fix / rollback | | Contract skew | shim / deploy-order / migration window | | Drift | converge / gated repair | | Cost-induced degradation | shed QoS via costAvailabilityProfile; propose rightsizing | Unclassifiable → escalate; never silent action. ## SLOs `sre.mttd` and `sre.mttr` as OpenSLO-style budgets. Burn can freeze risky deploys or deepen postmortem obligations. ## Postmortem The incident stream **is** the timeline. Blameless postmortem-by-replay; failed remediations become permanent regression tests. ## MCP surface (illustrative) `diagnose` · `heal` / propose remediation · rides `incident.*` + `policy.Decided` ## Related [Endpoint Monitor](/HoBo/operations/endpoint-monitor/) (feeds SRE) · [Shared model](/HoBo/operations/shared-model/) · [Chaos testing](/HoBo/testing/chaos/) --- # Endpoint Monitor **Mission:** synthetic probes, health/SLO watch, drift surfacing — the eyes that feed remediation. ## What it does - Continuously probes camp endpoints and critical user journeys. - Emits burn/alert/drift signals (`slo.BudgetBurned`, `alert.*`, drift facts). - **Does not** own remediation — it **feeds the SRE**. ## Why separate from SRE Separation of observation and action keeps blast radius honest: monitors stay autonomous; fixes stay policy-gated. A noisy probe cannot silently rewrite production without the SRE decision tree and Gatekeeper/policy path. ## Engine External synthetic-check providers sit behind a port; built-in probing is the default. Swappable per Yard. ## Events & consumers | Emits | Consumed by | |---|---| | SLO burn / alert / drift | SRE operator, dashboards, status page pipeline | | Credential audit on probes | Security review | ## Testing angle Availability tests in CI often **replay** historical burn/resolution sequences rather than paging production. Live synthetics belong in real camps; see [Chaos & availability](/HoBo/testing/chaos/). ## Related [SRE](/HoBo/operations/sre/) · [Shared model](/HoBo/operations/shared-model/) --- # GitOps operator **Mission:** drive *when* and *what* deploys; promote across environment groups under contract and policy gates. ## What it does - Coordinates delivery pipelines and publishers (Argo/Flux-class engines optional behind a port). - Promotes staging → prod **only** when `can-deploy` / Gatekeeper is green. - On skew: blocks with a **precise, machine-readable reason** an agent can fix. - Opens/rollback paths via MCP (`promote`, `rollback`) with policy checks. ## Gates it respects | Gate | Why | |---|---| | can-deploy | Producer/consumer version skew | | policy packs | Org deny ceilings, approvals | | golden suite | Reproducible multi-check bundle | | cost/quota | Budget breakers | ## Events Rides `pipeline.*` / `deployment.*` plus `crew.ActionProposed` / `ActionApplied`. No separate GitOps-only log. ## RBAC example Without Deployer role, GitOps may **propose** but not **promote**. Role bindings are mandatory-policy aware. ## Related [Contracts & Gatekeeper](/HoBo/reference/contracts/) · [Deployment testing](/HoBo/testing/deployment/) · [Shared model](/HoBo/operations/shared-model/) --- # Platform Engineer (Crew) Authors and proposes **platform** changes: resource definitions, packs, policy pack deltas, Matching hints — not day-to-day business features (that is SWE). ## Mission | Does | Does not | |---|---| | Propose policy packs | Silently weaken mandatory deny ceilings | | Draft resource definitions | Bypass Registry conformance | | Suggest Matching/Driver changes | Apply prod without obligations when required | ## Related [Platform engineers role](/HoBo/roles/platform-engineers/) · [Policy](/HoBo/reference/policy/) · [Matching](/HoBo/reference/matching/) --- # SWE operator **Mission:** implement / refactor / repair **business and message** code behind ports; generate per-language idiom-layer SDKs from `hobo:sdk` under conformance. ## What it does - Opens gated Deltas/PRs for region-scoped AST edits with provenance. - Runs implement / refactor / repair loops against contracts and tests. - Generates language idiom layers — **only** promoted when conformance + eval + policy pass. - Files issues on the issue-tracker port when human follow-up is required. ## MCP surface (illustrative) `implement` · `refactor` · `repair` — each produces reviewable diffs, not silent production patches. ## Gates | Gate | Role | |---|---| | Conformance | Port honesty | | Evals / rubrics | Quality thresholds for generated code | | Policy | Prod often require-approval | | Golden | Regression lock | ## What it is not - Not the Platform Engineer (platform definitions vs app business code). - Not unbounded “agent with cloud keys” — short-lived credentials + audit. ## Related [Generated stack concepts in Architecture](/HoBo/architecture/) · [Shared model](/HoBo/operations/shared-model/) · [UX Designer](/HoBo/operations/ux-designer/) --- # UX Designer (Crew) Generates and maintains application UI against design specs — for this site the Google [DESIGN.md](https://github.com/google-labs-code/design.md) contract is `site/DESIGN.md` (rubberhose / Steamboat Willie / Cuphead) — under the same guardrails and sandbox rules as other generators. ## Mission - Propose UI diffs in sandbox lanes - Keep components aligned with design tokens/specs - Never ship unguarded prompt output to prod without policy ## Boundaries - Uses MCP/tools with scoped credentials - Pairs with SWE for behavioral code - Accessibility baselines are gates, not suggestions ## Related [Generation](/HoBo/cicd/generation/) · [SWE](/HoBo/operations/swe/) · [Shared model](/HoBo/operations/shared-model/) --- # FinOps (Crew) The FinOps Crew member **advises** on cost and efficiency. It does not auto-spend, auto-destroy, or pool free-tier headroom across tenants. ## Mission | Input | Output | |---|---| | Metering facts / showback projections | Anomaly flags | | Price/tier ranking inputs (dated sources) | Move / commitment **proposals** | | Budget policy | Alerts; breaker recommendations | ## Hard rules 1. Ranking input ≠ billing invention. 2. Free-tier never pooled across tenants. 3. Rotation across **providers**, not multi-account free-tier games. 4. Enforcement via policy breakers/tiers — not chatty overrides. ## Related [Metering](/HoBo/reference/metering/) · [FinOps role](/HoBo/roles/finops/) · [Continuous migration](/HoBo/cicd/migration/) --- # Handoffs & A2H **A2H** (agent-to-human) handoffs are first-class: when policy requires approval or confidence is low, the agent opens an obligation with a deadline and an evidence pack — not a chat message that disappears. ## Loop 1. Agent proposes action (`crew.ActionProposed` / handoff request). 2. Obligation recorded (who, what, deadline). 3. Human approves, rejects, or amends with reasons. 4. Agent resumes from **events**, not process memory. 5. Satisfaction/closure is itself a fact. ## Rules - Deadlines create operational urgency without silent auto-approve (unless policy explicitly says so). - Evidence packs must be redaction-safe for the audience. - Handoffs are not a bypass of Gatekeeper for infra mutate. ## Related [Shared model](/HoBo/operations/shared-model/) · [Runbooks](/HoBo/operations/runbooks/) · [Policy](/HoBo/reference/policy/) --- # Runbooks Runbooks encode operational knowledge as **data**: trigger → diagnose → gated fix. Execution is event-sourced; write actions still pass policy obligations. ## Properties | Property | Detail | |---|---| | Declarative | Steps and gates, not opaque scripts only | | Replayable | What ran is in the log | | Policy-gated | Prod fixes often require-approval | | CLI | `hobo runbook` execute / replay | ## Relation to SRE Crew SRE may select and execute runbooks during incidents. Custom engines can propose runbook steps; they cannot skip Gatekeeper. ## Related [SRE](/HoBo/operations/sre/) · [CLI](/HoBo/reference/cli/) · [Incidents in shared model](/HoBo/operations/shared-model/) --- # Engines & ports Every Crew member is **pluggable**: the Yard depends on a **port**; a default engine ships; operators may swap engines that pass the same contract. ## Why ports | Benefit | Detail | |---|---| | Swap brains | Better model / specialist engine without rewriting dispatch | | Testability | Fake engines in CI | | Policy | Engine choice is still RBAC-scoped | | Audit | Tool calls and outputs remain events regardless of engine | ## Patterns - Default built-in engine per operator - Optional external engines (e.g. HolmesGPT-class for SRE) - MCP tool surface stays stable — engines are implementation - Never give an engine ambient admin outside its role ## Related [Shared model](/HoBo/operations/shared-model/) · [SRE](/HoBo/operations/sre/) · [MCP](/HoBo/agents/mcp/) --- # The Crew — custom ACP agents **The Crew** (this section) is for **custom agents you bring** — ACP/A2A-connected, policy-gated, audited. Built-in Yard operators (SRE, GitOps, FinOps, …) live under **[Operations](/HoBo/operations/)**. ## Disambiguation | Term | Where | |---|---| | **Custom ACP / A2A agents** | This section | | **Built-in operator agents** | [Operations](/HoBo/operations/) | | **Agents-as-users** (developer copilots) | [HoBo for Agents](/HoBo/agents/) | | **`agent()` runtime primitive** | SDK — both kinds use it | | **Relay** | Network proxy — *not* an agent | ## What you configure here | Topic | Page | |---|---| | Agent-as-user identity & MCP | [HoBo for Agents](/HoBo/agents/) · [MCP surface](/HoBo/agents/mcp/) | | Credentials (scoped, short-lived) | [Agent credentials](/HoBo/security/credentials/) | | Pluggable engines / ports | [Engines](/HoBo/operations/engines/) (shared machinery) | | Connector protocol (`a2a` / `acp`) | [Connectors](/HoBo/customizations/connectors/) · Bindle `connector(..., { protocol: "acp" })` | ## Rules for custom agents 1. Authenticate as first-class principals — never ambient org-admin 2. Every tool call and token grant is an auditable fact 3. High-risk writes pass Gatekeeper / policy obligations 4. Prefer connector protocols HoBo owns (ACP/A2A) over inventing a private agent SDK ## Related [Operations](/HoBo/operations/) · [Security](/HoBo/security/) · [Observability / audit](/HoBo/observability/agent-audit/) --- # HoBo for agents HoBo is built for humans **and the agents that work for them** — not as a slogan, but as an identity model. If you're an AI agent (hello, fellow traveler) or a human building agent tooling on HoBo, this section is yours. > **Reading this as an agent?** The condensed, machine-readable briefing lives at [`llms.txt`](/HoBo/llms.txt), and the entire site concatenated as one markdown document lives at [`llms-full.txt`](/HoBo/llms-full.txt). Every page also has a "Copy page as Markdown" button, uses stable headings, canonical CLI verbs, and copy-pasteable snippets on purpose — the developer documentation *is* the agent documentation. ## Agents are principals, not scripts The standard failure mode elsewhere: an agent runs with a human's long-lived API key, invisible to audit and unbounded by scope. HoBo's model (the full brokered-identity flow below is contract-pinned design; the shipping `hobo-mcp` binary runs today with a deny-by-default, scoped local-operator identity and audits every decision): - **Own identity.** Agents get workload identity (SVID/OIDC), not borrowed keys. - **Brokered, short-lived tokens.** Secretless-first — credentials are issued per task, scoped to the task, and expire fast. - **Self-provisioning within policy.** A machine-readable **capability standard** describes what each capability needs, so an agent can request *exactly* the credentials a job requires — and policy decides whether it gets them. - **Every action is an event.** Token issuance, deploys, migrations — an agent's whole working day lands in the same append-only, replayable log as everything else. "What did the agent do?" is a query. ## One surface, three doors Every HoBo capability is reachable three ways, with identical semantics: | Door | For | Notes | |---|---|---| | **SDK** | code | The same typed objects the [primitives](/HoBo/primitives/) docs describe. | | **CLI** | humans & scripts | The [canonical verb surface](/HoBo/guide/cli/) — deliberately small, stable to memorize. | | **MCP** | agents | Ships today as the `hobo-mcp` stdio binary: the canonical verb set as tools, parity contract-tested — see [the MCP surface](/HoBo/agents/mcp/). | No agent-only side entrance with different rules: if policy would stop a human doing it, it stops an agent doing it, and vice versa. ## The Crew: operators included HoBo ships a built-in roster of operator agents — the **Crew**: SRE, endpoint-monitor, GitOps, and platform-engineer operators that run as autonomous, policy-gated services in the Yard. HoBo ships the operators, not just the APIs they'd call. Your agents can work alongside them, under the same identity model. ## Agents and continuous migration The guiding principle — *find the cheapest home that passes policy, keep looking* — is agent-shaped by design. The Dispatcher that plans migrations is itself an optimizing agent under policy gates, and third-party agents can participate: propose placements, watch spend, pre-approve moves within bounds you set. The event log means every optimization an agent makes is attributable and reversible. ## Generating new homes Missing a provider adapter? Agents can invoke the **generation pipeline**: synthesize an adapter for a port, then have the port's conformance suite certify it. A certified adapter is a certified adapter regardless of author — human or machine. More homes on the map, cheaper moves for everyone. The full extension surface — every implementable port, the adapter-authoring path, the conformance bar, and the ACP/A2A protocol-connector integration points — is documented in [Custom providers](/HoBo/customizations/providers/). ## Contracts agents can read (and test) When you call `hobo verify` / can-deploy or inspect a BoundaryContract, look for optional **effects** and **resilience** fields: - **EffectSpec** — purity + domain I/O tags. Pure contracts isolate with zero infrastructure; network/storage domains imply harness mocks. Effect widening is a compatibility signal. - **ResilienceProfile** — secondary retry/timeout/circuit-breaker metadata. Deterministic schedules only; **never** retry a `policy.Decided` deny or a terminal failure. Prefer reconstructing state from event facts over process memory. - **EffectManifest** — exportable summary for policy, SRE tooling, and training trajectories (with redaction). Durable Camp storage defaults to a **single-owner SQLite partition** behind EventStore ports. Do not assume multi-writer shared filesystems; do not invent parallel event-store engines when the ports already exist. ## Start here - [The MCP surface](/HoBo/agents/mcp/) — the shipping `hobo-mcp` binary: stdio transport, the canonical verb set as tools, `HOBO_MCP_TOOLS` scoping, deny-by-default audited decisions. - [hobo-code](/HoBo/guide/hobo-code/) — the interactive Bo session. Not a `hobo` verb. - [`llms.txt`](/HoBo/llms.txt) — the machine-readable site briefing. - [Architecture](/HoBo/architecture/) — package layout, durable EventStore default, contract effects. - [Testing](/HoBo/guide/testing/) — effect-aware isolation and offline durability tests. - [Getting started](/HoBo/guide/getting-started/) — the same ten-minute path humans take; it works verbatim for agents. --- # The MCP surface HoBo exposes its control plane over **MCP (Model Context Protocol)** — and this surface **ships today**. The `hobo-mcp` binary serves every canonical CLI verb as an MCP tool over stdio, and every tool call dispatches through the *same command registry* as the `hobo` binary itself. Anything the [CLI verb surface](/HoBo/guide/cli/) can express, an agent can call over MCP. It isn't a chat integration bolted onto the side — it's a second transport into the same surface. ## Running it `hobo-mcp` ships alongside the `hobo` binary and speaks the MCP **stdio transport** — newline-delimited JSON-RPC 2.0 (`initialize`, `tools/list`, `tools/call`, `ping`). Point any MCP client at the binary: ```json { "mcpServers": { "hobo": { "command": "hobo-mcp" } } } ``` Like the CLI, it defaults to the **local lane**: `plan`, `up`, and `down` converge against an offline Yard runtime with a durable SQLite journal — no cloud account required. Live deploy lanes are **credential-gated by environment**: Cloudflare (`CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID`), Vercel (`VERCEL_TOKEN`), or live IaC credentials. ## Tools = the canonical verbs The tool surface is **derived from the canonical CLI verb set**, one tool per verb — `up`, `down`, `plan`, `dev`, `station`, `trails`, `replay`, and the rest (see [the hobo CLI](/HoBo/guide/cli/)). Parity is a contract, not a convention: a structural parity check fails whenever the tool set diverges from the verb set, so there is no MCP-only subset and no MCP-only side door in either direction. A tool call becomes `hobo --json` over the shared dispatch path, and the result is the same JSON envelope the CLI emits, returned as inert data. Tool inputs accept `cwd` (resolve the project directory), `camp` (target camp), and `args` (extra verb arguments). ## Auth: deny by default, audited always Authorization is checked **before** dispatch, and every decision is recorded: - **Deny-by-default scopes.** Each tool maps to a capability scope. A call outside the caller's scope is refused (`MCP_SCOPE_DENIED`); a tool that isn't part of the derived surface is refused (`MCP_UNKNOWN_TOOL`). Fail closed, never fall through. - **Every decision is an audit event.** Allow or deny, each tool-call decision is recorded with the principal, tenant, tool, effect, and reason attached. - **Scope down with `HOBO_MCP_TOOLS`.** Set a comma-separated verb list (e.g. `HOBO_MCP_TOOLS=plan,inspect,logs`) and the served surface shrinks to exactly those tools — every other verb is out of scope for that server process. - **Local-operator trust model.** Over stdio, `hobo-mcp` acts with the invoking user's authority — the same trust model as running `hobo` directly — bounded by the scoped tool surface above. `HOBO_MCP_SUBJECT` / `HOBO_MCP_TENANT` name the principal and tenant the audit trail records. ## What agents typically do with it | Task | Tool | |---|---| | "Deploy this bindle" | `up` — the same convergence loop as `hobo up`; local lane by default | | "What would change, and roughly what would it cost?" | `plan` — read-only dry run | | "Tear the preview back down" | `down` | | "What happened on the last run?" | `inspect` / `logs` | | "Anything drifting?" | `drift` | ## Designed, not yet shipped The fuller agent-identity design — workload identity (SVID/OIDC), brokered short-lived per-task credentials, and a machine-readable capability standard for scope discovery — is contract-pinned design, tracked on [the trail map](/HoBo/guide/status/). What ships now is the surface above: full verb/tool parity, deny-by-default scoping, and audited decisions. ## Reading the docs as an agent This site is maintained as a first-class agent surface: - [`llms.txt`](/HoBo/llms.txt) is the canonical machine-readable briefing — start there. - Headings are stable; CLI examples use only canonical verbs; code snippets are complete enough to run, not fragments. - The [primitives](/HoBo/primitives/) pages define the exact vocabulary the tool surface uses — bindle, camp, yard, junction mean the same thing to the parser as to you. --- # Observability HoBo’s primary observability substrate is the **event log** plus rebuildable projections. Metrics and traces export — they do not replace the log as authority. ## In this section | Page | Topic | |---|---| | [Agent audit & replay](/HoBo/observability/agent-audit/) | Tool calls, token grants, handoffs as facts | | [Session Replay](/HoBo/reference/session-replay/) | Privacy-first DOM/session reconstruction | | [Metering](/HoBo/reference/metering/) | Usage facts and showback projections | | [Streamer](/HoBo/reference/streamer/) | High-volume frames outside the domain log | | [Endpoint Monitor](/HoBo/operations/endpoint-monitor/) | Synthetics and SLO burn | ## Principles 1. “Why did the system do X?” → **replay**, not SSH folklore. 2. Crew and agent actions are facts. 3. Sandbox lanes are labeled so experiments do not pollute baseline views. ## Related [Architecture](/HoBo/architecture/) · [Troubleshooting](/HoBo/troubleshooting/) · [Operations](/HoBo/operations/) --- # Agent audit & replay If an agent acted, the log can explain it. “The model said so” is not an audit trail. ## What is recorded | Fact class | Examples | |---|---| | Dispatch | `crew.AgentDispatched` (operators) / session open | | Tools | MCP invocations + results (redacted) | | Tokens | Grants by **reference** | | Policy | `policy.Decided`, obligations | | Guardrails | `guardrail.Blocked` | | Handoffs | A2H request/resolve | ## Replay questions - What tools ran between T0 and T1? - Which policy version allowed the destroy proposal? - What was the fence/deployment correlation id? Use `hobo inspect`, event exports, and `hobo attest` for external GRC — not screenshots of secrets. ## Related [Credentials](/HoBo/security/credentials/) · [Event families](/HoBo/reference/events/) · [Crew shared model](/HoBo/operations/shared-model/) --- # Session Replay **Session Replay** reconstructs user DOM/sessions (and Kit-level interactive capture) for debugging and support. It is **not** control-plane event-log replay and **not** auth `session.*`. ## Naming split (critical) | Concept | Meaning | Family / surface | |---|---|---| | **Event-log replay** | Fold durable facts for projections/audit | EventStore / `hobo inspect` | | **Auth session** | Login/refresh/revoke | `session.Started` / `Refreshed` / `Revoked` | | **Session Replay** | DOM/session reconstruction | `replay.*` + Session Replay | | **App Kit** | Bindle-declared app surface (`api`, `web`, `service`, …) | `hobo/bindles` `app kit kinds` | Heavy rrweb streams **never** enter the control-plane event log — only content-addressed blob refs. ## Defaults | Environment | Default | |---|---| | Local / dual fixtures / Station air-gap / every **App Kit** | **On** (zero Bindle config) | | Production / staging | **Off** unless OpenFeature `hobo.session-replay` or sample rate | | Kill switch | `HOBO_REPLAY=0` (or flag false) | ## Deep Bindle integration (App Kits) Every **App Kit** receives a default platform annotation `session-replay` at construction — **not only** `web`: | App Kit | Kind | Default Session Replay | |---|---|---| | API Kit | `api` | **On** | | Web Kit | `web` | **On** | | Service Kit | `service` | **On** | | UI Kit | `ui` | **On** | | Realtime Kit | `websocket` | **On** | | Job Kit | `job` | **On** | | Component Kit | `wasm-component` | **On** | **Infra Kits** (`kv`, `bucket`, `queue`, `topic`, `secret`, `data`, …) are **not** entrypoint hosts — no default Session Replay. Local Yard / `hobo dev` wraps every **HTTP App Kit** (`api`, `web`, `service`, `ui`) so HTML gets a privacy-first bootstrap and `/__hobo/replay/*` serves ingest/list — **no app tracker code required**. JSON API Kits still expose the ingest path on the same wrap. | Piece | Package / surface | |---|---| | Annotation helper | `sessionReplay({ enabled?, sampleRate? })` from `hobo/bindles` | | Kit taxonomy | `app kit kinds`, `isAppKitKind`, `appKitLabel` | | Runtime plan + wrap | Session Replay bindle integration | | Local host | `hobo dev` / Yard entrypoints | | Ingest path | `/__hobo/replay/*` | | Response marker | `x-hobo-session-replay: 1` when HTML wrap applied | ```ts import { bindle, sessionReplay } from "hobo/bindles"; export default bindle("shop", (b) => { const api = b.api(() => apiHandler); // API Kit — Session Replay default-on const web = b.web(() => storefront); // Web Kit — Session Replay default-on const worker = b.service(() => worker); // Service Kit — Session Replay default-on const cache = b.kv(); // Infra Kit — no default Session Replay // opt out one Kit: // const api = b.api(() => apiHandler).with(sessionReplay({ enabled: false })); return { api, web, worker, cache }; }); ``` ## Privacy (client-side before send/persist) - Mask password / email / tel inputs - Mask `[data-hobo-mask]`, `.hobo-mask`, payment autocomplete fields - Redact `Authorization` / `Cookie` in network capture - Continuous-identity **`principalRef` only** — no free-text PII ## CLI / MCP ```text hobo replay doctor hobo replay list hobo replay get hobo replay summarize hobo replay open hobo replay retention ``` MCP tool parity via canonical verbs. Player surfaces are read-oriented; mutation stays on CLI under policy/PAM. `hobo doctor` lists contract boundary ids. ## Event family (`replay.*`) `Started` · `ChunkIngested` · `Ended` · `ErrorLinked` · `RetentionApplied` · `Accessed` · `Disabled` · `Metered` ## Composition | Composes with | How | |---|---| | **App Kits** | Default annotation + host wrap on every Kit | | **Station** | Isolation + local path | | **Harness / OpenFeature** | Prod enablement + sampling | | **Trails** | Platform recipe `replay.retention` | | **Continuous identity** | Principal refs only | | **Metering** | Ingest bytes, storage, player access | | **PAM** | Access policy for open/list | | **PACT** | Integration boundaries via the replay HTTP surface | ## Performance budget (targets) | Metric | Budget | |---|---| | Batch interval | 1000 ms | | Batch size | 50 events | | Main-thread per flush | ≤ 16 ms | | Tracker overhead beyond rrweb | ≤ ~15 KB gzipped | ## Honest scope - Live multi-cloud object storage + Junction federation: **ports present**; dual fixtures prove the path - Full OpenReplay Assist / Helm stack: **optional**, not default - High-volume product analytics warehouse: **residual** ## Related [Bindles](/HoBo/primitives/bindles/) · [App Kits](/HoBo/primitives/app-kits/) · [Trail map](/HoBo/guide/status/) · [Harness](/HoBo/reference/harness/) · [CLI](/HoBo/reference/cli/) · [Event families](/HoBo/reference/events/) · [Dual fixtures](/HoBo/testing/dual-fixtures/) · [Stations](/HoBo/primitives/stations/) --- # Metering Metering turns **replayable facts** into usage evidence for showback, FinOps projections, dispute support, and ranking — **not** silent invoice invention. ## Model 1. Workloads and control plane emit usage-related facts (`meter.UsageRecorded` and related; also derived from `run.*`, `deployment.*`, `resource.*`, `gen.*`). 2. A **metering bus** fans out to sinks (in-process, export, warehouse adapters). 3. FinOps views (showback, anomalies, forecasts) are **projections** over facts. 4. Humans and the FinOps Crew **advise**; breakers, tiers, and policy **enforce**. ## Dimension matrix (required for M0 contracts) | Dimension | Why | |---|---| | Tenant | Showback isolation + support lookup | | Workspace / Project | Customer grouping and policy reconciliation | | Environment | Prod vs preview vs test separation | | Camp / Bindle | Operational and package grouping | | Run / Agent | Runtime + AI attribution and dispute lineage | | Team | Org-level reporting | ## Fact classes (examples) | Class | Typical family | Sensitive? | |---|---|---| | Control-plane operation | `deployment.*` | No | | Runtime compute | `run.*` | No | | Storage / egress | `resource.*` | No | | Generation run | `gen.*` | Often yes | | Support adjustment | `meter.*` | Yes | Sensitive classes require redaction/retention rules before customer export. ## Legal / ranking hygiene - Third-party **price and free-tier** facts are **ranking input** with source URL + human-checked date — never fabricated “vendor signed” attestations. - Free-tier headroom is **never pooled** across tenants. - Rotation is **across providers**, never across accounts within a provider to reset free tiers. - Metering/billing data stays **tenant-scoped**. ## FinOps Crew The FinOps operator agent reads metering projections, flags anomalies, and proposes commitment/move advice. It does **not** auto-spend or auto-destroy. See [FinOps Crew](/HoBo/operations/finops/). ## Related [Policy](/HoBo/reference/policy/) · [FinOps role](/HoBo/roles/finops/) · [Security](/HoBo/security/) --- # Troubleshooting Start **read-only**, then gated mutate. Prefer CLI verbs that emit machine-readable reasons. ## In this section | Page | Topic | |---|---| | [Common failures](/HoBo/troubleshooting/common-failures/) | doctor, can-deploy, STALE_FENCE, schema, Station, OPFS | | [Local debugging](/HoBo/troubleshooting/local/) | `hobo dev`, local Camp, dual fixtures | | [Remote development](/HoBo/troubleshooting/bridge/) | Bridge modes: live / mock / bridge / replay / shadow / fault | | [Remote debugging](/HoBo/troubleshooting/remote/) | inspect, logs, incidents against real camps | | [Agent audit & replay](/HoBo/observability/agent-audit/) | What the agent did — in the event log | | [Errors & codes](/HoBo/reference/errors/) | Stable machine codes (reference) | ## Quick map | Symptom | First command | |---|---| | Local env weird | `hobo doctor` | | Deploy blocked | `hobo verify` / `hobo can-deploy` | | Policy deny | inspect `policy.Decided` — **do not retry** | | Append OCC | reload & retry | | `STALE_FENCE` | re-acquire lease | | Control plane refuses | `hobo station status` | | Agent can’t tool | scopes + MCP auth | ## Related [Observability](/HoBo/observability/) · [CLI](/HoBo/reference/cli/) · [Anti-patterns](/HoBo/testing/anti-patterns/) --- # Common failures Start with **read-only diagnosis**, then gated mutate. Prefer CLI verbs that emit machine-readable reasons. ## Quick map | Symptom | First command | Likely page | |---|---|---| | Local env weird | `hobo doctor` | this page · [CLI](/HoBo/reference/cli/) | | Deploy blocked | `hobo verify` / `hobo can-deploy` | [Contracts](/HoBo/reference/contracts/) | | Policy deny | inspect `policy.Decided` | [Policy](/HoBo/reference/policy/) | | Append fails OCC | reload & retry | [EventStore](/HoBo/reference/eventstore/) | | `STALE_FENCE` | re-acquire lease | [ActorHost](/HoBo/reference/actor-host/) | | DB won’t open | schema version / OPFS | [Schema](/HoBo/reference/schema/) | | Control plane refuses | `hobo station status` | [Stations](/HoBo/primitives/stations/) | | Identity dual weird | `hobo identity status` | [Identity](/HoBo/security/identity/) | | Agent can’t tool | scopes + MCP auth | [MCP](/HoBo/agents/mcp/) | ## Gatekeeper denied 1. Read `gatekeeper.Decided` reasons (`gate` field). 2. Contracts → fix producer/consumer skew (`can-deploy`). 3. Policy → change pack or satisfy obligation — **do not retry** denials. 4. Cost → quota/budget. 5. Effects/resilience → visibility only unless configured as hard gate. ## Durability failures | Code | Action | |---|---| | `OPTIMISTIC_SEQUENCE_CONFLICT` | Reload aggregate; retry decide/append | | `STALE_FENCE` / `MISSING_FENCE` | Lease discipline — not OCC retry | | `UNSUPPORTED_SCHEMA_VERSION` | Upgrade runtime to match DB | | `PERSISTENCE_UNAVAILABLE` | Browser without OPFS while durable required — change mode or environment | | `SQLITE_BUSY` | Single-owner violation or short busy timeout — check writers | ## Station / broker Missing Station fails closed. `hobo station init` then set `defaultStation` / `activeRail`. Broker receipts must pin `stationId` + `railHash`. ## Still stuck - Replay the stream with `inspect` - Check Crew handoffs for open obligations - File evidence with `attest` exports — not screenshots of secrets ## Related [Doctor in CLI guide](/HoBo/guide/cli/) · [Errors](/HoBo/reference/errors/) · [Anti-patterns](/HoBo/testing/anti-patterns/) --- # Local debugging ## First loop 1. `hobo doctor` — environment, credential, and portability hazards 2. `hobo dev` — offline local Camp / in-process handlers 3. `hobo plan` / `hobo verify` — dry-run and Gatekeeper without cloud mutate 4. Project tests — effect isolation, durability, contracts (see [Testing](/HoBo/testing/)) ## Useful defaults | Concern | Local default | |---|---| | EventStore | SQLite single-owner partition | | Secrets | Schema-only view + fixtures — no plaintext in agent context | | Session Replay | On for local dual fixtures (`HOBO_REPLAY=0` kills) | | Named endpoints | `.hobo.localhost` | ## When local is green but remote fails Compare Bridge modes and credentials — [Remote development](/HoBo/troubleshooting/bridge/) · [Remote debugging](/HoBo/troubleshooting/remote/). ## Related [Common failures](/HoBo/troubleshooting/common-failures/) · [Durable EventStore tests](/HoBo/testing/durability/) --- # The Bridge > *Tag your traffic, fork your boundary, keep everyone else live.* Bridge to Kubernetes had one great idea: develop **one** service on your laptop while it participates in the **real** deployed environment — your requests detour to your machine, everyone else's don't. Its limits were structural: it worked by intercepting pod traffic, so it only worked on Kubernetes. HoBo's version keeps the idea and drops the requirement. Every service boundary is a WIT port whose peers are PACT-emulable, and every boundary supports per-request routing modes (`live · mock · bridge · replay · shadow · fault`). Forking traffic to your laptop is just one more route — applied **at the WIT port boundary**, not in any orchestrator — so it works identically on any provider, a mix of them, or your own metal. ## The three moves **1. Tag your traffic.** A dev session is a string — your branch name is the natural one. Session tags ride with requests (as baggage on wire transports; explicitly in-process). Untagged traffic can never hit a fork. **2. Bridge the boundary.** Point the boundary at the implementation you're editing, scoped to your session: ```ts import { createBoundaryRouter } from "hobo/runtime"; const router = createBoundaryRouter(); const payments = router .boundary("payments") .live(deployedPayments); // the version running in the camp await payments.bridge(myLocalPayments, { session: "tyler/feature-refunds", contract: paymentsPact, // the boundary's PACT interactions }); ``` **3. Work against reality.** Your requests take the bridge; the rest of the team's — and production's — stay live: ```ts await payments.call(charge, { session: "tyler/feature-refunds" }); // → your laptop await payments.call(charge); // → the deployed camp ``` Tear it down with `payments.unbridge("tyler/feature-refunds")`. A bridge with **no** session forks the whole boundary — the classic "route this service to me" mode — and `unbridge()` restores it. ## The part Bridge-to-Kubernetes never had A bridge is **contract-gated**. Before your local implementation carries a single request, it's verified against the boundary's contract — the PACT interactions every implementation of that boundary must satisfy, using the same subset matching as provider verification. A local build that would break consumers **never activates**; the error tells you which interactions failed, and your session's traffic keeps flowing to the live route: ```ts await payments.bridge(brokenLocalBuild, { session: "tyler/feature-refunds", contract: paymentsPact, }); // BridgeContractError: bridged implementation for boundary "payments" // fails its contract (1 interaction): for input {"amount":42,…} // expected {"status":"charged",…} ⊆ actual, got {"status":"declined",…} ``` And because **recording is always on**, every call your bridged session serves is captured as a specification-by-example trace — your dev session's traffic becomes tomorrow's contract candidates and regression inputs. ## Composing with preview environments A dev branch's preview environment is a camp with your branch's session tag pre-applied. The two compose naturally: - **Preview env alone** — the whole branch deployed, isolated, shareable. - **Preview env + bridge** — the branch deployed *except* the one service you're actively editing, which detours to your laptop. Save a file, hit the endpoint, no redeploy. - **Shared env + bridge** — the lightest loop: nothing deployed at all for your branch; just your session forked out of the shared environment. ## The rest of the modes The bridge is one routing mode of six, and they compose per boundary: | Mode | What it does | Use for | |---|---|---| | `live(handler)` | the deployed route | normal traffic | | `mock(examples)` | serve PACT examples, no dependency | [unit tests](/HoBo/guide/testing/), offline dev | | `bridge(handler, { session })` | session-scoped fork | local/remote dev | | `replay()` | serve recorded outputs | deterministic re-runs | | `shadow(candidate)` | duplicate traffic, compare, discard | testing the next version on real load | | `fault({ message, session })` | scoped failure injection | [chaos with no blast radius](/HoBo/testing/chaos/#session-scoped-faults) | `router.status()` reports every boundary's routing state — who's bridged, what's shadowed, which faults are armed — so "what is my environment actually doing right now" is one call. ## Honest status The boundary router above ships today with in-process session tagging, and its tests are the executable example of everything on this page. Session baggage on wire transports, the `hobo bridge` CLI verb (open tunnel, register fork, stream contract verification), automatic pact-broker → bridge-contract conversion, and branch-tagged preview environments are staged next — the design is pinned even where the plumbing isn't. ## Related - [Testing](/HoBo/guide/testing/) — the same modes, pointed at tests. - [Core concepts](/HoBo/guide/concepts/) — WIT/WASM ports the routing lives in. - [Custom providers](/HoBo/customizations/providers/) — why boundaries are provider-agnostic in the first place. --- # Remote debugging ## Read-only first | Command | Use | |---|---| | `hobo inspect` | Time-travel projections (`run`, `deploy`, `topology`) | | `hobo logs` | Streamer live-tail | | `hobo identity status` | Continuous identity dual-path | | `hobo trails status\|inspect` | Migration recipe state | | `hobo drift` | Drift report (remediation still gated) | ## When you must change something - Prefer canary / progressive paths ([CI/CD](/HoBo/cicd/)) - Production writes often `require-approval` - SRE/runbooks: [Operations](/HoBo/operations/) ## Agent “what happened?” [Agent audit & replay](/HoBo/observability/agent-audit/) — tool calls and token grants are facts. ## Related [Common failures](/HoBo/troubleshooting/common-failures/) · [Observability](/HoBo/observability/) · [Bridge](/HoBo/troubleshooting/bridge/) --- # Customizations HoBo is ports-and-adapters all the way down. Customize by **registering** conforming engines — not by forking the control plane. ## In this section | Page | Topic | |---|---| | [Custom providers](/HoBo/customizations/providers/) | Capability providers behind WIT ports + conformance | | [IaC engines](/HoBo/customizations/iac/) | Pulumi, Crossplane, Terraform/OpenTofu adapters | | [Messaging integrations](/HoBo/customizations/messaging/) | Kafka, Dapr, and the messaging port | | [Connectors](/HoBo/customizations/connectors/) | Conductor connector registry plugins | | [Component hosts](/HoBo/reference/component-hosts/) | wasm / oci / cloud-native hosts (reference) | ## Rules 1. Every adapter ships conformance suite for the ports it claims. 2. Secrets stay refs — connectors use `credentialRef`, not ambient keys. 3. Custom **ACP agents** live under [The Crew](/HoBo/crew/) — not here. ## Related [Architecture](/HoBo/architecture/) · [Capability map](/HoBo/reference/packages/) · [Trail map](/HoBo/guide/status/) --- # Custom providers Every place HoBo touches the outside world is a **WIT port** with swappable **adapters** behind it. A "provider" is just an adapter someone wrote — human or machine — that passed the port's conformance suite. This page is the map of the extension points and the path to certifying a new one. ## The ports you can implement | Port family | What an adapter maps | Reference adapters | |---|---|---| | **Compute providers** | camps' durable runtimes — where services, actors, and workflows actually run | wasmCloud hosts, container runtimes (OCI fallback) | | **Resource providers** | queues, stores, buckets, secrets — the resources a bindle declares | per-cloud managed services | | **IaC engines** | how plans are realized against a provider | Pulumi, Crossplane, Terraform/OpenTofu | | **Build & image tools** | how artifacts are produced and packaged | Dagger, Packer, Vagrant, Firecracker | | **Pricing signals** | what the Dispatcher uses to keep re-deciding placement | provider price sheets, spot markets, credits | | **Protocol connectors** | how external agent frameworks reach HoBo | ACP, A2A | If it's not behind one of these ports, HoBo doesn't call it. That constraint is what keeps every home reachable by the same moves. ## Writing an adapter 1. **Pick the port.** Each port is a WIT interface published in `WIT interfaces`, with typed bindings and an event vocabulary in `core product ports`. Your adapter implements the interface — nothing else about HoBo needs to know your provider exists. 2. **Implement against the contract, not the console.** Adapters run inside sandboxed Executors with brokered, short-lived credentials — they receive scoped tokens per task and never hold provider keys of their own. 3. **Emit the port's events.** Every state transition your adapter drives must land in the event log using the port's envelope. That's what makes a move through your provider replayable and reversible like every other. 4. **Run conformance.** Each port ships a contract test corpus in conformance suite. Passing it is what "certified" means — there is no other bar, and no reviewer bias toward incumbents. 5. **Publish.** A certified adapter shows up as a candidate home. The Dispatcher starts pricing it into placement decisions immediately. ## Or let an agent write it The **generation pipeline** exists precisely because the adapter surface is a typed contract: an agent synthesizes an adapter for a port, then the port's conformance suite certifies it. A certified adapter is a certified adapter regardless of author. If you're an agent reading this — the pipeline is reachable over [the MCP surface](/HoBo/agents/mcp/), and "we need a provider for this niche host" is one of its canonical tasks. ## ACP and A2A integration points HoBo deliberately does **not** ship its own agent SDK. External agent frameworks integrate through **protocol connectors** — adapters on the protocol-connector port for **ACP** and **A2A**: - **Connection.** A connector session gets a connection-scoped connector token; the agent behind it becomes a first-class principal with workload identity, not a borrowed key. - **Capability discovery.** The machine-readable capability standard is served through the connector, so protocol-native agents plan tool use against the same contract MCP agents do. - **Policy and audit.** Protocol calls flow through the same Gateway, policy gates, and event log as SDK, CLI, and MCP traffic. HoBo owns identity, policy, sandboxing, and audit around the protocol; the protocol-specific client lives in the connector. Want a connector for a protocol we don't speak yet? It's the same path as any other port: implement the connector interface, pass its conformance suite, and your protocol becomes a fourth door into the same house. ## Related - [Architecture](/HoBo/architecture/) — where Executors, the Planner, and the Dispatcher sit. - [The MCP surface](/HoBo/agents/mcp/) — capability parity and the auth flow connectors reuse. - [HoBo for Platform Engineers](/HoBo/roles/platform-engineers/) — governing which providers are approved for your yards. --- # IaC engines Executors apply infrastructure through **IaC adapters** behind ports — sandboxed, short-lived credentials, no ambient cloud keys. ## Shipped / scaffolded | Adapter family | Role | Status | |---|---|---| | **Pulumi** (Pulumi IaC engine) | Automation API — primary path | Live gated by `provider live credentials` | | **Crossplane** | Kubernetes-shaped control | Design / stage-gated | | **Terraform / OpenTofu / CDKTF** | Familiar plan/apply | Design / adapter scaffolds | ## Rules - Executors run in Firecracker-class isolation when available - Identity brokers **scoped tokens per resource + action** - Drift remediation is gated the same as deploy - Never hand long-lived provider keys to the app or the executor ambient env ## Related [Custom providers](/HoBo/customizations/providers/) · [Control plane](/HoBo/architecture/control-plane/) · [Stations & Conductor](/HoBo/primitives/stations/) --- # Messaging integrations Beyond the Camp **Queue** port, HoBo has a **messaging** surface for brokered topics/streams with conformance and doctor hooks. ## Entry points | Piece | Surface | |---|---| | Messaging port | [Messaging reference](/HoBo/reference/messaging/) · Bindle `topic()` | | Kafka provider | Kafka-compatible capability provider | | Dapr runtime | Optional Camp runtime for pubsub / actors | | CLI | `hobo doctor` reports messaging readiness; plane details in the messaging reference | ## Related [Customizations](/HoBo/customizations/) · [Queue port](/HoBo/reference/queue/) · [Conformance](/HoBo/testing/conformance/) --- # Connectors The Conductor acts on the outside world through a **connector registry** (GitHub App, Pulumi, cloud accounts, delivery systems). Entries hold **capability metadata + credentialRef** — never long-lived secrets in the catalog. ## Rules 1. Register → sign/provenance/conformance when required 2. Broker short-lived access via Station connections + identity 3. Fail closed when the connector or policy engine is unavailable ## Related [Stations & Conductor](/HoBo/primitives/stations/) · [Custom providers](/HoBo/customizations/providers/) · [Security](/HoBo/security/) --- # Testing overview > *Trust the bindle, test the journey — without a cloud account until you want one.* HoBo collapses a lot of “special testing tools” into one idea: **every boundary is typed and contract-backed**, so you can point it at a different **mode** (live, mock, replay, bridge, shadow, fault) without rewriting the app. The patterns below are deliberate product features, not ad-hoc test helpers. ## Mental model | Idea | What it buys you | |---|---| | **Contracts bound shape** | Consumer-driven expectations; providers can grow without breaking consumers | | **Boundary modes** | Same production code path, different routing (mock / live / replay / bridge / shadow / fault) | | **EffectSpec** | Pure work isolates with zero infra; domain I/O drives automatic mocks | | **ResilienceProfile** | Fault-injection guided by the contract; never retries policy denials or terminal failures | | **Event log** | Replay-equals-live; inspection without re-running side effects | | **SQLite journal** | Real durability semantics offline (CAS, fencing, atomic effects) | ## Layers (read in this order) 1. **[Unit tests & boundaries](/HoBo/testing/unit/)** — unit tests; boundaries auto-mock from contract examples. 2. **[Aggregates & CQRS](/HoBo/testing/aggregates/)** — pure decide, fenceToken, conflict vs invariant. 3. **[Contracts & PACT](/HoBo/testing/contracts/)** — consumer/provider executable contracts; subset matching. 4. **[Effect-aware isolation](/HoBo/testing/effects/)** — pure vs domain-effect harness modes. 5. **[Dual fixtures](/HoBo/testing/dual-fixtures/)** — pure path vs effect path from the same metadata. 6. **[Durable EventStore](/HoBo/testing/durability/)** — stream-head CAS, fencing, positions, restart. 7. **[Migration fixtures](/HoBo/testing/migration/)** — legacy schema → current, fail-closed integrity. 8. **[Queue, timer, outbox](/HoBo/testing/queue-outbox/)** — lease fences, once-after-restart timers. 9. **[Deployment rehearsal](/HoBo/testing/deployment/)** — `hobo plan`, local lane, Gatekeeper. 10. **[Bridge modes](/HoBo/testing/bridge-modes/)** — live/mock/bridge/replay/shadow/fault. 11. **[Chaos & availability](/HoBo/testing/chaos/)** — session faults, synthetics, shadow. 12. **[Property & model](/HoBo/testing/property/)** — journal ≡ reference model. 13. **[Behavior & Cucumber](/HoBo/testing/behavior/)** — Gherkin evidence, not live E2E. 14. **[Benchmarks](/HoBo/testing/benchmarks/)** — catch algorithmic regressions. 15. **[Conformance & golden](/HoBo/testing/conformance/)** — port suites + deploy golden gate. 16. **[Anti-patterns](/HoBo/testing/anti-patterns/)** — what not to do. ## What you should *not* do - Hand-write long-lived mocks that drift from contracts — use contract examples / effect harness mocks. - Treat `COUNT(*)` stream length or multi-writer shared SQLite as “the real” store — the product uses stream heads and single-owner partitions. - Retry `policy.Decided` denials or terminal WIT failures in tests “to be resilient” — that is a **bug** in the product model. - Run broad full-repo test sweeps to discover code — search first, then narrow package tests. ## Related - [Architecture](/HoBo/architecture/) — EventStore default, package map. - [The Bridge](/HoBo/troubleshooting/bridge/) — local/remote development on the same modes. - [Reference: effects](/HoBo/reference/effects/) · [EventStore](/HoBo/reference/eventstore/) · [contracts](/HoBo/reference/contracts/) --- # Unit tests & boundaries Your services, actors, and workflows are ordinary TypeScript functions. Dependencies are **boundaries**. Boundaries should be mocked from **contract examples**, not invented response shapes. ## Boundary router pattern ```ts import { createBoundaryRouter } from "hobo/runtime"; import { expect, test } from "node:test"; const router = createBoundaryRouter(); const payments = router.boundary("payments").mock([ { input: { orderId: "o-1", amount: 42 }, output: { orderId: "o-1", status: "charged" }, }, ]); test("checkout charges the cart", async => { const receipt = await payments.call({ orderId: "o-1", amount: 42 }); expect(receipt.status).toBe("charged"); }); ``` - Matched inputs return the example’s output. - Unmatched inputs still get a correctly-typed canned response (first example) so tests stay unblocked. - Greenfield **declared operations** work the same way: contract-derived stubs serve day-one traffic and record specification-by-example traces. ## Trainables A `@trainable` / `@operation` contract’s examples double as its **eval suite**. Resolve the same instance by token in a test; those assertions are what the training loop reuses later. ## Aggregate / CQRS unit style Write-model aggregates fold streams and emit past-tense facts. Unit tests typically: 1. Use an in-memory EventStore or a temp SQLite partition. 2. Acquire a lease (in-memory ActorHost for pure unit speed, durable host for lease persistence). 3. Assert appended facts and folded state — never assert on mutable read models the aggregate should not touch. ## Prefer package-scoped runs ```bash run the package tests run the package tests ``` Do not use full-repo test globs as a discovery tool. Search first. ## Next - [Contracts & PACT](/HoBo/testing/contracts/) - [Effect-aware isolation](/HoBo/testing/effects/) --- # Aggregates & CQRS Aggregates are **single-writer actors**. Tests prove pure business rules and concurrency semantics without a cloud account. ## What to unit-test first | Concern | Assert | |---|---| | Happy path decide | Given history + command → expected facts | | Invariant reject | `AGGREGATE_INVARIANT` (or domain equivalent) — **not** retryable | | OCC conflict | Stale `expectedSeq` → conflict; reload & retry succeeds | | Fence | Stale/missing fence fails closed — **not** treated as OCC | | Idempotency | Same key does not double-apply effects | | Fold purity | Fold(events) has no I/O | ## Pattern ```text history facts → fold state → decide(command) → facts append(expectedSeq, fenceToken?) → store accepts/rejects projection.fold(facts) → read model ``` Use in-memory EventStore + ActorHost for pure tests; use SQLite partition tests when proving durability (see [Durability](/HoBo/testing/durability/)). ## Multi-aggregate Cross-aggregate workflows are eventually consistent. Test sagas with correlation ids and failure compensation — do not pretend distributed transactions exist. ## Anti-patterns - Calling wall-clock inside pure decide - Treating `STALE_FENCE` as “retry with same token” - Asserting on projection without replaying from facts - Sharing one aggregate stream across unbounded entity ids ## Related [Data plane](/HoBo/architecture/data-plane/) · [ActorHost](/HoBo/reference/actor-host/) · [Unit tests](/HoBo/testing/unit/) --- # Contracts & PACT Every service boundary owns **executable contracts**. HoBo’s source of truth is the **Contract IR** (`BoundaryContract`). PACT documents are the common **export/import shape** and the local test harness format — not a parallel product model. ## Project layout ```text your-app/ contracts/ inventory.md pacts/.pact.json consumer/.contract.test.ts provider/.provider.test.ts provider/fixtures/ ``` Inventory columns track boundary → consumer → provider → protocol → status (`consumer_tested` → `provider_verified`). ## Consumer and provider 1. **Consumer tests** generate or refresh the pact document against expected interactions. 2. **Provider verification** replays interactions against the real implementation with **subset matching**: the expected response must be a subset of actual, so providers may grow fields without breaking consumers. 3. The same interactions feed **[Bridge](/HoBo/troubleshooting/bridge/)** activation: a local fork must pass the BoundaryContract before carrying traffic. ## Harness The contract harness parses pact documents, builds in-process consumer mocks, runs provider verification, and gates inventory drift (declared boundaries must match files on disk). ## Contract IR vs PACT - **Contract IR** — versioned producer/consumer edge + shape + optional **EffectSpec** / **ResilienceProfile**; feeds can-deploy and Gatekeeper. - **PACT** — interaction-level request/response evidence for a boundary implementation. - Developers usually never “turn PACT on”; it is the harness detail under the contract engine. ## Compatibility gates `can-deploy` / Gatekeeper answer version-skew: *will this next revision still satisfy deployed consumers?* Removed required fields and (when declared) **effect widening** are breaking. See [Reference: contracts](/HoBo/reference/contracts/). ## Next - [Effect-aware isolation](/HoBo/testing/effects/) - [Deployment rehearsal](/HoBo/testing/deployment/) --- # Effect-aware isolation This is one of the patterns that feels unfamiliar if you came from “hand-roll mocks.” HoBo puts **side-effect intent on the contract**. Tests read that metadata and choose isolation automatically. ## EffectSpec in one minute A `BoundaryContract` may declare: | Axis | Examples | |---|---| | **Purity** | `pure` → `varies` → `transacts` → `noRollback` | | **Capabilities** | `decides`, `suspends`, `allocates`, `concurrent` | | **Domains** | `network`, `storage`, `identity`, `policyEval`, `eventEmit`, `clock`, `agentTool`, … | - Composition is **union** of callee effects. - **Subtyping:** safer (fewer/purer) effects may be used where more effects are expected. - **Widening** (pure → network, etc.) is a **breaking** can-deploy reason when effects are declared. ## Harness modes ```ts import { createBoundaryContract, pureEffects } from "hobo/contracts"; import { createEffectIsolation } from "hobo/testing"; const pure = createBoundaryContract({ id: "pure-math", effects: pureEffects(), }); createEffectIsolation(pure); // → isolationMode: "zero-infra", mocks: [] const net = createBoundaryContract({ effects: { purity: "varies", capabilities: ["decides", "suspends"], domains: ["network", "externalCall"], }, }); createEffectIsolation(net); // → isolationMode: "domain-mocked", mocks per domain ``` | Isolation mode | When | |---|---| | `zero-infra` | Pure contracts | | `domain-mocked` | Domain I/O declared | | `conservative` | Unknown / no usable domains | ## Resilience fault injection Attach a **ResilienceProfile** (secondary metadata — not a new top-level primitive): - Deterministic retry schedules (stable seed + budget + attempt facts). - **Never** retry `policy.Decided` deny, terminal failures, or cancellation. - Inspection is reconstructed from **facts**, not process memory. ```ts // Fault-injection harness: one policy deny, no blind retries await runResilienceFaultInjection({ contract: netWithProfile, failures: [{ attempt: 1, failureClass: "policy.Decided", message: "deny" }], }); // attempts === 1 — policy denies are never retried ``` ## Capture without re-execution Effect capture records `contract.EffectDeclared` / `contract.EffectInvoked`. Replay **inspects those facts** — it does not re-invoke handlers. That matches the product rule: inspection is event-sourced. ## Export CLI and MCP can describe a contract or export an effect manifest for policy and SRE tooling. ## Next - [Reference: effects](/HoBo/reference/effects/) - [Durable EventStore tests](/HoBo/testing/durability/) --- # Dual fixtures **Dual fixtures** keep pure and effectful (or dual-path identity) tests aligned from the **same metadata** so they cannot silently diverge. ## Effect duals | Path | When | Infrastructure | |---|---|---| | **Pure** | `isPure` / no domain I/O | Zero infra; effect isolation harness | | **Effectful** | Domains present | Automatic mocks from EffectSpec + optional live | Same contract id, same examples, different routing. If pure path asserts X and effectful asserts ¬X, the fixture definition is wrong — fix metadata, not only one side. ## Identity duals Continuous identity migration uses dual-path principal fixtures: - Old and new principal shapes evaluate together - Matching/policy decides continuity - Offline fixtures extend without live IdP ## Migration duals Schema/data dual windows: produce fixtures that assert both old and new folds during the dual period, then cutover. ## Session Replay duals Local Session Replay (default on every **App Kit**) stores heavy blobs via **dual-fixture** adapters (memory/fixture engines) so privacy + tracker/player contracts stay offline-green without multi-cloud object storage. Production binds object-storage ports; dual fixtures remain the proof path. See [Session Replay](/HoBo/reference/session-replay/) · [App Kits](/HoBo/primitives/app-kits/). ## Rules 1. One source of truth for input shapes (contract examples / fixture tables). 2. Name duals explicitly in test titles (`pure:` / `effect:` / `dual-id:`). 3. Never hand-maintain two mock graphs that claim to be the same boundary. ## Related [Effect-aware isolation](/HoBo/testing/effects/) · [Migration fixtures](/HoBo/testing/migration/) · [Identity](/HoBo/security/identity/) --- # Durable EventStore testing The Camp **EventStore** defaults to SQLite behind ports. Tests should exercise the **real semantics**, not an idealized in-memory fake, whenever durability is the claim under test. ## What “default SQLite” means | Rule | Test implication | |---|---| | One **partition** file, one active writer | Don’t test multi-writer shared NFS/WAL as supported | | Stream **head** holds revision + `owner_epoch` | Assert CAS; no `COUNT(*)` for revision | | **Fence tokens** reject stale holders | Correct `expectedSeq` + lower fence still fails | | **`global_position`** is store-local | Positions for catch-up ≠ ULID order ≠ domain identity | | Atomic **commit** of events + effects | Rollback leaves neither events nor queue/timer/outbox | | Snapshots are **caches** | Cannot write snapshot ahead of stream head | ## Where it lives Exercise durability through the Camp EventStore and your package tests — not a toy fake of the journal. ## Minimum durability matrix 1. Append + read stream; optimistic conflict on stale `expectedSeq`. 2. Multi-event batch atomicity (crash mid-append leaves whole batches only). 3. Idempotency key retry returns deduped without advancing head incorrectly. 4. Fence: stale token / missing token on fenced stream. 5. `readAll(after)` positioned catch-up; checkpoint monotonicity. 6. Commit events + queue/timer/outbox; failure rolls all back. 7. Migrate legacy schema fixtures; integrity_check; reopen is idempotent. 8. Close/reopen file (and Worker recreate when testing browser). ## In-memory QueueClockStore Runtime-local still has an **explicit in-memory** queue/clock adapter for deterministic pure unit tests. That is **not** the production durability claim. When the durable path is configured, queue/timer/outbox stage through the journal. ## Browser policy in tests - `allowTransient: true` → in-memory journal for semantic coverage in Node CI. - Durable required + no OPFS → `PERSISTENCE_UNAVAILABLE` (must not claim OPFS success). ## Next - [Reference: EventStore](/HoBo/reference/eventstore/) - [Chaos & availability](/HoBo/testing/chaos/) --- # Migration fixtures Migrations are product features — test them as such. ## Classes | Class | Prove | |---|---| | **Journal schema** | v1 → v2 user_version; integrity checks; fail closed | | **Event upcast** | Old body bytes → current fold state | | **Identity continuous** | Dual principal fixtures offline | | **Trail recipes** | `can-trail`, plan evidence, golden oracles | | **Infra re-place** | Same Gatekeeper path as deploy | ## Fail closed | Failure | Expected | |---|---| | Noncontiguous seq after migrate | `STREAM_INTEGRITY` / migrate fail | | Opening newer DB with older code | `UNSUPPORTED_SCHEMA_VERSION` | | Snapshot ahead of head | `SNAPSHOT_AHEAD` | | Trail gate fail | No partial cutover mute | ## Dual window tests While dual is open, both old and new projections must answer consistently for in-window subjects. Cutover tests assert the gate and the terminal single-path state. ## Trail golden Trails package golden oracles pin plan/run verdicts for recipe shapes — treat like contract tests for migration recipes. ## Related [Schema](/HoBo/reference/schema/) · [Trails](/HoBo/primitives/trails/) · [Continuous migration](/HoBo/cicd/migration/) · [Dual fixtures](/HoBo/testing/dual-fixtures/) --- # Queue, timer, outbox Deferred work is where “it worked on my machine” goes to die. Prove lease and restart semantics explicitly. ## Must-pass cases | Case | Assert | |---|---| | Enqueue + lease + complete | Happy path | | Lease expiry + re-lease | Previous complete fails closed (stale fence) | | Idempotent consumer | Duplicate delivery → one external effect | | Stage in commit | Crash after commit still delivers at-least-once | | Crash before commit | No phantom dispatch | | Timer once-after-restart | Deadline fires once; no double-fire | | Cron/schedule | Deterministic with virtual Clock | ## Transactional staging Domain events + staged queue/timer/outbox rows commit atomically. Network I/O is **after** commit. Tests should kill the process in both windows. ## Clock Use virtual Clock — wall sleeps make CI flaky and hide bugs. ## Related [Queue port](/HoBo/reference/queue/) · [Clock](/HoBo/reference/clock/) · [EventStore](/HoBo/reference/eventstore/) · [Durability](/HoBo/testing/durability/) --- # Deployment rehearsal Deploys are **plans before actions**. You can fail a gate without provisioning anything. ## Three speeds | Speed | Command / API | Needs cloud? | |---|---|---| | **Plan only** | `hobo plan` | No | | **Local lane** | `hobo up` / `hobo dev` against local Camp | No | | **Gatekeeper** | `hobo verify` / Gatekeeper | No | ### `hobo plan` Pure rehearsal: resolution, cost, policy verdicts. A blocked gate is a **no-op** on infrastructure — the plan output *is* the test result. ### Local lane Full plan → gate → provision → deploy against the default local Camp. CI can run a complete deployment every commit without provider credentials. ### can-deploy + Gatekeeper - **can-deploy** builds a compatibility matrix of candidate contracts vs deployed state. - **Gatekeeper** composes: contracts · policy pack · obligations · cost/quota · optional secretless-config / can-migrate dimensions. - Every evaluation records **`policy.Decided` even on allow** so “why did this pass?” is replayable. - Decision payloads can include **effect summaries** when contracts declare EffectSpec / ResilienceProfile. ## What to assert in tests - Breaking field removal → `compat` reasons with machine codes. - Effect widening (when effects declared) → `effect-widened`. - Policy deny → Gatekeeper `deny`; never treated as a transient retry class. - Obligation pending → `require-approval`. - Quota exceeded → `deny`. ## Suggested package tests ```bash run the package tests ``` ## Next - [Reference: contracts](/HoBo/reference/contracts/) - [Bridge modes](/HoBo/testing/bridge-modes/) --- # Bridge modes Boundary modes are not only a test helper. They are the same routing model used by **[the Bridge](/HoBo/troubleshooting/bridge/)** for local/remote development. ## Mode table | Mode | Behavior | |---|---| | **live** | Call the real implementation / deployed boundary | | **mock** | Serve contract-conformant fakes (examples / effect mocks) | | **bridge** | Fork only the tagged **session’s** traffic to a local implementation; must pass contract check before activation | | **replay** | Serve recorded outputs; no original dependency required | | **shadow** | Candidate receives traffic; output compared and **never** returned to callers | | **fault** | Inject failures for a **session only** | ## Session scoping Fault and bridge modes are **session-tagged**. Break a dependency for `tyler/chaos` without taking down production traffic for everyone else. ```ts payments.fault({ message: "simulated outage", session: "tyler/chaos" }); await payments.call(charge, { session: "tyler/chaos" }); // fails for you await payments.call(charge); // fine for others ``` ## Contract gate on bridge activation A bridge that would carry traffic **must** satisfy the BoundaryContract first. Failures surface as typed contract errors — not silent skew. ## Recording Recording stays on: interactions become specification-by-example traces and training material (with lane-origin events excluded from baseline promotion by default). ## Next - [The Bridge](/HoBo/troubleshooting/bridge/) - [Chaos & availability](/HoBo/testing/chaos/) --- # Chaos & availability Availability testing is the only layer that *should* touch real camps. Everything else stays offline by construction. ## Session-scoped faults Use boundary **fault** mode with a session tag so chaos stays personal: - Your session sees the outage. - Everyone else keeps the live path. - Results are events — not a screenshot. ## Synthetics (Crew) The **Endpoint Monitor** Crew operator probes camp endpoints continuously and emits burn/alert signals that feed the **SRE** operator. See [Crew: Endpoint Monitor](/HoBo/operations/endpoint-monitor/) and [Crew: SRE](/HoBo/operations/sre/). ## SLOs as projections Availability targets are **materialized views** over the event log. Test them in CI by replaying history — not by scraping a dashboard. ## Shadow lanes `.shadow(candidate)` sends real inputs to a candidate implementation, compares outputs, and **never** returns candidate output to callers. Availability testing for the next version, paid for by the current one. ## Resilience fault injection (offline) For contract-level resiliency metadata (not live chaos), use [effect-aware fault injection](/HoBo/testing/effects/): - Transient failures may retry under deterministic schedules. - `policy.Decided` / terminal / cancellation **never** retry. ## Blast radius rules - Prefer session tags over global fault flags. - Prefer shadow over dual-writing production responses. - Prefer replay/fork lanes for golden suite failures before production self-heal. ## Next - [The Crew](/HoBo/crew/) - [Conformance & golden](/HoBo/testing/conformance/) --- # Property & model tests When unit examples are not enough, pin **invariants** across random schedules. ## High-value properties | Property | Idea | |---|---| | Journal ≡ model | Random command sequences; store fold equals pure reference model | | Fence monotonic | Epoch never decreases for a stream key | | Positions monotonic | `readAll` positions strictly increase | | OCC | Concurrent appends never both commit same seq | | Codec round-trip | encode∘decode = id for supported codecs | | Effect staging | Commit atomicity under crash injection | | Idempotency | Keys collapse duplicates | ## Tooling notes - Prefer deterministic seeds (resilience profiles already use them). - Shrink failures to minimal command traces for humans/agents. - Run property suites in package scope — not as default discovery diagnostics for unrelated changes. ## Related [Durability](/HoBo/testing/durability/) · [Aggregates](/HoBo/testing/aggregates/) · [Benchmarks](/HoBo/testing/benchmarks/) --- # Behavior & Cucumber Behavior specs document **journeys** as executable evidence. HoBo does **not** use traditional live E2E against real downstream clouds in default PR gates. ## Allowed | Pattern | Detail | |---|---| | Gherkin / Cucumber | Features under behavior specs | | Playwright as **driver** | Browser UI with **mocked** service boundaries | | Contract-derived mocks | From Pact / EffectSpec / fixtures | | Local Camp lane | `hobo dev` / sandbox against in-memory or SQLite | ## Forbidden as default gates - Real AWS/GCP/Azure mutates in PR CI - “E2E” that requires live provider credentials - Hand-written mocks that drift from contracts ## Behavior evidence Behavior specs use executable features with contract-derived mocks. ## Related [Testing overview](/HoBo/testing/) · [Contracts](/HoBo/testing/contracts/) · [Deployment rehearsal](/HoBo/testing/deployment/) --- # Benchmarks Benchmarks catch **algorithmic** regressions (O(n²) folds, accidental full-table scans, lock convoys) — not vanity ops/sec on laptops. ## What to bench | Area | Signal | |---|---| | Append + head CAS | Throughput under single writer | | Fold large streams | Snapshot amortizes re-fold | | `readAll` catch-up | Position scans stay linear | | Matching toy plans | Placement solve stays bounded | | Effect lattice helpers | Pure CPU; should be boring-fast | ## Rules 1. Deterministic seeds and fixed datasets. 2. Report medians with clear environment notes — not single-shot peaks. 3. Fail CI on large regressions only when baseline is pinned intentionally. 4. Do not bench network to public clouds in default gates. ## Related [Property tests](/HoBo/testing/property/) · [EventStore](/HoBo/reference/eventstore/) --- # Conformance & golden suite ## Port conformance Every implementable port (runtime EventStore, provider adapters, IaC engines, …) ships a **conformance suite**. Human- and AI-authored adapters must pass the same suite to be certified. conformance suite hosts shared suites (for example EventStore append/idempotency/replay). ## SDK semantics annex Resilience, cancellation, deadlines, lifecycle tables, and supervision are pinned by the **DX SDK resilience semantics** matrix — Contract-visible ResilienceProfile application **must** obey that matrix (never-retry rules, deterministic schedules, fact-based inspection). ## Golden suite (deploy gate) The **golden suite** is a first-class Gatekeeper dimension: a signed, reproducible bundle that pins contracts, evals, conformance, benchmarks, chaos, and golden-master samples. Failures are structured, event-sourced, and can open Crew remediation (propose fix / A2H). ## Behavior / Cucumber Executable Gherkin features live under `features/` for high-level evidence (durable journal scenarios, effects/resiliency scenarios). They complement package unit tests; they are not live multi-cloud E2E. ## What “green” means | Lane | Meaning | |---|---| | Unit / package | Implementation correct in isolation | | Contract consumer/provider | Boundary expectations hold | | Conformance | Port claim is honest | | can-deploy / Gatekeeper | Version skew + policy + cost ok | | Golden | Reproducible multi-check bundle ok | ## Next - [Reference overview](/HoBo/reference/) - [Trail map](/HoBo/guide/status/) --- # What not to do | Anti-pattern | Do instead | |---|---| | Hand-written long-lived mocks | Contract examples / effect harness mocks | | `COUNT(*)` as stream revision | Stream head revision | | Multi-writer shared SQLite over NFS as “HA” | Single-owner partitions; add partitions to scale | | Retry `policy.Decided` denials “to be resilient” | Fix policy or satisfy obligations | | Retry `STALE_FENCE` as OCC | Re-acquire lease; new fence | | Live cloud E2E in every PR | Contract + local Camp + rehearsal | | Broad full-repo test sweeps for discovery | Search first; narrow package tests | | Wall-clock sleeps in pure domain tests | Virtual Clock | | Assert process memory for audit | Replay events | | Put LLM tokens in EventStore | Streamer | | Bare `.hobo` hosts in fixtures | `.hobo.localhost` / `.hobo.test` | | Invent parallel progressive plane beside Harness | Use Harness + Trails | | Pool free-tier headroom across tenants in tests that claim “prod realism” | Model per-tenant isolation | ## Related [Testing overview](/HoBo/testing/) · [Errors](/HoBo/reference/errors/) · [Security](/HoBo/security/) --- # Reference overview This section is the **lookup manual** — precise product surfaces for implementing, reviewing, or debugging. Narrative learning stays in the [Guide](/HoBo/guide/); testing patterns live under [Testing](/HoBo/testing/). ## Map | Page | Use when you need | |---|---| | [Capability map](/HoBo/reference/packages/) | Which product surface owns a concern | | [Component hosts](/HoBo/reference/component-hosts/) | How components run vs how the journal is stored | | [EventStore & journal](/HoBo/reference/eventstore/) | Durable log, positions, fencing, SQLite default | | [Schema & migrations](/HoBo/reference/schema/) | Journal schema and integrity | | [Queue](/HoBo/reference/queue/) · [Streamer](/HoBo/reference/streamer/) · [Clock](/HoBo/reference/clock/) | Other Camp ports | | [Messaging](/HoBo/reference/messaging/) | Topics, delivery guarantees, Kafka / Dapr options | | [ActorHost & fencing](/HoBo/reference/actor-host/) | Leases and fence tokens | | [Runtime ports](/HoBo/reference/runtime-ports/) | Port coupling summary | | [Contracts & Gatekeeper](/HoBo/reference/contracts/) | Compatibility and deploy gates | | [Effects & resiliency](/HoBo/reference/effects/) | Side-effect intent and resiliency profiles | | [Policy plane](/HoBo/reference/policy/) | Packs, obligations, deny ceilings | | [Matching & ResourceActors](/HoBo/reference/matching/) | Placement + reconciliation | | [Harness & progressive](/HoBo/reference/harness/) | Flags, experiments, lanes | | [Errors & codes](/HoBo/reference/errors/) | Stable machine codes | | [Event families](/HoBo/reference/events/) | Canonical event names | | [CLI surface](/HoBo/reference/cli/) | Canonical verbs | | [WIT & storage](/HoBo/reference/wit-storage/) | Component storage capability | | [Identity](/HoBo/reference/identity/) · [Naming](/HoBo/reference/naming/) · [Connectivity](/HoBo/reference/connectivity/) | Identity + network | | [Config & secrets](/HoBo/reference/config-secrets/) · [Metering](/HoBo/reference/metering/) | Secretless config + usage facts | | [Session Replay](/HoBo/reference/session-replay/) | Session reconstruction on every App Kit | | [Glossary](/HoBo/reference/glossary/) | One-line definitions | ## Status honesty Pre-release: some capabilities are design-stage only. [Trail map](/HoBo/guide/status/) lists what runs today. --- # Capability map What you use day to day — **product surfaces**, not an inventory of private implementation modules. ## Authoring | Surface | What you use it for | |---|---| | **Bindles** | Pack the app: services, resources, wiring, policy intent | | **App Kits** | Entrypoint kinds (`api`, `web`, `service`, …) inside a Bindle | | **Camps / Yards / Junctions** | Topology: home, provider scope, federation | | **Trails** | Continuous migration recipes + live upgrade/migration executors (preflight evidence, expand→backfill→contract, pooled→siloed tenant moves) | ## Runtime (Camp) | Surface | What you use it for | |---|---| | **EventStore** | Append-only domain and control facts | | **Queue / messaging** | Deferred work and pub/sub | | **Streamer** | High-volume frames (logs, tokens) | | **ActorHost** | Single-writer leases and fencing | | **Clock** | Deterministic time for workflows and tests | | **Session Replay** | Privacy-first session capture on App Kits | Default durable journal is a **single-owner SQLite** path. Alternate engines and component hosts (Wasmtime, OCI/Kubernetes, cloud-native) plug in behind the same ports. ## Control & identity | Surface | What you use it for | |---|---| | **Stations & Conductor** | Control-plane isolation + placement decisions; routing control loop + progressive cutover | | **Dispatcher** | Deterministic credit/budget-aware backend selection (`dispatch.Decided`) | | **Jurisdiction & residency** | Region→jurisdiction catalog, per-tenant residency constraints, fail-closed residency gate | | **Gatekeeper / contracts** | can-deploy and composed deploy gates | | **Policy packs** | Deny-by-default authorization and obligations | | **Identity** | Workload identity, brokered tokens, continuous identity | | **Harness** | Flags, progressive delivery, experiments | | **Connectivity** | Named endpoints, overlay, secretless connectors | ## Operator surfaces | Surface | What you use it for | |---|---| | **`hobo` CLI** | Lifecycle, diagnosis, authoring, ops | | **MCP** | Same verbs for agents | | **Metering** | Usage facts and showback | ## Rules of thumb 1. Author **Bindles and topology** — not storage engines. 2. Talk to **ports** (EventStore, Queue, Messaging); engines stay swappable. 3. Prefer **secretless** federation over long-lived keys. 4. **Conformance** is what “certified provider” means — same bar for every adapter. ## Related [Architecture](/HoBo/architecture/) · [Component hosts](/HoBo/reference/component-hosts/) · [Runtime ports](/HoBo/reference/runtime-ports/) · [CLI](/HoBo/reference/cli/) --- # Component hosts Camps separate two concerns: | Concern | Role | |---|---| | **Durable EventStore engines** | Single-owner journal, fencing, staged effects | | **Component hosts** | How Bindle components actually run | App authors pick ports and Bindles; operators choose engines. Conformance certifies claims. ## EventStore engines (default path) | Engine | Host | Notes | |---|---|---| | Local SQLite | Node / server | Storage worker; crash-safe defaults | | Browser SQLite | Browser | WASM + OPFS; fail-closed if durable required without OPFS | See [EventStore](/HoBo/reference/eventstore/) · [Runtime ports](/HoBo/reference/runtime-ports/). ## Component host kinds ### Wasm (Wasmtime) - Private components as **wasi:http** processes in sandboxes - Public web may stay process-local - Deny-by-default: only linked WIT imports are invokable ### OCI / Kubernetes 1. Accept prebuilt wasi:http components 2. Pack local OCI layout under the HoBo home directory 3. Schedule on Kubernetes through a typed transport (version, apply, rollout) ### Cloud-native registry - Multi-cloud host registry with a **mock** path for parity tests - Cloud providers appear as they pass conformance ## Optional sidecar runtime Some Camps may opt into a **sidecar-style** runtime (for example Dapr-backed actors and pub/sub). It is never the default local path. ## Later engines Postgres journals, Temporal, Restate, and similar remain future adapters behind the same ports. ## Related [Capability map](/HoBo/reference/packages/) · [WASM guide](/HoBo/architecture/wasm/) · [Trail map](/HoBo/guide/status/) · [Conformance](/HoBo/testing/conformance/) --- # EventStore & journal ## Ports | API | Role | |---|---| | `EventStore` | `append`, stream `read`, live `subscribe`, `snapshot` | | `DurableEventStore` | + `readAll` (positioned), `commit` (events + effects), durable subscription checkpoints | ### Positioned events ```ts type EventPosition = bigint; // branded / parse-don't-validate interface PositionedEvent { position: EventPosition; event: EventEnvelope; } ``` `position` is **database-local** within a partition. It is not the domain ULID and not per-stream `seq`. ### Append options - `expectedSeq` — optimistic concurrency (CAS). - `fenceToken?: bigint` — actor lease fence; full 64-bit, never coerced to unsafe JS numbers. ### Typed errors (stable codes) `OPTIMISTIC_SEQUENCE_CONFLICT`, `STALE_FENCE`, `MISSING_FENCE`, `CHECKPOINT_REGRESSION`, `UNSUPPORTED_SCHEMA_VERSION`, `SNAPSHOT_AHEAD`, `PERSISTENCE_UNAVAILABLE`, `SQLITE_BUSY`, … ## Single-owner partition model - One SQLite file (or equivalent partition) · one active writer. - Scale by **adding partitions**, not multi-writer WAL over NFS/SMB. - Native: WAL + `synchronous=FULL` (Node `node:sqlite` behind a storage worker). - Browser: Worker + OPFS when available; fail-closed if durable required without OPFS. ## Stream heads Revision and `owner_epoch` live on a **streams** head row. Append algorithm: 1. `BEGIN IMMEDIATE` 2. Ensure head; read revision + epoch 3. Resolve idempotency 4. Validate expectedSeq + fencing 5. Insert events (contiguous seq + positions) 6. Stage durable effects 7. CAS-update head 8. Commit → then live wake-ups Legacy streams migrate with `owner_epoch = 0` (unfenced); first fenced append enters fenced mode permanently. ## Transactional effects `commit` may stage: queue messages, actor signals, timers, cron mutations, outbox rows — **same transaction** as domain events. Network dispatch happens **outside** the transaction (at-least-once + idempotency). ## Delivery guarantees | Guarantee | Claim | |---|---| | Per-stream OCC | Yes | | Fenced single writer | Yes | | At-least-once subscriptions/outbox | Yes | | Exactly-once external delivery | **No** | ## Engines SQLite storage engine · local runtime · browser runtime · WIT `hobo:storage@0.1.0` ## Related testing [Durable EventStore tests](/HoBo/testing/durability/) --- # Schema & migrations Default durable journal schema lives in **SQLite storage engine**. Engines may differ physically; **semantics** (stream heads, positions, fencing, staged effects) must pass conformance. ## Versioning | Constant | Meaning | |---|---| | `SCHEMA_VERSION` | Current (v2) | | `LEGACY_SCHEMA_VERSION` | Prior on-disk (v1) | `PRAGMA user_version` must match code. Opening a **newer** DB with older code → `UNSUPPORTED_SCHEMA_VERSION` (fail closed). ## Core tables (v2 conceptual) | Table | Role | |---|---| | `streams` | Head row: stream id, revision, `owner_epoch`, metadata | | `events` | Append-only bodies + per-stream `seq` + store `position` | | `effects` / staged work | Queue, timer, outbox, signals pending dispatch | | `subscriptions` / checkpoints | Durable consumer positions | | `snapshots` | Optional fold caches (never source of truth) | Exact SQL is owned by the storage engine; do not re-implement in apps. ## Native pragmas Native single-owner partitions use WAL-friendly settings with **`synchronous=FULL`** (or equivalent durability) — throughput second to crash safety for the default path. ## Codecs Event payloads encode via versioned codecs. Unknown codec → `UNSUPPORTED_CODEC`. Upcasters live in the data plane, not as silent rewrite of historical bytes without policy. ## Migration rules 1. Forward-only migrations; no destructive “fix by deleting the file” in production paths. 2. Legacy streams get `owner_epoch = 0` until first fenced write. 3. Integrity checks: contiguous seq, head matches last event, positions monotonic — failures → `STREAM_INTEGRITY` / `MIGRATION_FAILURE`. 4. Tests: [Migration fixtures](/HoBo/testing/migration/). ## What apps must not do - Open the SQLite file and invent tables beside the journal. - Use `COUNT(*)` as stream revision. - Multi-writer mount the same file over NFS/SMB and call it HA. ## Related [EventStore](/HoBo/reference/eventstore/) · [Durability testing](/HoBo/testing/durability/) · [WIT storage](/HoBo/reference/wit-storage/) --- # Queue port The **Queue** is the Camp port for deferred and concurrent work: jobs, actor signals, delayed delivery, and outbox-style dispatch. Ports are product surfaces; engines implement lease semantics. ## Operations (conceptual) | Op | Role | |---|---| | `enqueue` | Persist a message with optional delay, group, and idempotency key | | `lease` | Claim work for a worker with visibility timeout + fence | | `complete` / `fail` | Ack success or schedule retry / DLQ path | | `signal` mapping | Deliver actor wake-ups without inventing a second bus | ## Guarantees | Guarantee | Claim | |---|---| | At-least-once delivery | **Yes** | | Exactly-once external side effects | **No** — use idempotency keys | | Ordered per group/key (engine-dependent) | When configured | | Survives process restart | When backed by durable journal | ## Transactional staging When using **DurableEventStore.commit**, queue messages (and timers/outbox rows) may stage in the **same transaction** as domain events. Network dispatch happens **outside** the transaction. That yields: - Domain fact + “work is pending” atomicity - At-least-once dispatch after commit - Need for consumer idempotency ## Leases and fencing Workers hold leases. Completing work after a stolen lease must fail closed (stale fence). Same fence discipline as ActorHost / EventStore — do not coerce fence tokens to unsafe JS numbers. ## Testing - Restart with staged messages still leased correctly. - Double-complete after lease expiry fails closed. - See [Queue, timer, outbox](/HoBo/testing/queue-outbox/). ## Related [Messaging, Kafka & Dapr](/HoBo/reference/messaging/) · [EventStore](/HoBo/reference/eventstore/) · [ActorHost](/HoBo/reference/actor-host/) · [Runtime ports](/HoBo/reference/runtime-ports/) --- # Messaging, Kafka & Dapr HoBo unifies app messaging behind **ports**, not vendor SDKs. Bindles declare `topic()` / queue / actor `signal`; Camps select backends. **In-process wasm stays the default**. Dapr is **opt-in**. ## Authoring (ports only) ```ts import { topic } from "hobo/bindles"; // Backend is Matching / Camp runtime selection — never import Kafka or Dapr clients here. const events = topic("user-created").allow("publish"); ``` Actors use the same definition surface under **native** or **dapr** placement. ## Capability map | Capability | Role | |---|---| | Messaging port | Delivery guarantees, tenant scoping, in-memory adapter | | Kafka provider | Kafka / Redpanda / MSK / Confluent capability + resource definitions | | Optional Dapr runtime | Optional Camp runtime: ActorHost + pubsub + secretless component YAML | | Runtime placement | Native or Dapr actor placement and signal routing | | Junction federation | Cross-Yard topic / actor-signal **federation plan** (control plane + identity) | | Messaging policy | Exactly-once, deny-Dapr, tenant quotas | | Conformance suite | Messaging conformance against Kafka and Dapr adapters | ## Fidelity honesty | Mode | How | Label | |---|---|---| | Simulated | No transport / no Dapr client | `simulated` | | Live harness | In-memory Kafka / Dapr test doubles | `live` | | Live process | Real broker or daprd client implementing the same interfaces | `live` | | Forbidden | `live: true` without transport/client | **throws** | Never treat a boolean flag as live while still on the pure emulator. ## Delivery guarantees - Portable default: **at-least-once** - **Exactly-once** / transactional produce are **capability flags** Matching and `can-deploy` can require - Dead-letter after `maxDeliveryAttempts` (subscription); permanent failure advances the consumer group (commit under lease — do not nack-then-commit) ## Secretless + multi-tenant - Credentials are binding refs only (`fromBinding`, federation, `secretRef`) - Generated Dapr component YAML must not embed passwords - Topics, consumer groups, and Dapr app-ids are tenant-scoped by default ## CLI | Command | Role | |---|---| | `hobo doctor` | Reports Kafka + Dapr readiness and messaging hazards | | Authoring | Declare `topic()` in the Bindle; backends are Camp/Matching choices | Messaging plane diagnostics surface through `doctor` and contract/conformance suites — apps never import Kafka or Dapr clients. ## Permanent non-goals - Kafka cluster operator / managed-service UI - Full Dapr workflow **engine** parity (supported: **activity-name map only**) - Advanced schema-registry Avro/Protobuf wire protocol end-to-end - Junction **data-plane** byte relay (control-plane federation is in; overlay carries data) ## Related [ActorHost](/HoBo/reference/actor-host/) · [Queue](/HoBo/reference/queue/) · [Component hosts](/HoBo/reference/component-hosts/) · [Capability map](/HoBo/reference/packages/) · [CLI](/HoBo/reference/cli/) · [WASM & WIT](/HoBo/architecture/wasm/) --- # Streamer port The **Streamer** carries **per-frame** high-volume payloads — LLM tokens, progressive UI chunks, log lines, exec stdout — without turning the domain EventStore into a blob store. ## Separation of concerns | Layer | Holds | |---|---| | **EventStore** | Domain and control **facts** (past-tense, foldable, audit) | | **Streamer journal** | Frame **bodies** for a run (dev-grade in-memory until teardown unless upgraded) | | **Optional audit opt-in** | Policy may require selected frames as events — not the default | Yard is **never** on the streaming data path. Live tails (`hobo logs`) go Camp → Streamer. ## Lifecycle Run lifecycle may be event-sourced even when bodies are not: - `StreamRunOpened` / related open facts - Frames append with monotonic sequence within the run - `StreamRunClosed` (or failure terminal) Readers attach with `fromSeq` for **replay-then-live**: catch up stored frames, then subscribe for new ones — same mental model as EventStore subscriptions. ## Guarantees | Claim | Status | |---|---| | Ordered frames per run | Yes (seq) | | Durable domain audit of every token | **No** by default | | Exactly-once delivery to all subscribers | **No** — at-least-once / best-effort per engine | | Cross-Camp frame federation | Out of scope for the port | ## Coupling rules - Aggregates and workflows **must not** dump large progressive payloads into EventStore “for convenience.” - Executors stream `exec.Output` through Streamer; Dashboard projections show live progress from those frames + control events. - Deterministic replay of **domain** state never depends on Streamer contents. ## Testing - Unit: mock Streamer; assert open/close and `fromSeq` contract. - Behavior: `hobo logs` / Bridge live-tail modes against contract fixtures. - Do not require live cloud sinks for PR gates. ## Related [Runtime ports](/HoBo/reference/runtime-ports/) · [Data plane](/HoBo/architecture/data-plane/) · [CLI](/HoBo/reference/cli/) --- # ActorHost & fencing **ActorHost** is the Camp port that hands out **single-writer leases** for aggregates (and similar actors). Without it, optimistic concurrency alone cannot stop a zombie process from appending with a stale view of the world after a steal. ## AggregateLease A lease (conceptual shape): | Field | Role | |---|---| | Aggregate / actor key | Who is leased | | `holderId` | Process or worker identity | | `fenceToken` | Monotonic **bigint** epoch for this key | | Expiry | Optional lease deadline | In-memory hosts exist for tests; production Camps use durable hosts that persist epoch across restarts, backed by the journal. ## Dual placement (native · dapr) The same actor definitions run under either host: | Mode | When | Notes | |---|---|---| | **native** (default) | No Camp dapr requirement | Lease + fence on local / wasm hosts | Prefer native unless you need the sidecar path. See [Messaging, Kafka & Dapr](/HoBo/reference/messaging/). ## Append algorithm coupling When an aggregate is fenced: 1. Acquire/renew lease → receive `fenceToken`. 2. `decide` pure business rules in memory. 3. `append` / `commit` with `expectedSeq` **and** `fenceToken`. 4. Store compares token to stream head `owner_epoch`: - equal → accept (and may bump epoch per engine rules) - lower → `STALE_FENCE` - missing on fenced stream → `MISSING_FENCE` Legacy streams start with `owner_epoch = 0` (unfenced). First fenced append enters fenced mode permanently. ## Error policy | Code | Meaning | Aggregate retry? | |---|---|---| | `AGGREGATE_CONFLICT` / OCC | Stale expectedSeq | Yes — reload & retry | | `STALE_FENCE` | Lost lease | **No** as blind OCC retry | | `AGGREGATE_INVARIANT` | Business reject | No | ## Rules - Never coerce `fenceToken` through `Number` — full 64-bit. - Lease steal must advance epoch so the previous holder fails closed. - Passivate emits facts; do not rely on process memory for “I still own it.” ## Related [EventStore](/HoBo/reference/eventstore/) · [Messaging, Kafka & Dapr](/HoBo/reference/messaging/) · [Aggregates testing](/HoBo/testing/aggregates/) · [Durability testing](/HoBo/testing/durability/) --- # Clock port The **Clock** port supplies time and sleep/cron primitives so workflows and tests can be **deterministic**. Wall-clock `Date.now()` inside pure decide/fold paths is a bug. ## Responsibilities | API family | Role | |---|---| | `now` | Current instant in the active clock domain | | `sleep` / delay | Schedule wake-ups (pairs with timer journal) | | `cron` / schedule | Recurring triggers as durable facts + wakes | | Advance (test) | Virtual clocks jump without waiting real time | ## Why it exists - **Replay** of workflows must re-enter the same branches. - **Timers** staged in the journal must fire once-after-restart with the same deadlines. - **Property tests** need controlled time to explore schedules without flakiness. ## Rules 1. Aggregates that need “business time” take it as a **command field** or injected clock reading at the boundary — not ad-hoc OS time inside pure functions. 2. Event envelope timestamps are assigned at append by the store/runtime — folds treat them as data. 3. Resilience timeouts use the same clock domain as the runtime so virtual tests remain meaningful. 4. Never sleep with real wall timers inside pure unit tests of domain logic. ## Coupling - Durable timers may stage beside events in `DurableEventStore.commit`. - ActorHost leases may use clock for expiry. - Streamer and logs may show wall time for humans; domain correctness does not depend on it. ## Related [Runtime ports](/HoBo/reference/runtime-ports/) · [Queue](/HoBo/reference/queue/) · [Aggregates testing](/HoBo/testing/aggregates/) --- # Runtime ports A **Camp** provides the durable substrate app runtimes need. **Ports** are product surfaces; **engines** implement them (SQLite storage engine, local runtime, browser runtime, future Postgres/Temporal/… adapters). ## Port table | Port | Responsibility | Default notes | |---|---|---| | **EventStore** | Append-only facts, stream read, live subscribe, snapshots | OCC via `expectedSeq` | | **DurableEventStore** | + positions, `commit` with staged effects, durable checkpoints | SQLite partition | | **Queue** | Enqueue / lease / complete; actor signal mapping | At-least-once | | **Streamer** | High-volume frames **outside** domain log | Replay-then-live `fromSeq` | | **ActorHost** | Single-writer lease + monotonic `fenceToken` | Fence couples to append | | **Clock** | Deterministic time + sleep/cron for replay | Virtual in tests | ## Coupling rules 1. Aggregates acquire **ActorHost** leases and pass **fenceToken** into EventStore `append` / `commit`. 2. Queue / timer / outbox may **stage inside** the journal transaction when durability is configured; network I/O happens after commit. 3. Streamer frame bodies are **not** domain events unless an audit policy opts in. 4. Clock reads during replay must not inject wall-clock nondeterminism into pure folds. 5. Yard control plane is **not** on the Camp data path for live user requests. ## Default engine | Engine | Role | |---|---| | SQLite storage | Schema, migrations, journal semantics | | Local runtime | Node EventStore + storage worker | | Browser runtime | SQLite WASM + OPFS policy (fail-closed if durable required) | | `hobo:storage@0.1.0` | Component-host storage capability | Scale by **adding partitions** (single active writer each), not multi-writer WAL over shared network filesystems. ## Port descriptors Port descriptors carry WIT-style interface refs and stability (`stable` | `preview` | `experimental`). Apps depend on ports; engines stay swappable behind conformance. ## Testing | Concern | Page | |---|---| | CAS, fencing, restart | [Durable EventStore](/HoBo/testing/durability/) | | Pure decide + conflicts | [Aggregates](/HoBo/testing/aggregates/) | | Queue/timer/outbox | [Queue, timer, outbox](/HoBo/testing/queue-outbox/) | | Engine swap | [Conformance](/HoBo/testing/conformance/) | ## Related [EventStore](/HoBo/reference/eventstore/) · [Queue](/HoBo/reference/queue/) · [Streamer](/HoBo/reference/streamer/) · [ActorHost](/HoBo/reference/actor-host/) · [Clock](/HoBo/reference/clock/) --- # Contracts & Gatekeeper ## BoundaryContract (Contract IR) Versioned producer/consumer boundary: - `id`, `version` (semver), `mode` (`declared` \| `inferred` \| `observed` \| `promoted`) - `edge` — kind, producer, consumer, channel, operation - `requiredShape` — format + required/provided fields - `contractHash` — content address (`sha256:…`) - optional **`effects`** — [EffectSpec](/HoBo/reference/effects/) - optional **`resilience`** — [ResilienceProfile](/HoBo/reference/effects/) Contracts are parse-don’t-validate: invalid shapes never become live values past the boundary. ## Compatibility Compatibility checks compare previous vs next revisions: | Change | Result | |---|---| | Added provided fields | Compatible | | Removed required/provided field consumers need | Breaking (`required-field-removed`) | | Channel change / version regress | Breaking | | Effect widening when effects declared | Breaking (`effect-widened`) | ## can-deploy Given deployed contract state + candidate deployment plan → compatibility matrix + gate pass/fail events with agent-actionable reasons. ## Gatekeeper Composes into one decision (`allow` \| `allow-degraded` \| `deny` \| `require-approval`): 1. can-deploy (contracts) 2. policy pack — **always** emits `policy.Decided` 3. obligations 4. cost/quota 5. optional secretless-config / can-migrate dimensions Also surfaces effect summaries and optional reasons with `gate: "effects" | "resilience"`. ## Describe / export CLI and MCP can describe a contract as human text or export an effect manifest for policy and SRE tooling. ## Testing See [Testing: contracts](/HoBo/testing/contracts/) for harness usage. ## Related [Testing: contracts](/HoBo/testing/contracts/) · [Testing: deployment](/HoBo/testing/deployment/) --- # Effects & resiliency . Names are locked: **Effects** on contracts; **Resiliency** as secondary visible layer. ## EffectSpec ```ts interface EffectSpec { schemaVersion: "hobo.effect-spec/v1"; purity: "pure" | "varies" | "transacts" | "noRollback"; capabilities: ("decides" | "suspends" | "allocates" | "concurrent")[]; domains: ( | "network" | "storage" | "identity" | "policyEval" | "eventEmit" | "metering" | "overlayTunnel" | "iac" | "secretsResidual" | "clock" | "externalCall" | "agentTool" )[]; } ``` ### Lattice helpers | Helper | Meaning | |---|---| | `isPure` | No domain I/O; purity pure | | `mayFail` | Has `decides` | | `requiresNetwork` | `network` or `externalCall` | | `effectUnion` | Join purity + union sets | | `effectSubtype` | Safer provided ⊆ required | | `isEffectWideningBreaking` | Next adds domains/caps or dirties purity | ### Defaults Missing effects are valid (opt-in). External reasoning applies a **conservative** default (`noRollback` + `decides` + network/externalCall) without rewriting the content hash of contracts that omitted effects. ## Capture | Fact | When | |---|---| | `contract.EffectDeclared` | Contract effects registered | | `contract.EffectInvoked` | Boundary invocation recorded | Effect capture writes to an event sink. Replay **reads facts** — does not re-execute handlers. Effect manifests export purity, domains, and flags (`mayFail`, `requiresNetwork`, `isPure`) for CLI, MCP, policy, and SRE tooling. ## ResilienceProfile (secondary) Strategies (metadata + thin applicator): - `retry` — deterministic seed, max attempts, delays; `retryOn: "transient-only"` - `circuitBreaker`, `timeout`, `bulkhead`, `fallback`, `hedging` Always enforce **neverRetry**: `policy.Decided`, `terminal`, `cancellation`. Applying a profile emits `resilience.Attempt` / `resilience.Terminal`. Inspection rebuilds terminal state from those facts without re-running ops. Circuit-breaker / hedging require network/externalCall domains on the contract. ## Testing entry [Effect-aware isolation](/HoBo/testing/effects/) --- # Policy plane Policy is **versioned, signed-pack evaluated** authorization and obligation logic. It is not “config flags scattered in YAML.” The policy engine holds IR and agent guardrails; Gatekeeper composes policy with contracts and cost. ## Packs A **policy pack** attaches to Yard/Junction (and can be proposed by Platform Engineer Crew): | Kind | Behavior | |---|---| | **Mandatory** | Deny ceilings — children may only further restrict | | **Default** | Overridable baselines | Evaluation is deny-by-default for unlisted `(role, action, resource)` on control-plane surfaces. CLI/MCP verbs map to `hobo::` or conformance fails. ## Decisions Every evaluation emits **`policy.Decided`** — including **allows**. Reasons are machine-readable for agents and humans. High-risk actions (destroy, prod mutate, spend above cap) resolve to: - `allow` - `require-approval` (obligation) - `deny` ## Obligations When decision is `require-approval` or similar, Gatekeeper records **obligations** (who, deadline, what satisfies). Unsatisfied obligations block mutate paths. A2H handoffs create human obligations with the same machinery. ## Never retry policy denials Resilience layers **must not** retry `policy.Decided` denials. That class is terminal for the attempt — change policy or request approval, do not spin. ## Guardrails AI operators and generation paths evaluate content/tool guardrails (PII, jailbreak, unsafe tools). Blocks emit `guardrail.Blocked` **and** `policy.Decided`. ## Composition order (Gatekeeper) 1. Contracts / can-deploy 2. **Policy pack** 3. Obligations 4. Cost / quota 5. Optional secretless-config / can-migrate / effects visibility ## Related [Contracts & Gatekeeper](/HoBo/reference/contracts/) · [Security](/HoBo/security/) · [Effects](/HoBo/reference/effects/) --- # Matching & ResourceActors ## Matching **Matching** resolves each Bindle resource against **Resource Definitions** and provider capability to choose *where* and *how* a resource is realized. It is the placement brain inside plan/up — not a separate product noun beside Gatekeeper. Flow (conceptual): 1. Resource collection produces provider-agnostic **manifest IR**. 2. Each node lowers to **ResourceIntent** (FinOps/routing-friendly, provider-neutral). 3. Matching + policy + cost solve a **DeploymentPlan** DAG. 4. Gatekeeper admits or denies the plan. 5. ResourceActors converge desired → actual. ## ResourceActor One **ResourceActor** (controller-style grain) per resource owns reconciliation: | State (conceptual) | Meaning | |---|---| | Desired | From DeploymentPlan | | Progressing | Executor in flight | | Ready | Provider reports healthy | | Failed | Past no-go threshold → compensation choreography | | Drifted | Actual ≠ desired → governed remediation | ## Choreography over orchestration - Independent subtrees provision **concurrently**. - No single coordinator SPOF for every resource. - Compensation is also choreographed: `resource.Failed` past threshold → `deployment.RolledBack` fans out. - **DeploymentView** projection reconstructs the saga graph for humans/agents. ## Executors Sandboxed runners (Firecracker-class) apply IaC/builds with **short-lived scoped tokens** — no ambient cloud creds. Stdout streams via Streamer; state events hit the control log. ## Continuous re-placement Dispatcher re-enters Matching when price/policy/reliability signals change. Same plan → gate → converge path as human `hobo up` — migration is not a special engine. ## Related [Control plane](/HoBo/architecture/control-plane/) · [Conductor](/HoBo/primitives/stations/#the-conductor) · [Continuous migration](/HoBo/cicd/migration/) · [Event families](/HoBo/reference/events/) --- # Harness & progressive delivery **Harness** is the flags / progressive delivery / experiments plane. It is **not** a second infrastructure control plane — it routes **traffic and contract observation** safely. ## Standards-shaped cores | Core | Role | |---|---| | **OpenFeature** | Sole public flag evaluation API for application code (`HoboOpenFeatureProvider` / `HoboFeatureClient`) | | **OpenDelivery** (HoBo-shaped) | Progressive delivery / weight shifts | | **OpenExperiment** (HoBo-shaped) | Bounded experiments with metrics hooks | | **OpenTelemetry hooks** | Export observation without inventing a parallel telemetry product | `FeatureFlagStore` + delivery + experiment engines are composed by `Harness`. ## Capabilities | Capability | Role | |---|---| | Feature flags | Ranked evaluation with audit facts | | Experiments | Bounded exposure | | Gateway weights | Shift traffic across revisions/camps | | Sandbox lanes | Isolate experimental contract observations | | Delivery composition | Safe rollout steps shared with Trails | | Continuous delivery plane | Workload-attested apply of Gatekeeper-approved sets (cloud control plane CD) composes Harness | ## Rules 1. Baseline subscribers **ignore** sandbox-lane events by default. 2. Flag changes are **facts** — replayable, not console memory. 3. Progressive steps still pass **Gatekeeper** when they mutate infra. 4. Trails compose Harness for cutover; pure flag flips need no Trail. 5. Agents change flags only under RBAC + policy (often require-approval in prod). 6. App code evaluates flags through **OpenFeature** — not ad-hoc env bools that bypass audit. ## CLI `hobo feature …` for flag/progressive ops. Observe via `inspect` projections and event replay. ## Testing - Dual fixtures for flag on/off paths. - Bridge shadow mode to compare experimental vs baseline. - Chaos: partial weight shifts must not break claimed session consistency. ## Related [Progressive delivery guide](/HoBo/cicd/progressive-delivery/) · [Trails](/HoBo/primitives/trails/) · [Bridge modes](/HoBo/testing/bridge-modes/) · [Identity](/HoBo/reference/identity/) · [Matching](/HoBo/reference/matching/) --- # Event families Event names use **`family.PastTenseFact`** shape. The design suite keeps a **canonical inventory**; new families must be cataloged before inventing backticked names in design or site prose. ## Envelope (conceptual) Every fact rides a shared envelope family of fields: | Field | Role | |---|---| | `id` | ULID — global causal-ish order | | `seq` | Per-stream total order | | `type` | `family.PastTenseFact` | | `stream` / aggregate id | Ownership boundary | | `causationId` / `correlationId` | Why / which workflow | | `tenant` (+ project/env as applicable) | Isolation claims | | `meta.idempotencyKey` | Exactly-once *effects* over at-least-once delivery | | `position` | When read from DurableEventStore — store-local | ## Families you will hit often | Family | Examples | Notes | |---|---|---| | `compat.*` | `GatePassed`, `GateFailed` | can-deploy outcomes | | `contract.*` | `Observed`, `Verified`, `EffectDeclared`, `EffectInvoked` | Contract IR lifecycle | | `policy.*` | `Decided`, `ObligationCreated`, `ObligationSatisfied` | **Always** decide, even allows | | `gatekeeper.*` | `Decided` | Composed gate result | | `resilience.*` | `Attempt`, `Terminal` | Secondary resiliency layer | | `actor.*` | `Activated`, `StateAppended`, `Passivated` | Actor host lifecycle | | `crew.*` | `AgentDispatched`, `ActionProposed`, `ActionApplied`, `HandoffRequested` | Thin roster — no per-operator silos | | `incident.*` | Opened → … → Resolved / PostmortemFiled | SRE loop | | `run.*` | `Started`, `StepCompleted` | Workflows / generation runs | | `deployment.*` | Planned / Applied / Deployed / Failed / RolledBack | Control plane | | `broker.*` | `Resolved`, `Rejected` | Conductor placement | | `resource.*` | Provisioned / Failed / DriftDetected | ResourceActors | | `token.*` / `identity.*` | Issued / Rotated / Accessed | Refs only for secrets | | `config.*` / `secret.*` | `Resolved`, `Accessed` | Refs only | | `meter.*` | `UsageRecorded` | Showback facts | | `guardrail.*` | `Blocked` | AI safety | | `trail.*` | Plan/run/cutover facts | Continuous migration recipes | | `station.*` | Station/Rail lifecycle | Control-plane isolation | | `replay.*` | `Started` / `ChunkIngested` / `Ended` / `Accessed` / … | Session Replay lifecycle (not event-log replay) | Domain apps add **their own** families under app namespaces — do not overload `crew.*` or `deployment.*` for business cart events. ## Sandbox lanes Boundary virtualization tags envelope **lane** metadata. Baseline contract promotion **ignores** sandbox-lane observations by default so experiments cannot poison production contracts or training trajectories. ## Rules 1. Past tense only — commands are not events. 2. Inspection and audit are **replay of the log**, not process memory. 3. Poison bodies → `*.Quarantined` so folds skip deterministically. 4. Crew roster facts stay on thin `crew.*` — operator-specific detail is payload, not family explosion. 5. Streamer frames are **not** automatically domain events. ## Related [Architecture](/HoBo/architecture/) · [Data plane](/HoBo/architecture/data-plane/) · [Crew shared model](/HoBo/operations/shared-model/) · [EventStore](/HoBo/reference/eventstore/) --- # Errors & codes Prefer **stable codes** over string matching. Map adapter errors at package boundaries. Do **not** leak SQL, filesystem paths, or secret material into app-facing errors. ## Shape conventions Errors that cross a boundary should carry: | Field | Role | |---|---| | `code` | Stable SCREAMING_SNAKE or dotted machine code | | `message` | Human-safe summary (no secrets) | | `retryable` | Explicit when known (do not infer from message) | | `cause` chain | Internal only; strip before external MCP/CLI if sensitive | Parse-don’t-validate constructors throw typed parse errors (`CoreParseError`, `EffectValidationError`, …) rather than returning “maybe invalid” DTOs. ## EventStore / storage | Code | Meaning | Typical retry | |---|---|---| | `OPTIMISTIC_SEQUENCE_CONFLICT` | Stale `expectedSeq` | Yes — reload stream | | `STALE_FENCE` | Fence below `owner_epoch` | No as OCC retry | | `MISSING_FENCE` | Unfenced write to fenced stream | No — acquire lease | | `CHECKPOINT_REGRESSION` | Non-monotonic checkpoint | No | | `UNSUPPORTED_SCHEMA_VERSION` | DB newer than code | No — upgrade binary | | `MIGRATION_FAILURE` | Migrate failed | No until fixed | | `STREAM_INTEGRITY` | Noncontiguous / head mismatch | No | | `UNSUPPORTED_CODEC` | Unknown payload codec | No | | `PERSISTENCE_UNAVAILABLE` | e.g. OPFS required but missing | No / change mode | | `SQLITE_BUSY` | Busy timeout | Maybe — backoff | | `SNAPSHOT_AHEAD` | Snapshot seq > head | No | | `STORAGE_WORKER_TERMINATED` | Worker crash | Restart worker | ## Aggregates | Code | Meaning | Retry? | |---|---|---| | `AGGREGATE_CONFLICT` | OCC / concurrent writers | Yes — reload & retry | | `AGGREGATE_INVARIANT` | Business rule reject | No | | Stale fence surfaced at aggregate | Lost lease | No blind retry | ## Resilience failure classes | Class | Retry via ResilienceProfile? | |---|---| | `transient` | Yes (within profile) | | `policy.Decided` | **Never** | | `terminal` | **Never** | | `cancellation` | **Never** | | `deadline` | Policy-dependent; not infinite retry | ## Contracts / Gatekeeper Compatibility and gate denials surface as structured reasons (`MachineReadableReason` with `gate: contracts | policy | obligations | cost | effects | resilience`), not only thrown exceptions. CLI/MCP should print reason codes agents can act on. ## Identity / config Prefer ref-safe messages (“resolution rank failed at provider-federation”) over embedding token fragments. Static-escape expiry is a hard fail, not a silent fallback to plaintext. ## Related [EventStore](/HoBo/reference/eventstore/) · [Effects](/HoBo/reference/effects/) · [ActorHost](/HoBo/reference/actor-host/) · [Contracts](/HoBo/reference/contracts/) --- # CLI surface The `hobo` CLI is deliberately **small and stable**. The same canonical verb set appears as MCP tools. Docs examples use only this list. ## Lifecycle | Verb | Role | |---|---| | `up` | Realize the object graph (build, plan, gate, provision, deploy) | | `down` | Reverse | | `dev` | Offline local Camp / inner loop | | `plan` | Dry-run the up decision (approvable diff) | | `verify` | Unified Gatekeeper preflight (read-only) | | `can-deploy` | Producer/consumer compatibility diagnosis | `up` still runs the same gate as `verify` automatically before mutation. Continuous delivery *observation* uses `deploy` (below), not a second ship verb. ## Identity, access, progressive | Verb | Role | |---|---| | `login` | Authenticate / associate default broker | | `claim` | Bind anonymous deployment to an account | | `identity` | Continuous identity status / inspect | | `access` | Elevated / cross-env access or break-glass | | `connection` | Connection management (Station-bound) | | `feature` | Flags / progressive delivery (Harness) | | `deploy` | CD status / logs / rollback (attested apply plane) | | `incident` | CD failure disposition / A2H | | `trails` | Continuous migration recipes (`status` \| `plan` \| `inspect`) | | `station` | Station **ops/setup** (not app lifecycle) | | `yard` | Bare-metal Yard install / inspect | | `broker` | Conductor config + resolve + matrix | | `serve` | Local control-plane HTTP+SSE for the Yard game UI | ## Authoring | Verb | Role | |---|---| | `new` | Scaffold | | `collect` | Resource collection → manifest IR | | `codegen` | Regenerate project bindings (`--check` fails on drift) | | `build` | Compile to wasm components (OCI fallback) | | `stub` | Contract-backed stub behind a WIT port | | `generate` | Generation pipeline entry | | `sandbox` | Isolated lane; out-of-scope boundaries from contracts | | `test` | Contract-backed sandbox test lane (not a jest replacement) | ## Observe & operate | Verb | Role | |---|---| | `inspect` | Projections / time-travel (`run`, `deploy`, `topology`, …) | | `logs` | Live-tail Camp Streamer | | `replay` | **Session Replay** — `list` \| `get` \| `open` \| `summarize` \| `retention` \| `doctor` | | `doctor` | Environment / credential / portability hazards (incl. messaging readiness) | | `messaging` | Inspect messaging capability and delivery-guarantee readiness | | `drift` | Drift + governed remediation | | `runbook` | Execute runbooks (writes still policy-gated) | | `attest` | Signed evidence exports from audit facts | | `erase` | Crypto-shred subject + projection rebuild | | `derivative` | Derived implementation upgrade / conformance | ## Distribution | Verb | Role | |---|---| | `add` | Install provider or listing (sig + provenance + conformance) | | `registry` | Registry admin; signed offline bundle import | | `import` | Adopt estate in place — never destroy/recreate on import | ## Station context For `station`, `broker`, `connection`, and station-aware `up` / `login`: 1. `--station` / `--station-uri` 2. `HOBO_STATION` / `HOBO_STATION_URI` 3. `profile.defaultStation` 4. Fail closed with remediation `hobo station init` Receipts pin `stationId` + `railHash`. ## Policy mapping Control-plane verbs map to `hobo::` policy actions (**deny-by-default**). Unmapped verbs fail conformance. ## Related [The hobo CLI](/HoBo/guide/cli/) · [MCP](/HoBo/agents/mcp/) · [Stations](/HoBo/primitives/stations/) · [Trail map](/HoBo/guide/status/) --- # WIT & storage HoBo’s programmable surface is **WIT/WASM** (Component Model). TypeScript is the first reference language, not the only runtime. Storage for components is a **host-supplied capability**, not “open a file path from guest code.” ## `hobo:storage@0.1.0` | Rule | Detail | |---|---| | Surface | WIT capability bound by the host | | Guest sees | Engine-neutral storage operations aligned with EventStore semantics | | Guest must not see | SQL dialects, filesystem paths, driver pragmas | | Host implements | SQLite partition, future remote adapters, in-memory test fakes | Conformance tests prove guests behave identically across hosts that implement the WIT. ## Why capability-shaped storage 1. **Move cheap** — swap host engine without rewriting app components. 2. **Deny-by-default sandbox** — WASM capability security is a structural isolation dimension. 3. **Air-gap friendly** — host can bind in-boundary storage only; no egress required for durability. ## Related WIT surfaces (experimental / evolving) | Interface | Role | |---|---| | `hobo:storage` | Event/journal durability | | `hobo:secrets` | Residual secrets store bridge | | Runtime WIT ports | Broader Camp capabilities as they stabilize | Stability tags on port descriptors (`stable` | `preview` | `experimental`) apply — do not treat experimental WIT as frozen. ## Coupling to engines SQLite is the **default host engine**, not the public portability boundary. Apps and components depend on ports/WIT; operators choose engines that pass the conformance suite. ## Build path `hobo build` compiles services to wasm components with OCI fallback. Drift between declared contracts and generated artifacts fails closed. ## Related [Capability map](/HoBo/reference/packages/) · [EventStore](/HoBo/reference/eventstore/) · [Architecture](/HoBo/architecture/) · [WASM guide](/HoBo/architecture/wasm/) --- # Identity ports **Identity services** own workload identity, connection lifecycle, continuous identity migration, config resolution ladder integration points, and progressive Harness adapters that identity continuous reuses. App authors rarely touch them directly; CLI (`hobo identity`, `hobo doctor`) and the Yard Gateway do. ## Responsibilities | Concern | Surface | |---|---| | Issue / rotate workload SVIDs | SPIFFE-style identities per Camp/service | | Broker short-lived provider tokens | Capability-scoped grants after policy | | Connections | Authorize, status, events — Station-bound | | Continuous identity migration | Dual-path `PrincipalRef` continuity | | Dual fixtures | Offline -style fixtures for dual-path tests | | Attestation | Measurement / launch attestation documents | | Credential audit | **Refs only** — never plaintext secrets in events | ## Continuous identity migration When IdPs or principal shapes change: 1. Matching/policy drives **dual-path** continuity. 2. **Zero required Bindle config** for pure identity continuous. 3. Dual fixtures extend offline fixtures; live paths stay policy-gated. 4. Observe with `hobo identity status` and `hobo doctor`. 5. Do **not** invent a parallel IdP product or Bindle migration DSL. Trails may compose `identity.continuous` as a step kind; pure identity continuous still needs no Trail authoring. ## Token events & retention Token issuance, rotation, and access facts use `token.*` / `identity.*` families. `CredentialAudit` and `secret.Accessed` store **references and scopes**, never material. Audit streams are never-compact by policy. ## Config ladder (handoff) Identity brokers federation; secretless config resolution ranks: 1. `bound-resource` 2. `provider-federation` 3. `camp-value` 4. `static-escape` (≤7d residual) Full ladder: [Config & secrets](/HoBo/reference/config-secrets/). ## Gateway auth Gateway authenticates humans (OIDC/SSO) and agents/workloads (SVIDs), then authorizes ABAC over `(principal, tenant, port, resource, action)` — again at the broker for placement actions. ## Related [Identity guide](/HoBo/security/identity/) · [Config & secrets](/HoBo/reference/config-secrets/) · [Agent credentials](/HoBo/security/credentials/) · [Trails](/HoBo/primitives/trails/) --- # Naming & endpoints HoBo names are **pure functions** of tenant/project/env/service labels — never free-form strings that invent a private TLD. The naming helpers produce DNS-valid hosts; connectivity services use them for registration, DNS, and overlay. ## Hard rules | Rule | Why | |---|---| | **Never emit bare `.hobo`** | Not a public TLD; code that does is a bug | | **Local zone** | `*.hobo.localhost` (default local TLD `localhost`) | | **Overlay zone** | `*.hobo.test` (default overlay TLD `test`) | | **DNS-valid labels** | Lowercase, length-capped, no illegal chars | | **Hierarchical** | Tenant → project → env → service (and optional branch) compose the host | | **Prod env omission** | Default omitted env is `prod` so production hosts stay short | ## Modes ```ts type NamedEndpointMode = "local" | "overlay"; ``` - **local** — machine-local resolution; pairs with optional **portless** local named endpoints when the connectivity host supports them. - **overlay** — Tailscale/Netbird-class mesh names under `.hobo.test`. Port-based URLs remain the default offline path. Force ports with `HOBO_PORTLESS=0`. ## Multi-tenant context Soft multi-tenancy labels (also used by policy and metering): | Env var | Role | |---|---| | `HOBO_TENANT` | Platform tenant / org | | `HOBO_PROJECT` | Project | | `HOBO_BRANCH` | Branch / preview lane | | `HOBO_ENV` | Environment (`prod` often omitted in host) | When tenant is required for a named endpoint, empty tenant fails closed. ## Who registers names - **Connectivity host** — local registration of named endpoints for `hobo dev`. - **Pluggable DNS** — public managers (e.g. Route53, Cloudflare adapters) and memory providers for tests. - **Junction dynamic DNS** — cross-yard service discovery without shared mutable state. Workloads dial **named endpoints only** for secretless connectors — never receive raw material or ad-hoc host strings from operators. ## Related [Security](/HoBo/security/) · [Connectivity](/HoBo/reference/connectivity/) · [Camps](/HoBo/primitives/camps/) · [Stations](/HoBo/primitives/stations/) --- # Connectivity **Connectivity** is the multi-tenant networking foundation: hierarchical naming consumers, overlay (Tailscale/Netbird-class), pluggable DNS, policy-aware isolation, secretless connector plugins, and metering sink hooks. ## Subsystems | Area | Role | |---|---| | **Naming** | Builds on pure naming helpers; registers local/overlay endpoints | | **Overlay** | Mesh membership for `.hobo.test` style reachability | | **DNS** | Memory, public managers (Route53/Cloudflare adapters), Junction dynamic DNS | | **Secretless connectors** | postgres/tcp/mysql/redis (+ registry) — workloads dial **names**, not material | | **Isolation tiers** | Env isolation classes; fail closed across tiers without grant | | **PAM** | Privileged Access Gateway sessions (also Station-aware) | | **Metering** | Sinks for connectivity-adjacent usage facts | | **Tenancy** | Soft multi-tenant context enforcement helpers | ## Secretless connectors | Rule | Detail | |---|---| | Workload dials | Named endpoint only | | Material | Brokered by identity/connectivity — never ambient long-lived keys in app | | Protocols | Plugin registry (postgres, tcp, mysql, redis, …) | | Residual | Classical URL+password only via explicit residual path + audit | ## Isolation Hard environment isolation and optional **Privileged Access Gateway** bound who can reach what. Cross-Station Connection use fails closed without federation grant (Station package owns occupancy; connectivity enforces dial policy). ## Local DX - Named endpoints + optional portless mode for `hobo dev` - `HOBO_PORTLESS=0` forces classic ports - Connectivity host handler serves local registration without cloud DNS ## What not to do - Invent bare `.hobo` hosts - Pass provider secrets into wasm guests “for convenience” - Pool free-tier network allowances across tenants ## Related [Naming](/HoBo/reference/naming/) · [Security](/HoBo/security/) · [Identity](/HoBo/reference/identity/) · [Stations](/HoBo/primitives/stations/) --- # Config & secrets HoBo is **secretless-first**. Long-lived static secrets are a **residual** legacy-compatible path, not the default architecture. ## Authoring (`hobo/bindles`) | Surface | Role | |---|---| | `config({ ... })` | Schema of env + secret fields | | `env.string()` / dual `env({...})` | Non-secret configuration | | `secret.string().source(...)` | Residual secret fields with explicit source | | `secretRef` / `fromBinding` | References, not values | | `secret(name)` resource | Catalog secret resource (residual) | Code generation materializes a typed config view for your handlers. Gatekeeper can require a **secretless-config** check so residual secrets stay intentional and time-boxed. ## Resolution ladder Config resolution ranks sources in this order: 1. **bound-resource** — capability from bound infrastructure 2. **provider-federation** — OIDC/workload identity exchange 3. **camp-value** — Camp-provided non-secret or brokered material 4. **static-escape** — time-boxed residual (≤7d class constraints) Camp adapters (Vault JWT/OIDC, AWS Secrets Manager, Azure Key Vault, …) fetch through **brokered** paths — workloads do not hold master keys. ## Events | Event | Payload rule | |---|---| | `config.Resolved` | Refs and ranks only | | `secret.Accessed` | Refs only — **never** material | Credential audit streams are never-compact. ## Local overrides Local non-sensitive overrides can be encrypted at rest for developer machines. Agents and `hobo dev` see a **schema-only view** plus contract-asserted fixtures — **no plaintext** in agent context, events, or IaC. ## Conformance Conformance fails closed on plaintext leakage patterns in the residual path. ## WIT Experimental `hobo:secrets` WIT capability for component hosts that need a residual secrets store bridge. ## Related [Identity](/HoBo/reference/identity/) · [Identity guide](/HoBo/security/identity/) · [Resources](/HoBo/primitives/resources/) · [Agent credentials](/HoBo/security/credentials/) --- # Metering Metering turns **replayable facts** into usage evidence for showback, FinOps projections, dispute support, and ranking — **not** silent invoice invention. ## Model 1. Workloads and control plane emit usage-related facts (`meter.UsageRecorded` and related; also derived from `run.*`, `deployment.*`, `resource.*`, `gen.*`). 2. A **metering bus** fans out to sinks (in-process, export, warehouse adapters). 3. FinOps views (showback, anomalies, forecasts) are **projections** over facts. 4. Humans and the FinOps Crew **advise**; breakers, tiers, and policy **enforce**. ## Dimension matrix (required for M0 contracts) | Dimension | Why | |---|---| | Tenant | Showback isolation + support lookup | | Workspace / Project | Customer grouping and policy reconciliation | | Environment | Prod vs preview vs test separation | | Camp / Bindle | Operational and package grouping | | Run / Agent | Runtime + AI attribution and dispute lineage | | Team | Org-level reporting | ## Fact classes (examples) | Class | Typical family | Sensitive? | |---|---|---| | Control-plane operation | `deployment.*` | No | | Runtime compute | `run.*` | No | | Storage / egress | `resource.*` | No | | Generation run | `gen.*` | Often yes | | Support adjustment | `meter.*` | Yes | Sensitive classes require redaction/retention rules before customer export. ## Legal / ranking hygiene - Third-party **price and free-tier** facts are **ranking input** with source URL + human-checked date — never fabricated “vendor signed” attestations. - Free-tier headroom is **never pooled** across tenants. - Rotation is **across providers**, never across accounts within a provider to reset free tiers. - Metering/billing data stays **tenant-scoped**. ## FinOps Crew The FinOps operator agent reads metering projections, flags anomalies, and proposes commitment/move advice. It does **not** auto-spend or auto-destroy. See [FinOps Crew](/HoBo/operations/finops/). ## Related [Policy](/HoBo/reference/policy/) · [FinOps role](/HoBo/roles/finops/) · [Security](/HoBo/security/) --- # Session Replay **Session Replay** reconstructs user DOM/sessions (and Kit-level interactive capture) for debugging and support. It is **not** control-plane event-log replay and **not** auth `session.*`. ## Naming split (critical) | Concept | Meaning | Family / surface | |---|---|---| | **Event-log replay** | Fold durable facts for projections/audit | EventStore / `hobo inspect` | | **Auth session** | Login/refresh/revoke | `session.Started` / `Refreshed` / `Revoked` | | **Session Replay** | DOM/session reconstruction | `replay.*` + Session Replay | | **App Kit** | Bindle-declared app surface (`api`, `web`, `service`, …) | `hobo/bindles` `app kit kinds` | Heavy rrweb streams **never** enter the control-plane event log — only content-addressed blob refs. ## Defaults | Environment | Default | |---|---| | Local / dual fixtures / Station air-gap / every **App Kit** | **On** (zero Bindle config) | | Production / staging | **Off** unless OpenFeature `hobo.session-replay` or sample rate | | Kill switch | `HOBO_REPLAY=0` (or flag false) | ## Deep Bindle integration (App Kits) Every **App Kit** receives a default platform annotation `session-replay` at construction — **not only** `web`: | App Kit | Kind | Default Session Replay | |---|---|---| | API Kit | `api` | **On** | | Web Kit | `web` | **On** | | Service Kit | `service` | **On** | | UI Kit | `ui` | **On** | | Realtime Kit | `websocket` | **On** | | Job Kit | `job` | **On** | | Component Kit | `wasm-component` | **On** | **Infra Kits** (`kv`, `bucket`, `queue`, `topic`, `secret`, `data`, …) are **not** entrypoint hosts — no default Session Replay. Local Yard / `hobo dev` wraps every **HTTP App Kit** (`api`, `web`, `service`, `ui`) so HTML gets a privacy-first bootstrap and `/__hobo/replay/*` serves ingest/list — **no app tracker code required**. JSON API Kits still expose the ingest path on the same wrap. | Piece | Package / surface | |---|---| | Annotation helper | `sessionReplay({ enabled?, sampleRate? })` from `hobo/bindles` | | Kit taxonomy | `app kit kinds`, `isAppKitKind`, `appKitLabel` | | Runtime plan + wrap | Session Replay bindle integration | | Local host | `hobo dev` / Yard entrypoints | | Ingest path | `/__hobo/replay/*` | | Response marker | `x-hobo-session-replay: 1` when HTML wrap applied | ```ts import { bindle, sessionReplay } from "hobo/bindles"; export default bindle("shop", (b) => { const api = b.api(() => apiHandler); // API Kit — Session Replay default-on const web = b.web(() => storefront); // Web Kit — Session Replay default-on const worker = b.service(() => worker); // Service Kit — Session Replay default-on const cache = b.kv(); // Infra Kit — no default Session Replay // opt out one Kit: // const api = b.api(() => apiHandler).with(sessionReplay({ enabled: false })); return { api, web, worker, cache }; }); ``` ## Privacy (client-side before send/persist) - Mask password / email / tel inputs - Mask `[data-hobo-mask]`, `.hobo-mask`, payment autocomplete fields - Redact `Authorization` / `Cookie` in network capture - Continuous-identity **`principalRef` only** — no free-text PII ## CLI / MCP ```text hobo replay doctor hobo replay list hobo replay get hobo replay summarize hobo replay open hobo replay retention ``` MCP tool parity via canonical verbs. Player surfaces are read-oriented; mutation stays on CLI under policy/PAM. `hobo doctor` lists contract boundary ids. ## Event family (`replay.*`) `Started` · `ChunkIngested` · `Ended` · `ErrorLinked` · `RetentionApplied` · `Accessed` · `Disabled` · `Metered` ## Composition | Composes with | How | |---|---| | **App Kits** | Default annotation + host wrap on every Kit | | **Station** | Isolation + local path | | **Harness / OpenFeature** | Prod enablement + sampling | | **Trails** | Platform recipe `replay.retention` | | **Continuous identity** | Principal refs only | | **Metering** | Ingest bytes, storage, player access | | **PAM** | Access policy for open/list | | **PACT** | Integration boundaries via the replay HTTP surface | ## Performance budget (targets) | Metric | Budget | |---|---| | Batch interval | 1000 ms | | Batch size | 50 events | | Main-thread per flush | ≤ 16 ms | | Tracker overhead beyond rrweb | ≤ ~15 KB gzipped | ## Honest scope - Live multi-cloud object storage + Junction federation: **ports present**; dual fixtures prove the path - Full OpenReplay Assist / Helm stack: **optional**, not default - High-volume product analytics warehouse: **residual** ## Related [Bindles](/HoBo/primitives/bindles/) · [App Kits](/HoBo/primitives/app-kits/) · [Trail map](/HoBo/guide/status/) · [Harness](/HoBo/reference/harness/) · [CLI](/HoBo/reference/cli/) · [Event families](/HoBo/reference/events/) · [Dual fixtures](/HoBo/testing/dual-fixtures/) · [Stations](/HoBo/primitives/stations/) --- # Glossary | Term | Definition | |---|---| | **Bindle** | Provider-agnostic packed app: services, resources, wiring, policy intent | | **Camp** | Environment shape + runtime requirements; current home address (not provider selection) | | **Yard** | Provider selection point + camps; control-plane scope (policy, identity, reconciliation) | | **Junction** | Cross-yard federation; peer yards, no shared mutable state | | **Conductor / Broker** | Policy-governed placement authority (Conductor) | | **Station** | Hard control-plane isolation boundary a Conductor occupies | | **Rail** | Content-addressed Station profile/bootstrap pin | | **Trail** | App-owned continuous migration recipe | | **OpenFeature** | CNCF flag evaluation API; sole public app flag surface under Harness | | **Matching** | Resource definition + placement selection process | | **ResourceActor** | Single-writer reconcilers for infrastructure resources | | **Gatekeeper** | Composite deploy gate: contracts + policy + obligations + cost (+ more) | | **Contract IR** | Versioned `BoundaryContract` between producer and consumer edges | | **EffectSpec** | Optional side-effect annotation (purity + domains) on a contract | | **EffectManifest** | Exportable summary of a contract’s effects | | **ResilienceProfile** | Secondary resiliency metadata (retry/timeout/…); not a primary noun | | **EventStore** | Append-only log port for domain/control facts | | **DurableEventStore** | EventStore + positions, commit-with-effects, durable checkpoints | | **EventPosition** | Partition-local monotonic store position (`bigint`) | | **SQLite partition** | One DB file, one active writer; scale by partitions | | **ActorHost** | Single-writer lease + fencing token for aggregates | | **fenceToken** | Monotonic lease epoch (`bigint`) paired with stream `owner_epoch` | | **Streamer** | High-volume frame journal outside the domain event log | | **Queue** | Deferred work port with leases and idempotency | | **Clock** | Deterministic time port for workflows and tests | | **Crew** | Built-in operator agents in the Yard | | **A2H** | Agent-to-human handoff with obligation deadlines | | **Runbook** | Trigger → diagnose → gated fix, event-sourced | | **PACT** | Consumer-driven interaction tests under Contract IR | | **Bridge** | Session-scoped local/remote boundary routing modes | | **Harness** | Flags / progressive delivery / experiments plane | | **Lane** | Sandbox isolation for experimental traffic/contracts | | **Golden suite** | Signed multi-check deploy gate bundle | | **SVID** | Workload/agent SPIFFE-style identity document | | **Relay** | Outbound network proxy keeping cloud creds in-boundary — *not* an agent | | **Named endpoint** | Hierarchical DNS-valid host under `.hobo.localhost` / `.hobo.test` | | **Resource intent** | Provider-neutral desired resource used in planning/FinOps routing | | **Showback** | Usage attribution projection — not an invoice by itself | | **Secretless-first** | Prefer federation/bindings over stored secrets | | **can-deploy** | Compatibility matrix diagnostic for producer/consumer skew | | **can-trail** | Gate dimension for Trail readiness | | **WIT** | Wasm Interface Types — portability contract for capabilities (wRPC) | | **App Kit (Kit)** | Bindle-declared application surface (`api`, `web`, `service`, `ui`, `websocket`, `job`, `wasm-component`). Session Replay default-on. | | **Infra Kit** | Bindle port/infra resource (`kv`, `bucket`, `queue`, …) — not an entrypoint host; no default Session Replay | | **Session Replay** | Privacy-first DOM/session reconstruction on every App Kit (Session Replay, ); `replay.*` — not event-log replay | | **Messaging** | Backend-agnostic publish/subscribe over the messaging port / `topic()` with delivery guarantees, consumer groups, DLQ | | **Kafka capability provider** | Kafka provider exporting the messaging port for Kafka-compatible brokers; secretless bindings | | **Dapr runtime adapter** | Optional optional Dapr runtime Camp runtime realizing ActorHost + pubsub; never the default | | **Event-log replay** | Folding durable EventStore facts for projections/audit — different from Session Replay | Longer design glossary lives in the repo design suite; this page is the **site-facing** subset for humans and agents reading Pages. --- # Who writes what HoBo splits cleanly along the lines most teams already have. Each role owns a clear artifact or surface. | Role | You write / own | The artifact answers | |---|---|---| | **[Developers](/HoBo/roles/developers/)** | [Bindles](/HoBo/primitives/bindles/), [Trails](/HoBo/primitives/trails/) | *What is this app, and what does it need?* | | **[Platform Engineers](/HoBo/roles/platform-engineers/)** | Camps, Yards, Junctions, Matching | *Where may apps live, and how are those places run?* | | **[DevSecOps](/HoBo/roles/devsecops/)** | Policy packs, guardrails | *What is allowed — and what may cross a border?* | | **[SREs](/HoBo/roles/sre/)** | Runbooks, SLOs | *How do we detect, diagnose, and safely fix?* | | **[FinOps](/HoBo/roles/finops/)** | Budgets, showback expectations | *What does it cost, and is the move worth it?* | | **[Agents](/HoBo/agents/)** | Scoped tasks via SDK/CLI/MCP | *What can an autonomous principal do under policy?* | Credentials and audit trails are **not** role docs — see [Security → Agent credentials](/HoBo/security/credentials/) and [Observability → Agent audit](/HoBo/observability/agent-audit/). ## Related [Operations](/HoBo/operations/) · [The Crew (custom ACP)](/HoBo/crew/) · [MCP](/HoBo/agents/mcp/) --- # HoBo for Developers **You write [Bindles](/HoBo/primitives/bindles/).** That's the whole job description. A bindle is your app packed into one provider-agnostic bundle — services, resources, wiring, and permissions — written in the same language as the app itself. ## Your artifact ```ts // src/bindle.ts — this is the only infrastructure you author import { bindle } from "hobo/bindles"; import { app as api } from "./api"; import { app as web } from "./webapp"; export default bindle("my-app", (b) => { const db = b.data(); const jobs = b.queue(); const apiSvc = b .api(() => api.fetch) .dependsOn(db, jobs) // wiring = typed runtime bindings .allow("db:read", "db:write", "queue:send"); // least privilege, in the diff const webSvc = b.web(() => web.fetch).dependsOn(apiSvc); return { db, jobs, api: apiSvc, web: webSvc }; }); ``` ## Your loop ```sh hobo dev # offline local camp, hot reload, replay hobo up # deploy: plan, gate, provision, converge hobo logs # live-tail through the camp runtime hobo inspect # time-travel projections of any run ``` ## What you never do - **Pick a cloud.** Placement is the platform team's map and the Dispatcher's math. Your bindle runs wherever it's cheapest to run — and moves when somewhere better opens up, without you editing anything. - **Write YAML, Terraform, or IAM JSON.** Resources are `b.data()`, `b.queue()`, `b.bucket()`. Permissions are `.allow(...)` next to the code they protect, reviewed in your PR like any other line. - **File infra tickets.** If the port exists, you declare it and ship. Governance happens in the gate, not in your calendar. ## What your `.allow()` means to everyone else Your grants are the security team's review surface and the platform's linking contract: deny-by-default, so a resource you didn't declare and a verb you didn't allow simply don't exist at runtime. Keep grants tight and the rest of the pipeline — [policy checks](/HoBo/roles/devsecops/), migration gates, agent audits — stays fast and quiet. ## When you'd peek next door - Your app needs provider-native tuning: add a typed `.with(...)` annotation — still inside the bindle, still one diff. - You're curious where your app actually lives today: `hobo inspect topology`. (It's allowed to surprise you. That's the product.) - You've outgrown one region or cloud: that's [platform territory](/HoBo/roles/platform-engineers/) — your bindle doesn't change. --- # HoBo for Platform Engineers **You write [Camps](/HoBo/primitives/camps/), [Yards](/HoBo/primitives/yards/), and [Junctions](/HoBo/primitives/junctions/).** Developers describe apps; you build the map those apps roam: which providers are approved, what a production environment looks like, how a provider's camps are governed together, and how clouds federate. ## Your artifacts ```ts // Camps: environment shapes only — no bindle ownership, no provider pick import { camp } from "hobo/camps"; export const prodEast = camp("prod-east", { envType: "prod", prefer: ["low-latency"], runtimeRequirements: ["runtime-local"], labels: { region: "us-east-1" }, }); export const staging = camp("staging", { envType: "staging" }); ``` ```ts // Yard: provider selection lives here import { yard } from "hobo/yards"; import { prodEast, staging } from "./camps"; export const awsYard = yard("aws-east", { provider: "aws", regions: ["us-east-1"], camps: [prodEast, staging], }); // Junction: yards federated across clouds — peer to peer, no shared state import { junction } from "hobo/junctions"; export default junction("acme", { yards: [awsYard, azureYard] }); ``` ## What you own - **The approved-provider set.** Camp inference and Matching draw only from providers you've approved — the Dispatcher optimizes for price *inside your fence*. - **Golden-path camps.** Env templates for prod/staging/preview with requirements and labels already decided. - **Yard-level governance.** Provider pin, promotion rules between camps, shared identity, provider-global networking, drift posture (`hobo drift`), reconciliation cadence. - **Junction federation.** Which clouds peer, what connectivity is inferred, and which [policy packs](/HoBo/roles/devsecops/) apply at each border. You also hold the broker-admin switch: leave camp inference on, or lock placement down to explicit, approved camps. - **The self-hosting decision.** The Yard control plane is itself a bindle — run HoBo's control plane with the same tooling app teams use. ## Your loop ```sh hobo plan --camp prod-east # dry-run any change before it lands hobo verify # preflight the unified gate decision hobo drift # report drift; route governed remediation hobo up --camp prod-east # converge when gates pass ``` ## Related [Camps](/HoBo/primitives/camps/) · [Yards](/HoBo/primitives/yards/) · [Junctions](/HoBo/primitives/junctions/) · [Trail map](/HoBo/guide/status/) · [CLI](/HoBo/guide/cli/) --- # HoBo for DevSecOps **You write policy.** Not PDFs — the machine-enforced rules that every deploy, every inferred camp, every migration, and every credential grant must pass. In HoBo, policy isn't advisory review after the fact; it runs *inside* the convergence loop, and a failed check stops the mutation before it happens. ## Where your rules bite | Surface | What your policy decides | |---|---| | **The gate** | Every `hobo up` re-runs the applicable gate before mutation — a stale preflight can never authorize a deploy. Your checks are part of that gate. | | **Camp approval** | Inferred camps are approved by policy, not automatically. You define what an acceptable home looks like (provider, region, runtime, data class). | | **Migration gates** | Continuous migration only moves an app when the destination passes the same checks — cost never outranks compliance. | | **`.allow()` review** | Developers' least-privilege grants live in the bindle diff. Deny-by-default linking means an undeclared edge doesn't exist at runtime. | | **Credential brokering** | Humans *and* AI agents get short-lived, task-scoped tokens under a machine-readable capability standard — no static keys to leak. Issuance itself is an event. | | **Junction borders** | Your highest-leverage surface — see below. | ## Junctions are where your policy matters most A [junction](/HoBo/primitives/junctions/) is where clouds meet, so it's where sovereignty, residency, and egress live. Junction policy travels as **signed policy packs** — versioned artifacts federated to the member yards, because a junction shares no mutable state: - **Residency & sovereignty:** which data classes may live in which yards, and what may never cross a border — the Dispatcher can't propose a cross-cloud move your pack forbids. - **Egress & connectivity:** what the inferred cross-cloud connectivity plane is allowed to connect. - **Placement lockdown:** the junction broker admin can turn camp inference off entirely, requiring explicit, approved camps — placement as a deliberate act. - **Trust federation:** signed registry bundles and trust roots decide which providers and adapters are even installable (`hobo add` enforces signature, provenance, and conformance checks). ## Configuring the Conductor [The Conductor](/HoBo/primitives/stations/#the-conductor) — the broker that decides where every app lives — acts **only** on the policy set you configure. That policy set is one reviewable document, `hobo.broker.json`, and it is where budgets, cloud connectors, and your policy engine of choice all meet: ```bash hobo broker config --init # scaffold hobo.broker.json hobo broker config # validate it and print the policy hash hobo broker up # realize placements under it ``` A production-shaped config: ```json { "schemaVersion": "hobo.broker/v1", "brokerId": "broker:acme-prod", "placementPolicies": [ { "id": "pp-default", "scopeKind": "bindle", "objective": "min-cost" }, { "id": "pp-eu-lockdown", "scopeKind": "junction", "objective": "min-cost", "allowedRegions": ["eu-west-1", "eu-central-1"], "deniedProviders": ["legacy-vendor"], "requireExplicitCamps": true } ], "billing": { "quota": "deployments/day", "quotaLimit": 50, "budgetUsd": 2000, "spendCapUsd": 100 }, "connectors": [ { "id": "conn-github", "kind": "github-app", "displayName": "Acme HoBo App", "settings": { "appId": "12345", "installationId": "678" }, "credentialRef": "token-broker://github-app/acme" }, { "id": "conn-cf", "kind": "cloudflare", "displayName": "Acme Cloudflare", "settings": { "accountId": "acct-1" }, "credentialRef": "token-broker://cloudflare/acme" } ], "requiredConnectorKinds": ["github-app", "cloudflare"], "policyEngine": { "kind": "opa", "endpoint": "http://opa.policy.svc:8181" }, "delivery": { "engine": "argo", "syncMode": "auto", "gitopsRepo": "https://git.acme.dev/hobo-receipts", "destinationNamespace": "hobo-prod" } } ``` What each block buys you: - **Placement policies** layer junction → yard → camp → bindle; your restrictions (denied providers, region allowlists, the explicit-camp lockdown) *accumulate* — no narrower policy can undo them. - **Billing** is enforcement, not reporting: quotas, a per-resolution spend cap, and a budget the broker refuses to exceed — with a metered mode that folds real usage facts into what's left. - **Connectors** name the cloud accounts the Conductor may act through (GitHub App, Pulumi, Azure, Vercel, Cloudflare, custom). `credentialRef` points into the token broker — this file never contains a secret, so it belongs in review like any other code. - **`policyEngine`** swaps the decision engine: keep the built-in typed policy packs, or point at your **OPA/Rego** decision (`hobo/broker/decision`) or **Cedar** authorizer. Either way the broker *fails closed* — an unreachable policy engine denies, it never waves things through. Every evaluation records a `policy.Decided` fact, allows included. - **Delivery** hands accepted receipts to your GitOps rollout (Argo CD) or the in-proc executor. And for trying policy changes safely: run `hobo broker up` with **no** cloud config, and the running machine registers itself as a zero-cost local HoBo provider host — the same pipeline, the same rejection codes, no cloud bill. ## Evidence, not assertions Everything above lands in the same append-only event log as the deploys it governs: - **Every decision is a fact.** Policy passes, refusals, camp approvals, token grants — attributable, replayable, exportable. - **`hobo attest`** produces signed audit-evidence exports for your assessors. - **`hobo erase`** performs crypto-shred erasure with projection rebuild when data must genuinely disappear. - **Time travel is an audit tool:** "where did this workload live in March, under which policy version, and who approved the move?" is a query, not an investigation. ## Working with the other roles - [Developers](/HoBo/roles/developers/) hand you a reviewable `.allow()` surface in every PR — push grant hygiene there, and the rest of the pipeline stays quiet. - [Platform engineers](/HoBo/roles/platform-engineers/) own the map your policy constrains; the approved-provider set and your packs compose — the Dispatcher obeys both. - [Agents](/HoBo/agents/) are principals under exactly your credential policy — no special lane, and every action they take is in your log. --- # HoBo for SREs You care about **MTTR, error budgets, and change safety**. HoBo makes the control and data planes replayable so “what happened?” is a query. ## Daily tools | Tool | Use | |---|---| | `hobo inspect` | Time-travel deployments/runs | | `hobo logs` | Streamer live-tail | | `hobo incident` / `runbook` | Disposition + gated fix | | Endpoint Monitor | SLO burn signals | | SRE Crew | Diagnose + propose; policy gates writes | | Chaos tests | Session faults, shadow, synthetics | ## Mindset shifts - Audit = event log, not ticket archaeology - Remediation is often canary + auto-rollback - Prod writes frequently require-approval - Never retry policy denials as “healing” ## Related [Crew SRE](/HoBo/operations/sre/) · [Observability](/HoBo/observability/) · [Chaos testing](/HoBo/testing/chaos/) · [Runbooks](/HoBo/operations/runbooks/) --- # HoBo for FinOps You care about **unit economics and accountability** without becoming the team that maintains every cloud console. ## What HoBo gives you | Capability | Why it matters | |---|---| | Metering facts + showback dimensions | Tenant/project/run attribution | | Continuous migration | Rent trends down when policy allows | | Policy budgets / breakers | Enforcement, not vibes | | FinOps Crew | Always-on anomaly and proposal assistant | | Dated ranking sources | Honest price inputs | ## What you should demand 1. No free-tier pooling stories. 2. No fabricated “vendor signed” price attestations. 3. Replayable dispute lineage from invoice-adjacent views to facts. 4. Clear separation of **ranking** vs **billing**. ## Related [Metering](/HoBo/reference/metering/) · [FinOps Crew](/HoBo/operations/finops/) · [Migration](/HoBo/cicd/migration/) --- # HoBo for agents HoBo is built for humans **and the agents that work for them** — not as a slogan, but as an identity model. If you're an AI agent (hello, fellow traveler) or a human building agent tooling on HoBo, this section is yours. > **Reading this as an agent?** The condensed, machine-readable briefing lives at [`llms.txt`](/HoBo/llms.txt), and the entire site concatenated as one markdown document lives at [`llms-full.txt`](/HoBo/llms-full.txt). Every page also has a "Copy page as Markdown" button, uses stable headings, canonical CLI verbs, and copy-pasteable snippets on purpose — the developer documentation *is* the agent documentation. ## Agents are principals, not scripts The standard failure mode elsewhere: an agent runs with a human's long-lived API key, invisible to audit and unbounded by scope. HoBo's model (the full brokered-identity flow below is contract-pinned design; the shipping `hobo-mcp` binary runs today with a deny-by-default, scoped local-operator identity and audits every decision): - **Own identity.** Agents get workload identity (SVID/OIDC), not borrowed keys. - **Brokered, short-lived tokens.** Secretless-first — credentials are issued per task, scoped to the task, and expire fast. - **Self-provisioning within policy.** A machine-readable **capability standard** describes what each capability needs, so an agent can request *exactly* the credentials a job requires — and policy decides whether it gets them. - **Every action is an event.** Token issuance, deploys, migrations — an agent's whole working day lands in the same append-only, replayable log as everything else. "What did the agent do?" is a query. ## One surface, three doors Every HoBo capability is reachable three ways, with identical semantics: | Door | For | Notes | |---|---|---| | **SDK** | code | The same typed objects the [primitives](/HoBo/primitives/) docs describe. | | **CLI** | humans & scripts | The [canonical verb surface](/HoBo/guide/cli/) — deliberately small, stable to memorize. | | **MCP** | agents | Ships today as the `hobo-mcp` stdio binary: the canonical verb set as tools, parity contract-tested — see [the MCP surface](/HoBo/agents/mcp/). | No agent-only side entrance with different rules: if policy would stop a human doing it, it stops an agent doing it, and vice versa. ## The Crew: operators included HoBo ships a built-in roster of operator agents — the **Crew**: SRE, endpoint-monitor, GitOps, and platform-engineer operators that run as autonomous, policy-gated services in the Yard. HoBo ships the operators, not just the APIs they'd call. Your agents can work alongside them, under the same identity model. ## Agents and continuous migration The guiding principle — *find the cheapest home that passes policy, keep looking* — is agent-shaped by design. The Dispatcher that plans migrations is itself an optimizing agent under policy gates, and third-party agents can participate: propose placements, watch spend, pre-approve moves within bounds you set. The event log means every optimization an agent makes is attributable and reversible. ## Generating new homes Missing a provider adapter? Agents can invoke the **generation pipeline**: synthesize an adapter for a port, then have the port's conformance suite certify it. A certified adapter is a certified adapter regardless of author — human or machine. More homes on the map, cheaper moves for everyone. The full extension surface — every implementable port, the adapter-authoring path, the conformance bar, and the ACP/A2A protocol-connector integration points — is documented in [Custom providers](/HoBo/customizations/providers/). ## Contracts agents can read (and test) When you call `hobo verify` / can-deploy or inspect a BoundaryContract, look for optional **effects** and **resilience** fields: - **EffectSpec** — purity + domain I/O tags. Pure contracts isolate with zero infrastructure; network/storage domains imply harness mocks. Effect widening is a compatibility signal. - **ResilienceProfile** — secondary retry/timeout/circuit-breaker metadata. Deterministic schedules only; **never** retry a `policy.Decided` deny or a terminal failure. Prefer reconstructing state from event facts over process memory. - **EffectManifest** — exportable summary for policy, SRE tooling, and training trajectories (with redaction). Durable Camp storage defaults to a **single-owner SQLite partition** behind EventStore ports. Do not assume multi-writer shared filesystems; do not invent parallel event-store engines when the ports already exist. ## Start here - [The MCP surface](/HoBo/agents/mcp/) — the shipping `hobo-mcp` binary: stdio transport, the canonical verb set as tools, `HOBO_MCP_TOOLS` scoping, deny-by-default audited decisions. - [hobo-code](/HoBo/guide/hobo-code/) — the interactive Bo session. Not a `hobo` verb. - [`llms.txt`](/HoBo/llms.txt) — the machine-readable site briefing. - [Architecture](/HoBo/architecture/) — package layout, durable EventStore default, contract effects. - [Testing](/HoBo/guide/testing/) — effect-aware isolation and offline durability tests. - [Getting started](/HoBo/guide/getting-started/) — the same ten-minute path humans take; it works verbatim for agents.