quickable

Operations

Observability through Victoria, the durable-state inventory, ramp-to-zero and restart-idempotency, the standing probes, and the honest residuals.

Observability (Victoria)

Every capability handler is a pumped-fn lite flow executed through bus.serve(), and the scope wires observability once — flows emit their signals structurally (A7′); a handler cannot forget to instrument itself.

  • Logs → VictoriaLogs (pino, ndjson push). Browse from the edge, basic-auth admin:

    curl -su admin:$ADMIN_PASSWORD \
  "$BASE/o11y/logs/select/logsql/query" --data-urlencode "query=_time:5m service:quickable-core"

  • Metrics → VictoriaMetrics (OTLP push + prom scrape):

    curl -su admin:$ADMIN_PASSWORD \
  "$BASE/o11y/metrics/api/v1/query?query=quickable_authz_violations_total"

  • Traces — the OTLP trace leg activates when VICTORIA_TRACES_URL is set on the services; the bundled compose file ships logs + metrics backends.

The admin credential is the ADMIN_HASH bcrypt env on the caddy service (the compose default corresponds to the password quickable-admin — override it for anything shared). Broker-level authorization violations are counted into quickable_authz_violations_total, so isolation (AG-1) is observable continuously, not only when probed.

Durable state — exactly two things (A8′)

VolumeHolds
credsOperator/account seeds, resolver JWT directory, per-service credentials, the generated nats config.
nats-dataThe JetStream store: all KV buckets (namespaces, apps, material, realtime state), the bundle object store, the event stream.

(vlogs-data / vmetrics-data hold telemetry history — operational, not application truth.) Every service is restart-stateless: kill and restart anything and the system reconverges from the broker.

  • Restart-idempotency is proven at the edge: qctl smoke run persists {ns, bearer, hash}; after a core restart qctl smoke verify asserts the same bearer still authenticates and the same app/manifest/bundle are still served.
  • Fenced online backup/restore ships as qctl backup <out.tar> / qctl restore <in.tar> (D3): a quiesce barrier sets a maintenance flag and broadcasts q.int.quiesce until every live core+transform replica acks drained-to-zero, then records a full-history manifest (per-message digest, every KV revision + tombstones, per-object digest), snapshots each JetStream stream, and captures creds — before thawing. Restore into a fresh stack re-seeds creds (so operator identity + all JWTs stay valid) and recomputes the same inventory, failing on any diff. qctl probe restore proves the round-trip is lossless under concurrent writes (tombstones and objects included).
  • Cold backup (copy the creds + nats-data volumes while stopped) remains a simple alternative; because the operator identity lives in creds, restored JWTs stay valid.

Ramp

  • To zero: docker compose -f deploy/compose.yaml down -v removes containers, internal networks and the named volumes — zero trash by construction (nothing is ever mounted from outside the compose project).
  • To many: core and transform subscribe as queue groups with no coordination state — docker compose -f deploy/compose.yaml up -d --scale core=4 --scale transform=2. Realtime correctness under replicas is the CAS design, not luck (see Architecture). The broker itself is single-node in this deliverable.

Standing probes

The anti-goal machinery ships as executable probes (run against a live stack; also exercised in the test suite):

ProbeProves
qctl probe isolationThe denial matrix: cross-namespace pub/sub, bridge-writes-material, expired bearer, JS API reach — every row must be a broker violation with zero delivery.
qctl probe gatewayREST/MCP callers get byte-identical broker enforcement; forged/absent/expired bearers rejected at connect; the route inventory is complete.
qctl probe app-lifecycle / material-transformCreate → put → build → serve, through real primitives.
qctl probe sandbox-rtRealtime races: concurrent conflicting intents across replicas serialize with no lost acknowledged move; dedup and resume behave.
qctl probe app-signal / material-signalAG-5: each capability's flow left a queryable trace in the Victoria backends.
qctl probe restoreAG-4: backup → wipe → restore round-trips losslessly under concurrent writes (full-history manifest diff empty, tombstones + objects preserved); ramp-to-zero leaves no trash.
qctl smoke run/verify <url>The full pipeline through the public edge, plus restart-idempotency.

Honest residuals

Documented limits, not fine print:

  • The sandbox is connection-sealed, not egress-sealed. Sandboxed code cannot reach server NATS or any credential, but script-enabled content can navigate its own frame and leak what it was given. Mitigation is scoping: an iframe only ever receives its own app's data (D4). Do not deliver secrets into an app you do not trust.
  • Realtime idempotency is window-bounded. A cmdId retried after the dedup window (60 s) re-applies as a new command. Clients must not retry beyond it.
  • Event fanout is best-effort. KV is the truth; a client that misses events reconciles via resume. Design consumers around revisions, not around receiving every event.
  • Single-node broker backup. The fenced qctl backup/restore is lossless for this single-node deliverable; cross-node JetStream cluster snapshots are out of scope.
  • Single-node broker. JetStream clustering/HA is out of scope for this deliverable; scale-out applies to the stateless services.
  • QSVC credentials are tenant-root. Inherent to a platform account; contained by A15′ (never on a user-traffic path, static-read-only in the gateway, server containers only).

A couch multiplayer game

Server-authoritative realtime with optimistic CAS — a shared tally game two people play from their phones.

On this page

Observability (Victoria)Durable state — exactly two things (A8′)RampStanding probesHonest residuals