Peer integration

How sibling engines and AGPL services attach to the Signals hub and consume the critical plane.

Signals holds control: signals.target, peer-contract, lattice-ci, platform Metaflow, YuniKorn, Atlas, the warehouse. Product engines (Gaius, Ægir, Atelier, Hermes) keep native gRPC and add zndx.engine.v1. See signals-protocol.

End state (lab): after each peer finishes its local unit work and is enabled under the group:

sudo systemctl start signals.target
# → foundation (signals.service) + signals-ready + every enabled peer

Bare systemctl start signals starts only the foundation unit — not peers.

Two layers

                    ┌─────────────────────────────┐
                    │     signals.target          │
                    │  (group controller)         │
                    └─────────────┬───────────────┘
                                  │
           ┌──────────────────────┼──────────────────────┐
           ▼                      ▼                      ▼
   signals.service      signals-ready.service     signals-engine.service
   just up / down       just signals-ready        :50551 After=ready
           │                      │                      │
           ▼                      ▼                      ▼
   critical plane          PASS ⇒ exit 0           Engine + Scheduler
   PG Kudu Impala          Kudu+Metaflow           (enabled peers after ready)
   YK Metaflow AF          critical included       gaius · aegir · atelier
   Eventing Broker                                 hermes · synth · metabase (AGPL opt.)

Peers are peer-to-peer with each other (no ordering edges among engines). All order after signals-ready.service.

Foundation lifecycle

# Manual / CI
just up
just signals-ready          # check-only; exit 0 when critical plane is ready
just down                   # lattice-safe stop

# Group control
sudo systemctl start signals.target
systemctl list-dependencies signals.target
sudo systemctl stop signals.target

Do not treat devenv up -d returning as “ready.” Peers wait on signals-ready (or the oneshot that polls it). Prefer just down over bare devenv processes down so Postgres :5455 is released.

Host tools (system-wide)

Service units do not inherit devenv/nix PATH. Install under /usr/local/bin (or equivalent on systemd PATH):

See infra/systemd/README.md.

What peers consume (not re-host)

A peer that maintains its own product (Gaius prospects, Ægir corpora, …) follows Peer data products. Contract: data_products.md.

Postgres port lattice (do not collide):

How to organize work on queues

YuniKorn leaves are resource classes, not project names. There is no root.gaius (or root.aegir) GPU slice. Stamp identity on the Application and pick a leaf from resource-classes.md.

Required stamps:

• yunikorn.apache.org/queue annotation (placement rule is provided)
• yunikorn.apache.org/app-id = federation.workload_id
• federation.project for C2 → Yield
• Host GPU: request federation.zndx.org/gpu only (never nvidia.com/gpu on a CPU-only sentinel). In-cluster CUDA: both keys.

YK counts cpu/memory/GPU-token/maxapplications. Token, RPM, and subscription math stay in the peer engine.

Platform services we deploy (Metaflow, Airflow, Eventing sink): just redeploy. That is the K8s refresh — not systemctl restart signals.target. Signals just rebuild is the host Polarisfork loop (assemble components/polaris, restart devenv polaris, wait :8182) — same shape as Metabase just rebuild, different tree. Metabase stays a host peer under signals.target.

MCP catalog/suggest tools come after this tree is live. Assignment RPCs land in signals-protocol only after peers use that catalog.

signals-protocol engines (gRPC lattice)

Each peer registers zndx.engine.v1.Engine (federation face) beside any native service. That face is the shared federation contract in progress — it evolves as real peer needs land in engines and get promoted into signals-protocol (see protocol core — evolution). Peer units and lattice-ci only attach process lifecycle to whatever Status the contract currently requires; they do not define the full engine architecture.

Lab lattice:

# Reflection required on the lattice port (signals-protocol install requirement)
grpcurl -plaintext 127.0.0.1:<port> list   # must include zndx.engine.v1.Engine
grpcurl -plaintext 127.0.0.1:<port> zndx.engine.v1.Engine/Status
just lattice-ci                      # SKIP absent; PASS listening (uses reflection)
just lattice-ci --require gaius,aegir,atelier

Python engines: depend on grpcio-reflection and enable reflection at server start. Spec: engine_grpc.md — Server reflection · signals-protocol.

OIP (KServe Open Inference Protocol) is the long-term portable inference face; Complete remains a transitional convenience on many engines.

Common peer unit pattern (process attachment)

Reference implementations: Metabase (unit shape, license-external) and Gaius (core peer, first full lattice accept under signals.target). The process pattern is shared; engine lineage for core peers is Gaius/Ægir/Atelier — Metabase is license-external and isolated.

Doctrine: federation peerhood is total commitment — full engine capacity for that project, honest start/stop/restart, remediation on failure. Lattice Status is a gate on the real engine, not a substitute for it. Signals being early on its engine axis does not license half- committed peers or freeze future engine work here.

Must

1. After=signals-ready.service + Wants=signals-ready.service
   Soft dependency: foundation failure does not hard-fail the peer (Requires= only if you want a hard gate).

2. PartOf= / WantedBy=signals.target
   Group stop/restart and opt-in membership via systemctl enable/disable.

3. Wrappers in the peer tree (not signals, not multiline shell in the unit):

   • scripts/systemd_start.sh — just up / full devenv up for this project (capability engine + product UI/gateway/DB); block until lattice Status and product surfaces answer
   • scripts/systemd_stop.sh — peer-scoped full-stack stop of this unit’s devenv graph (and vLLM children); lattice + product ports free
     Unit ExecStart= / ExecStop= point at those absolute paths under the peer checkout. systemd rejects fragile multiline shell; Metabase hit this first. Reference shapes: Gaius / Metabase (already full-stack); Ægir / Atelier remediated 2026-08-13 (were engine-only — incorrect under doctrine).

4. Accept = full stack ready, not “Status alone” or “engine-only”:

   • Codegen Status (preferred in-peer): generated zndx.engine.v1 stubs, project matches contract (see Signals scripts/zndx_engine_status.py / Gaius scripts/zndx_status_ok.py)
   • Honest health — no synthetic always-healthy capabilities when backends are down (total-commitment doctrine)
   • Product UI/gateway as that peer documents (e.g. Ægir :8091+:5173, Atelier :8090+:3000, Metabase :3200)
   • Reflection on the lattice port so external bare grpcurl works (install grpcio-reflection / enable ServerReflection — required)
   • TCP listen alone is not enough (Gaius lesson)
   • Transition failures → remediation (peer-local health/FMEA; e.g. Gaius /health fix engine)

5. Idempotent start only when the unit already owns a healthy single listener
   If accept probes pass and this unit is the sole owner of the lattice port with real capacity, exit 0 without tearing down a live stack. Multi- listener, orphans outside the unit cgroup, or optimistic Status that hides missing capacity are errors — stop/remediate, do not skip-up.

6. Single owner of the lattice port
   Avoid dual listeners (orphan devenv + unit). Reclaim or refuse multi-bind (Gaius lesson on :50051). Interactive devenv up must not fight the unit.

7. PATH
   System-wide just / grpcurl / kubectl on systemd PATH; bash -lc inside wrappers when devenv/direnv is required.

8. Never bind Signals :5455 or RustFS :9010.

9. Product doc in the peer tree (Gaius pattern): docs/current/src/operations/peer-unit.md — unit SoR for that project; not mesh/FEDERATION.md history.

Unit stop: peer-scoped (not “soft”)

Do not call this “soft stop.” That sounds like incomplete shutdown. The requirement is full stop of this peer’s unit, scoped so co-tenants survive.

Why the confusion: early Gaius unit stop used just down / devenv processes down and explicitly avoided teardown recipes that historically killed co-tenant GPUs. That is lease-safe / co-tenant-safe, not “don’t really stop.” Engine-only units (Ægir, Atelier) should stop the engine hard; they should not run product teardown.

Preferred names: peer-scoped stop, unit stop, or lease-safe stop.

Operator flow (any peer)

# In peer repo: implement wrappers + federation Status; standalone accept
# In signals:
just install-systemd --enable --start                    # foundation once
just install-systemd --peers <id> --enable               # opt-in peer sample
# Edit /etc/systemd/system/<id>.service paths if checkout ≠ lab default
sudo systemctl daemon-reload
sudo systemctl start signals.target                      # one-command group
just lattice-ci --require <id>

Acceptance templates: peer-unit-spec.

Federated in-org peers

These sections are the operations reference for peer-repo integration sessions and for Signals operators who enable the peer after that work lands.

Tick accept in peer-unit-spec.

Gaius

Landed in Gaius tree (peer session):

1. scripts/systemd_start.sh / systemd_stop.sh — wait for Status (not only TCP).
2. GaiusZndxEngineServicer on the existing gRPC server (project=gaius, capability cognition).
3. Product doc docs/current/src/operations/peer-unit.md.
4. Unit installed: just install-systemd --peers gaius --enable (lab host).

Lessons for other peers (from Gaius):

Operator — finish accept (after engine recycle):

# In Gaius: restart gaius-engine so the lattice servicer is bound
# e.g. devenv processes restart gaius-engine   (from this checkout only)

cd ~/local/src/wxs/signals
sudo systemctl start signals.target    # or: sudo systemctl start gaius
grpcurl -plaintext 127.0.0.1:50051 zndx.engine.v1.Engine/Status
just lattice-ci --require gaius

Caution: Gaius-local Metabase on :3100 is not federation dashboard. That is the optional AGPL Metabase peer (:3200 / :50451).

Aegir

Ægir instruct / inference peer (product name often styled “Ægir”). Next peer session after Gaius.

Agent one-liner (full paths for the Aegir session):

Implement peer-unit@aegir per
/home/rch/local/src/wxs/signals/docs/current/src/operations/peer-unit-spec.md
(Filled: aegir) and
/home/rch/local/src/wxs/signals/docs/current/src/operations/peer-integration.md
(Aegir / #aegir). Contract:
/home/rch/local/src/wxs/signals/config/platform/peer-contract.json.
Mirror:
/home/rch/local/src/zndx/gaius/docs/current/src/operations/peer-unit.md,
/home/rch/local/src/zndx/gaius/scripts/systemd_start.sh,
/home/rch/local/src/zndx/gaius/scripts/systemd_stop.sh,
/home/rch/local/src/zndx/gaius/scripts/zndx_status_ok.py.
Full-stack under signals.target: unit = just up (capability-engine + gateway + vite). Accept = Status + product UI health.

Remediation (2026-08-13): earlier engine-only unit left product UI dark under signals.target — incorrect under full-stack doctrine. Wrappers now mirror Gaius/Metabase (just up / full-stack stop).

Operator:

just install-systemd --peers aegir --enable
sudo systemctl restart aegir   # or signals.target
curl -sf http://127.0.0.1:8091/api/health
curl -sf -o /dev/null http://127.0.0.1:5173/
grpcurl -plaintext 127.0.0.1:50151 zndx.engine.v1.Engine/Status
just lattice-ci --require aegir

Atelier

Remediation (2026-08-13): engine-only unit was an oversight; both lattice and product UI belong under the unit.

Operator:

just install-systemd --peers atelier --enable
sudo systemctl restart atelier
curl -sf -o /dev/null http://127.0.0.1:3000/
grpcurl -plaintext 127.0.0.1:50251 zndx.engine.v1.Engine/Status
just lattice-ci --require atelier

Hermes Agent

Hermes is an in-org agent peer (Status.project=hermes, capability agent). It is not a user-session unit: signals.target is a system group, so membership is hermes.service with WantedBy=signals.target. A ~/.config/systemd/user/hermes.service on default.target never joins the group and stays down after a host reboot.

just install-systemd --peers hermes --enable
sudo systemctl start hermes.service
grpcurl -plaintext 127.0.0.1:50651 zndx.engine.v1.Engine/Status
just lattice-ci --require hermes

External engines: Metabase (AGPL)

Metabase is external by license requirement (AGPL vs ASL2), not merely “optional product preference.” The Metabase engine (mbengine) is therefore inherently isolated: separate checkout, no vendoring into Signals or core peer trees, process + wire only. Architecturally it is distinct from the core federated engine family (Gaius, Ægir, Atelier, and future core projects such as synth or vigil). See Core vs license-external engines.

The core Signals stack does not require Metabase. Install only when you want federated dashboard capability.

License boundary

Optional install (operators)

# 1. Standalone product (AGPL tree)
cd ~/local/src/agpl/metabase
just up   # or just rebuild
curl -sf http://127.0.0.1:3200/api/health
grpcurl -plaintext 127.0.0.1:50451 zndx.engine.v1.Engine/Status

# 2. From signals — opt in
cd ~/local/src/wxs/signals
just install-systemd --enable --start
just install-systemd --peers metabase --enable
# Edit unit paths if needed
sudo systemctl start signals.target   # not bare "signals"

# 3. Accept
systemctl is-active metabase.service
just lattice-ci --require metabase

Federation leverage (Metabase)

1. Foundation ready before start.
2. Optional platform Metaflow profile for flows that feed dashboards.
3. CloudEvents via platform Broker when DAGs exist.
4. YK queues for any RKE2 work.
5. No second critical plane (engine/app DBs stay product-local).

Synth (stub)

Systemd membership (multi-peer)

# Foundation once
just install-systemd --enable --start

# Enable only peers whose local unit + Status work is done
just install-systemd --peers gaius,aegir,atelier,hermes,metabase --enable

sudo systemctl start signals.target
systemctl list-dependencies signals.target
just lattice-ci --require gaius,aegir,atelier,metabase   # whatever is enabled

• PartOf=signals.target — stop/restart of the group propagates.
• WantedBy=signals.target — enable/disable controls membership.
• Do not enable a peer until peer-unit-spec accept is green.

Lattice CI

just lattice-ci
just lattice-ci --require gaius,aegir,atelier
just lattice-ci --all
just lattice-ci --json

Elevated CI gate — not part of signals-ready (critical plane only).

Checklist for a new peer

1. Own lattice Postgres + gRPC ports (document in peer-contract.json).
2. Vendor signals-protocol; implement zndx.engine.v1.Engine/Status (+ OIP path).
3. Peer-tree scripts/systemd_{start,stop}.sh; unit points at them.
4. After=signals-ready.service; never bind :5455 / :9010.
5. Start blocks until lattice Status (and product health if required).
6. Platform Metaflow / CE / YK when joining production federation.
7. If license ≠ ASL2 → external tree only (Metabase pattern).
8. just lattice-ci --require <id> green; then install-systemd --peers <id> --enable.

Related

• Critical plane
• Platform Metaflow
• signals-protocol
• Development environment
• Peer unit acceptance spec