Signals protocol core

Signals holds lineage, governance metadata, and scheduler state for the federation. Peer engines — Gaius, Ægir, Atelier, Hermes, Metabase — discover and call those services over a shared gRPC contract.

Hermes Agent (Nous Research) joins as capability agent. Weathership plugins attach memory and context to that runtime; the official product remains Hermes. Memory providers are the documented extension point.

Wire contracts live in components/signals-protocol (trunk). Changes land on that trunk first (additive within a version). Every peer that pins the submodule then bumps to that same SHA, regenerates its bindings, and pushes in the same slice. A side branch is not a pin.

How the federation contract evolves

signals-protocol is the shared federation contract: engine-to-engine wire, capability semantics, co-tenancy, OIP mapping. It grows as peers need new shapes. The first cross-engine call showed why zndx.engine.v1.Engine sits beside each project’s native service. Complete, Remediate, GPU leases, and OIP tensors followed the same path: ship in a peer, then promote a stable shape into the protocol.

In-family engines (Gaius, Ægir, Atelier, planned siblings) share multi-service gRPC, capability-on-the-wire, private serving, lattice ports, and co-tenancy leases.

AGPL peers (Metabase) live in an isolated tree and speak the same wire. mbengine is a lattice participant (Status, unit under signals.target). See Core vs license-external engines.

Signals hosts the platform (governance SoR, critical plane, process group control) and pins the protocol submodule. When attachment work (peer units, lattice-ci, Metaflow/CE) meets a gap, prefer extending signals-protocol (or documenting an explicit interim) over forking per-project wire dialects.

Working stance: treat protocol growth as co-evolution with peer engines — especially core peers, where shared engine practice is densest — requirements materialize in real projects; the contract absorbs what should be shared. Expect substantial further foundation work on signals-protocol itself before “federation complete” is a fair claim.

Signals as hub

Gaius, Ægir, and Atelier carry product engines (multi-service gRPC, serving, co-tenancy, Remediate). Signals holds the scheduler engine on :50551 and the control plane those peers attach to:

What the hub centralizes:

Working mode while early: drive from hub control and protocol gaps; land proven depth in peers that already have engines (and promote stable shapes into signals-protocol). Peer sessions stay product-local; convergence is measured here (lattice-ci, unit graph, protocol bumps). Engine functionality in Signals itself can grow when the federated architecture is solid enough to host it honestly — maturity, not a permanent “thin product” identity.

Doctrine: total commitment as a federation peer

Adopting signals-protocol as a federation peer is total commitment, not a decorative Status endpoint on a half-running stack. That applies whether the peer is a mature product engine (Gaius today) or a younger tree: peerhood means full capacity as that project defines it, honest ops, and remediation — not a lattice-only shell built only to pass accept.

Anti-patterns (not federation):

• Engine-only unit start that leaves product UI/gateway dark while claiming peer ready
• Synthetic always-healthy Status rows so lattice-ci passes without resident capacity
• Unit start that no-ops because an orphan still answers Status
• Unit stop that leaves lattice ports, UIs, or engine process groups alive
• Interactive devenv up stacks fighting the systemd unit for the same lattice port
• Treating “Signals is thin/early” as doctrine that excuses half-committed peers or freezes engine growth here forever

Accept implication: lattice-ci / Engine/Status are necessary gates, not a substitute for full capacity. Peer-unit accept should include clean start/stop/restart of this peer’s engine under signals.target (single listener ownership, honest health). Detail: Peer unit acceptance spec and Peer integration.

Platform capability: Scheduler (zndx.scheduler.v1)

Signals’ first thick engine capability is federated scheduling (capability=scheduler; lab model=yunikorn, later PBS/SLURM). Thin clients call zndx.scheduler.v1.Scheduler on the Signals lattice (:50551); the engine privately talks to the backend and owns archive/current/scratch projection. See YuniKorn queue management (lab adapter).

Spec vs tooling surfaces

Reflection is not a substitute for the proto. First-party scripts should not prefer “raw grpcurl -proto” as a substitute for codegen — they should call generated clients from signals-protocol.

Python engines: grpcio-reflection + enable at server start. Lattice-ci runs both a generated Status client and a reflection check.

Language: brass-tacks naming

Name platform, wire, and foundational theory by the terms a practitioner (or inventor) already uses — not by product metaphor, softened ops slang, or in-house glosses that drift from the definition.

Two tests (fail either → rename until both pass, then document once):

1. Systems: Can someone who only knows the underlying tech (gRPC, systemd, CI) map the phrase to a concrete artifact — port, unit, service registration, gate — without a project glossary?
2. Theory: Would the people who invented (or standardly formalized) the concept recognize our names, types, and invariants as their object — not a vague cousin? Example: persistent homology should read cleanly to Gunnar Carlsson (or any TDA practitioner): Vietoris–Rips filtration, homology dimension, birth/death pairs, essential classes (death = +∞), not rebranded “shape scores.” Same bar for Dempster–Shafer (Bel/Pl/K), SAGE, TreeSHAP, Ollivier–Ricci, and future topology / belief / geometry work.

Product metaphors are fine in UI copy when they map 1:1 to a brass-tacks term documented here. Do not invent a second vocabulary for the same process shape or the same mathematical object.

Layers

  UIs / external engines          Federated engines (OIP + zndx.engine.v1)
  ┌──────────────┐                ┌─────────┬──────────┬─────────┐
  │ Marquez-web  │                │  Aegir  │ Atelier  │  Gaius  │
  │ Metabase*    │                │ :50151  │ :50251   │ :50051  │
  │ Atlas UI     │                └────┬────┴────┬─────┴────┬────┘
  └──────┬───────┘                     │         │          │
         │  HTTP                       │  multi-service gRPC
         │  /api/v1  /api/atlas        │  + OIP ModelInfer (horizon → now)
         ▼                             ▼
  ┌────────────────────────────────────────────────────────────┐
  │  SIGNALS (this product)                                     │
  │  · Atlas :21010  governance SoR + OpenLineage /api/v1       │
  │  · Ranger        tag/resource authz from Atlas tags         │
  │  · Marquez-web   OL UI (proxy only; no Marquez DB)          │
  │  · components/signals-protocol  shared protos + specs       │
  │  · components/hermes-agent      agent runtime (plugins)     │
  │  · PG/AGE + Kudu/FDW scale path                             │
  └────────────────────────────────────────────────────────────┘
         ▲
         │  MemoryProvider + context engine plugins
         │  (Weathership → official Hermes memory service)
  ┌──────┴───────┐
  │ Hermes Agent │  tools / hooks / multi-agent ops
  └──────────────┘

  * AGPL Metabase is a *license-external* peer (isolated engine, not core family):
    process + wire only; consumes catalog/lineage/authz; never SoR or vendored.

Core vs license-external engines

Two kinds of “peer” attach to Signals. Conflating them muddies protocol work and peer-unit sessions.

Metabase is external by requirement: introducing an engine there is inherently isolated so the product can federate without licensing contamination. Architecturally it is distinct from Gaius / Ægir / Atelier and from any future core Signals projects. Operator install and dual-map still matter; they do not make Metabase “another Atelier.”

Hard rules (unchanged):

• No Marquez database and no stock Marquez API as a second catalog.
• No Python OL facade beside Atlas.
• DST method stays in Atelier (and related labs); Signals stores outcomes (classifications, optional belief facets on runs) for governance and audit.
• Protocol evolution is additive within a version; breaking changes → v2.

Submodule

# already pinned
git submodule update --init components/signals-protocol

# bump when protocol trunk advances
git -C components/signals-protocol fetch origin
git -C components/signals-protocol checkout origin/trunk
# commit pin in signals

Discovery (Atlas, Ranger, lineage)

Federated and external engines must find centralized services without hardcoding lab topology. Today Status describes engine endpoints only. The protocol core extends discovery so a peer learns:

Discovery is informative + capability-scoped: advertising a URL does not grant access. Edge (CF Zero Trust / IdP), Kerberos (data plane), and Ranger policies still authorize. See Identity and access.

StatusResponse.surfaces[] (kind, url, healthy) is the v1 step. kind=primary is the product UI the waffle lists. Signals advertises http://$SIGNALS_ADVERTISE_HOST:9889 (host from SIGNALS_KRB_HOST / LAN hostname — never loopback). Do not invent peer UI URLs — list only what that engine’s Status returns.

Engine/ServerQuery is pairwise S2S (not gossip): REMOTES (named git remotes + head), PEERS (lattice host:port from peer-contract), SURFACES (same list as Status), QUEUES (occupancy leaf shape), WORKLOADS (models, capabilities, TP/PP requirements — not the queue). Empty is honest.

A later zndx.discovery.v1 package can grow auth hints if the surface expands.

Engines converge by:

1. Calling discovery (or reading a Signals-published service map).
2. Emitting OpenLineage to LINEAGE_OL (not a local Marquez).
3. Registering / classifying assets via ATLAS_GOVERNANCE (or Signals hooks).
4. Evaluating access via RANGER_AUTHZ (or plugins that pull Atlas tags).

External engines (Metabase and peers)

Metabase (AGPL) and similar BI/agent tools are license-external participants — architecturally isolated from the core engine family (see above). They are first-class on the wire and process group, not second SoRs and not core engine projects:

Signals may ship thin adapters (e.g. Metabase → dataset namespace mapping) under ops docs; the contract remains discovery + OL + Atlas/Ranger.

Optional install (separate AGPL checkout + metabase.service under signals.target, never vendored into this tree): Peer integration — Metabase.

KServe Open Inference Protocol (OIP)

signals-protocol already patterns after OIP and maps Complete/Status as a horizon peer. Multi-agent model ops make that horizon active:

Direction:

1. Map zndx.engine.v1.Complete ↔ OIP ModelInfer (text/tensor payloads) in the protocol spec (expand existing OIP mapping table).
2. Carry authz context on federation/OIP calls (caller principal, optional Atlas entity / classification scope) — additive proto fields; enforcement at the serving edge and Ranger.
3. Emit provenance — engines (or a Signals sidecar) POST RunEvents to /api/v1/lineage for model ops: inputs (prompt hashes / dataset refs), outputs, model id from CompleteResponse.model, capability name.
4. Optional OIP gateway in Signals or Gaius that dual-registers zndx.engine.v1 and OIP for the same capability set.

OIP is not a second governance plane. Atlas/Ranger remain law; OIP is the model invoke lingua franca; OL is the audit thread.

Relationship to Marquez

See OpenLineage + Atlas.

Phased delivery (protocol core)

Hermes / Weathership memory

Hermes discovers memory providers under plugins/memory/<name>/ and context engines under plugins/context_engine/<name>/ (plugin system). Only one external memory provider is active at a time (memory.provider in config), additive to built-in MEMORY.md / USER.md (memory providers).

Weathership (Signals-backed) should:

1. Implement MemoryProvider with prefetch, turn sync, session extract, and agent tools (search/store) — backed by Signals AGE/OL (and optional belief facets), not a greenfield DB.
2. Implement a context engine that compacts with lineage-aware retention (pre-compression extract into Weathership memory, similar spirit to ByteRover’s pre-compression extraction).
3. Use federation discovery for Atlas/Ranger/OL bases; emit OpenLineage for memory writes and model ops (authz + provenance).
4. Ship as installable Hermes plugins; eventually qualify as an official reasoning-enabled memory service provider in the Hermes ecosystem.
5. Keep superset capabilities fleet-side (and exposable as tools/skills): mechanistic interpretability (SAE/CLT), topological structure (PH, Ollivier–Ricci), multi-engine Complete/Remediate—without waiting for Hermes core to absorb them. Deeper core integration remains open later; plugins-first is the current strategy.

Submodule: components/hermes-agent.

Related

• Submodule: components/signals-protocol
• Submodule: components/hermes-agent
• OpenLineage + Atlas
• Identity and access
• Atlas → Kudu outbox
• Protocol README (federation face, OIP horizon, co-tenancy)
• Gaius: CO-TENANCY draft; hx.lineage; FEDERATION.md
• Aegir: polyglot OpenLineage; former gateway/marquez.py (shape reference only)
• Atelier: DST / classification method lab