Signals

Signals is the hub of a federated compute and data plane. Independent engines — Gaius, Ægir, Atelier, Hermes Agent — keep their own products, native gRPC, and models. They attach here: one protocol, one scheduler, one warehouse, one governance record.

If you arrived from weathership.org or from the GitHub README, this book is the operator and architecture reference for that hub. Clone this repository, bring the stack up, and every peer you start is already on the same wire.

What you get

Nautilus sits beside each engine as a deterministic supervisor (zndx.supervision.v1). Its phase gates and Brier ledger are drawn in one diagram. Classification (sigint) still lives in this tree: it writes SIGDG tags into the same Atlas record the rest of the federation already uses.

The lattice

  Gaius :50051          Ægir :50151         Atelier :50251
  cognition             instruct            referee
         \                   |                   /
          \                  |                  /
           ┌─────────────────┴─────────────────┐
           │   signals-protocol  (gRPC)        │
           │   zndx.engine.v1 · scheduler.v1   │
           └─────────────────┬─────────────────┘
                             │
                    Signals engine :50551
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
   YuniKorn :30080     Airflow :30800      Metaflow :30180
        │                    │                    │
        └────────────────────┼────────────────────┘
                             │
              Kudu + Iceberg + Impala + PostgreSQL
              Atlas / Ranger / OpenLineage
              RustFS · Polaris · signals-ui :9889

  Hermes :50651 (agent)     Metabase :50451 (dashboard)

A federated engine is independently operable and a full participant. Capabilities travel on the wire (cognition, instruct, referee, agent, scheduler, dashboard); the serving engine chooses the model and reports what it ran. Launchers seed the hub, read Engine/Status.surfaces, and walk ServerQuery PEERS. A peer Announces itself; the Federated menu fills in.

Hermes interactive sessions keep WebRTC media on the engine and declare a coordination Activity that claims root.internal.inference.agent-rtc — one GPU, only while the session runs. Airflow observes that claim; YuniKorn admits it.

Bring it up

Linux workstation, devenv + Nix, recurse submodules. Impala, Kudu, and the RKE2 critical plane are Linux.

git clone --recurse-submodules git@github.com:weathership/signals.git
cd signals
devenv shell

# One-time native builds on a fresh machine
devenv tasks run kudu:build-cpp
devenv tasks run impala:build

cp .env.example .env
just up                 # PostgreSQL 16+AGE, Kerberos, RustFS, Polaris,
                        # Atlas, Ranger, Kudu, Impala, signals-ui, …
just signals-ready      # PASS / WARN / FAIL

Open http://127.0.0.1:9889. Then:

just kinit
just lattice-ci         # Engine/Status + reflection
just test               # hermetic pytest
just docs-serve         # this book, live

Group lifecycle on a lab host:

just install-systemd --peers gaius,aegir,atelier --enable
sudo systemctl start signals.target

Details: Quick Start, devenv, Peer integration.

How this book is organized

1. Architecture — protocol, queues, Activities, Metaflow, the warehouse, Atlas OpenLineage, the control UI.
2. Components — the Apache and sibling trees this repo pins.
3. Operations — devenv, Kerberos, secrets, backup, joining as a peer.
4. Classification — sigint, SIGDG, Dempster–Shafer fusion, SAGE. Useful if you tag columns; optional if you are attaching an engine.
5. Reference — configuration, ontology, roadmaps.

Start with System Overview if you want the diagram, or signals-protocol if you are implementing a peer.

Quick Start

Prerequisites

• Linux workstation (Impala, Kudu, and the RKE2 critical plane)
• devenv + Nix; direnv recommended
• just, kubectl, grpcurl on PATH
• Recurse submodules on clone

Bring-up

git clone --recurse-submodules git@github.com:weathership/signals.git
cd signals
devenv shell

# One-time native builds if this machine has never built them
devenv tasks run kudu:build-cpp
devenv tasks run impala:build

cp .env.example .env
just up
just signals-ready

just up starts PostgreSQL 16 + AGE (:5455), Kerberos, RustFS, Polaris, Atlas, Ranger, Kudu, Impala, signals-ui, and preflights YuniKorn, Knative, Metaflow, and Airflow on RKE2.

Open http://127.0.0.1:9889 — that is the control plane.

just kinit
just kerberos-status    # Impala HS2 GSSAPI OK
just lattice-ci
just test               # pytest tests/
just behave             # tier-0 BDD; SIGNALS_BDD_TIER1=1 for the live stack
just docs-serve

Attach a peer

just install-systemd --peers gaius,aegir,atelier --enable
sudo systemctl start signals.target
just lattice-ci --require gaius,aegir,atelier

A new engine implements zndx.engine.v1 on its lattice port, waits on signals-ready.service, Announces, and consumes Metaflow, Atlas, and the warehouse. Contract: config/platform/peer-contract.json. Walkthrough: Peer integration.

Classify columns (optional)

The in-tree sigint pipeline samples Impala, fuses evidence, and writes Atlas tags:

just tag-dry-run default.my_table
just tag default.my_table

See Metadata Tagging.

Next

• System Overview
• signals-protocol
• Query engine
• devenv

System Overview

Signals is the federation hub: protocol pin, scheduler, warehouse, and governance record. Sibling engines attach over signals-protocol and consume this plane.

Layers

Planes

Critical-plane services (Postgres, RustFS, Atlas, Ranger, Kudu, Impala, YuniKorn, Knative, Metaflow, Airflow) come up together. just up is the entry point; just signals-ready is the check. See Critical plane.

Workloads

YuniKorn Applications use leaves such as root.internal.inference.{reasoning,coding,instruct,heavy,extract,agent-rtc}. Project is identity (federation.project). Interactive Hermes sessions claim agent-rtc for one GPU while they run.

Coordination Activities are two hops: local process → peer engine → Signals :50551 → Airflow. Only Signals talks to the DAG.

Classification

sigint classifies columns into the SIGDG taxonomy (Dempster–Shafer fusion, SAGE-measured features) and writes Atlas tags. It is one consumer of the governance record. See Metadata Tagging and Evidence Fusion.

Signals protocol core

Signals holds lineage, governance metadata, and scheduler state for the federation. Peer engines — Gaius, Ægir, Atelier, Hermes, Metabase — discover and call those services over a shared gRPC contract.

Hermes Agent (Nous Research) joins as capability agent. Weathership plugins attach memory and context to that runtime; the official product remains Hermes. Memory providers are the documented extension point.

Wire contracts live in components/signals-protocol (trunk). Changes land on that trunk first (additive within a version). Every peer that pins the submodule then bumps to that same SHA, regenerates its bindings, and pushes in the same slice. A side branch is not a pin.

How the federation contract evolves

signals-protocol is the shared federation contract: engine-to-engine wire, capability semantics, co-tenancy, OIP mapping. It grows as peers need new shapes. The first cross-engine call showed why zndx.engine.v1.Engine sits beside each project’s native service. Complete, Remediate, GPU leases, and OIP tensors followed the same path: ship in a peer, then promote a stable shape into the protocol.

In-family engines (Gaius, Ægir, Atelier, planned siblings) share multi-service gRPC, capability-on-the-wire, private serving, lattice ports, and co-tenancy leases.

AGPL peers (Metabase) live in an isolated tree and speak the same wire. mbengine is a lattice participant (Status, unit under signals.target). See Core vs license-external engines.

Signals hosts the platform (governance SoR, critical plane, process group control) and pins the protocol submodule. When attachment work (peer units, lattice-ci, Metaflow/CE) meets a gap, prefer extending signals-protocol (or documenting an explicit interim) over forking per-project wire dialects.

Working stance: treat protocol growth as co-evolution with peer engines — especially core peers, where shared engine practice is densest — requirements materialize in real projects; the contract absorbs what should be shared. Expect substantial further foundation work on signals-protocol itself before “federation complete” is a fair claim.

Signals as hub

Gaius, Ægir, and Atelier carry product engines (multi-service gRPC, serving, co-tenancy, Remediate). Signals holds the scheduler engine on :50551 and the control plane those peers attach to:

What the hub centralizes:

Working mode while early: drive from hub control and protocol gaps; land proven depth in peers that already have engines (and promote stable shapes into signals-protocol). Peer sessions stay product-local; convergence is measured here (lattice-ci, unit graph, protocol bumps). Engine functionality in Signals itself can grow when the federated architecture is solid enough to host it honestly — maturity, not a permanent “thin product” identity.

Doctrine: total commitment as a federation peer

Adopting signals-protocol as a federation peer is total commitment, not a decorative Status endpoint on a half-running stack. That applies whether the peer is a mature product engine (Gaius today) or a younger tree: peerhood means full capacity as that project defines it, honest ops, and remediation — not a lattice-only shell built only to pass accept.

Anti-patterns (not federation):

• Engine-only unit start that leaves product UI/gateway dark while claiming peer ready
• Synthetic always-healthy Status rows so lattice-ci passes without resident capacity
• Unit start that no-ops because an orphan still answers Status
• Unit stop that leaves lattice ports, UIs, or engine process groups alive
• Interactive devenv up stacks fighting the systemd unit for the same lattice port
• Treating “Signals is thin/early” as doctrine that excuses half-committed peers or freezes engine growth here forever

Accept implication: lattice-ci / Engine/Status are necessary gates, not a substitute for full capacity. Peer-unit accept should include clean start/stop/restart of this peer’s engine under signals.target (single listener ownership, honest health). Detail: Peer unit acceptance spec and Peer integration.

Platform capability: Scheduler (zndx.scheduler.v1)

Signals’ first thick engine capability is federated scheduling (capability=scheduler; lab model=yunikorn, later PBS/SLURM). Thin clients call zndx.scheduler.v1.Scheduler on the Signals lattice (:50551); the engine privately talks to the backend and owns archive/current/scratch projection. See YuniKorn queue management (lab adapter).

Spec vs tooling surfaces

Reflection is not a substitute for the proto. First-party scripts should not prefer “raw grpcurl -proto” as a substitute for codegen — they should call generated clients from signals-protocol.

Python engines: grpcio-reflection + enable at server start. Lattice-ci runs both a generated Status client and a reflection check.

Language: brass-tacks naming

Name platform, wire, and foundational theory by the terms a practitioner (or inventor) already uses — not by product metaphor, softened ops slang, or in-house glosses that drift from the definition.

Two tests (fail either → rename until both pass, then document once):

1. Systems: Can someone who only knows the underlying tech (gRPC, systemd, CI) map the phrase to a concrete artifact — port, unit, service registration, gate — without a project glossary?
2. Theory: Would the people who invented (or standardly formalized) the concept recognize our names, types, and invariants as their object — not a vague cousin? Example: persistent homology should read cleanly to Gunnar Carlsson (or any TDA practitioner): Vietoris–Rips filtration, homology dimension, birth/death pairs, essential classes (death = +∞), not rebranded “shape scores.” Same bar for Dempster–Shafer (Bel/Pl/K), SAGE, TreeSHAP, Ollivier–Ricci, and future topology / belief / geometry work.

Product metaphors are fine in UI copy when they map 1:1 to a brass-tacks term documented here. Do not invent a second vocabulary for the same process shape or the same mathematical object.

Layers

  UIs / external engines          Federated engines (OIP + zndx.engine.v1)
  ┌──────────────┐                ┌─────────┬──────────┬─────────┐
  │ Marquez-web  │                │  Aegir  │ Atelier  │  Gaius  │
  │ Metabase*    │                │ :50151  │ :50251   │ :50051  │
  │ Atlas UI     │                └────┬────┴────┬─────┴────┬────┘
  └──────┬───────┘                     │         │          │
         │  HTTP                       │  multi-service gRPC
         │  /api/v1  /api/atlas        │  + OIP ModelInfer (horizon → now)
         ▼                             ▼
  ┌────────────────────────────────────────────────────────────┐
  │  SIGNALS (this product)                                     │
  │  · Atlas :21010  governance SoR + OpenLineage /api/v1       │
  │  · Ranger        tag/resource authz from Atlas tags         │
  │  · Marquez-web   OL UI (proxy only; no Marquez DB)          │
  │  · components/signals-protocol  shared protos + specs       │
  │  · components/hermes-agent      agent runtime (plugins)     │
  │  · PG/AGE + Kudu/FDW scale path                             │
  └────────────────────────────────────────────────────────────┘
         ▲
         │  MemoryProvider + context engine plugins
         │  (Weathership → official Hermes memory service)
  ┌──────┴───────┐
  │ Hermes Agent │  tools / hooks / multi-agent ops
  └──────────────┘

  * AGPL Metabase is a *license-external* peer (isolated engine, not core family):
    process + wire only; consumes catalog/lineage/authz; never SoR or vendored.

Core vs license-external engines

Two kinds of “peer” attach to Signals. Conflating them muddies protocol work and peer-unit sessions.

Metabase is external by requirement: introducing an engine there is inherently isolated so the product can federate without licensing contamination. Architecturally it is distinct from Gaius / Ægir / Atelier and from any future core Signals projects. Operator install and dual-map still matter; they do not make Metabase “another Atelier.”

Hard rules (unchanged):

• No Marquez database and no stock Marquez API as a second catalog.
• No Python OL facade beside Atlas.
• DST method stays in Atelier (and related labs); Signals stores outcomes (classifications, optional belief facets on runs) for governance and audit.
• Protocol evolution is additive within a version; breaking changes → v2.

Submodule

# already pinned
git submodule update --init components/signals-protocol

# bump when protocol trunk advances
git -C components/signals-protocol fetch origin
git -C components/signals-protocol checkout origin/trunk
# commit pin in signals

Discovery (Atlas, Ranger, lineage)

Federated and external engines must find centralized services without hardcoding lab topology. Today Status describes engine endpoints only. The protocol core extends discovery so a peer learns:

Discovery is informative + capability-scoped: advertising a URL does not grant access. Edge (CF Zero Trust / IdP), Kerberos (data plane), and Ranger policies still authorize. See Identity and access.

StatusResponse.surfaces[] (kind, url, healthy) is the v1 step. kind=primary is the product UI the waffle lists. Signals advertises http://$SIGNALS_ADVERTISE_HOST:9889 (host from SIGNALS_KRB_HOST / LAN hostname — never loopback). Do not invent peer UI URLs — list only what that engine’s Status returns.

Engine/ServerQuery is pairwise S2S (not gossip): REMOTES (named git remotes + head), PEERS (lattice host:port from peer-contract), SURFACES (same list as Status), QUEUES (occupancy leaf shape), WORKLOADS (models, capabilities, TP/PP requirements — not the queue). Empty is honest.

A later zndx.discovery.v1 package can grow auth hints if the surface expands.

Engines converge by:

1. Calling discovery (or reading a Signals-published service map).
2. Emitting OpenLineage to LINEAGE_OL (not a local Marquez).
3. Registering / classifying assets via ATLAS_GOVERNANCE (or Signals hooks).
4. Evaluating access via RANGER_AUTHZ (or plugins that pull Atlas tags).

External engines (Metabase and peers)

Metabase (AGPL) and similar BI/agent tools are license-external participants — architecturally isolated from the core engine family (see above). They are first-class on the wire and process group, not second SoRs and not core engine projects:

Signals may ship thin adapters (e.g. Metabase → dataset namespace mapping) under ops docs; the contract remains discovery + OL + Atlas/Ranger.

Optional install (separate AGPL checkout + metabase.service under signals.target, never vendored into this tree): Peer integration — Metabase.

KServe Open Inference Protocol (OIP)

signals-protocol already patterns after OIP and maps Complete/Status as a horizon peer. Multi-agent model ops make that horizon active:

Direction:

1. Map zndx.engine.v1.Complete ↔ OIP ModelInfer (text/tensor payloads) in the protocol spec (expand existing OIP mapping table).
2. Carry authz context on federation/OIP calls (caller principal, optional Atlas entity / classification scope) — additive proto fields; enforcement at the serving edge and Ranger.
3. Emit provenance — engines (or a Signals sidecar) POST RunEvents to /api/v1/lineage for model ops: inputs (prompt hashes / dataset refs), outputs, model id from CompleteResponse.model, capability name.
4. Optional OIP gateway in Signals or Gaius that dual-registers zndx.engine.v1 and OIP for the same capability set.

OIP is not a second governance plane. Atlas/Ranger remain law; OIP is the model invoke lingua franca; OL is the audit thread.

Relationship to Marquez

See OpenLineage + Atlas.

Phased delivery (protocol core)

Hermes / Weathership memory

Hermes discovers memory providers under plugins/memory/<name>/ and context engines under plugins/context_engine/<name>/ (plugin system). Only one external memory provider is active at a time (memory.provider in config), additive to built-in MEMORY.md / USER.md (memory providers).

Weathership (Signals-backed) should:

1. Implement MemoryProvider with prefetch, turn sync, session extract, and agent tools (search/store) — backed by Signals AGE/OL (and optional belief facets), not a greenfield DB.
2. Implement a context engine that compacts with lineage-aware retention (pre-compression extract into Weathership memory, similar spirit to ByteRover’s pre-compression extraction).
3. Use federation discovery for Atlas/Ranger/OL bases; emit OpenLineage for memory writes and model ops (authz + provenance).
4. Ship as installable Hermes plugins; eventually qualify as an official reasoning-enabled memory service provider in the Hermes ecosystem.
5. Keep superset capabilities fleet-side (and exposable as tools/skills): mechanistic interpretability (SAE/CLT), topological structure (PH, Ollivier–Ricci), multi-engine Complete/Remediate—without waiting for Hermes core to absorb them. Deeper core integration remains open later; plugins-first is the current strategy.

Submodule: components/hermes-agent.

Related

• Submodule: components/signals-protocol
• Submodule: components/hermes-agent
• OpenLineage + Atlas
• Identity and access
• Atlas → Kudu outbox
• Protocol README (federation face, OIP horizon, co-tenancy)
• Gaius: CO-TENANCY draft; hx.lineage; FEDERATION.md
• Aegir: polyglot OpenLineage; former gateway/marquez.py (shape reference only)
• Atelier: DST / classification method lab

Signals engine

The Signals engine is the scheduler-capability process on :50551. It implements zndx.engine.v1.Engine and zndx.scheduler.v1.Scheduler from signals-protocol. Product UIs and peer engines call this process; they do not speak YuniKorn REST or Airflow HTTP as a product path.

Responsibilities

• Engine/Status, ServerQuery, Announce, WatchWorkload, RecordLineage, Yield
• Queue policy, projection, and RequestQueueShare
• Declare / Renew / Release / List / WatchActivity — leases that Airflow sensors hold
• Discovery: Status.surfaces and pairwise PEERS

Where it runs

Health is gRPC Engine/Status plus server reflection. just lattice-ci is the accept probe.

grpcurl -plaintext 127.0.0.1:50551 zndx.engine.v1.Engine/Status
just lattice-ci

Primary UI: signals-ui :9889. Peer engines (Gaius :50051, Ægir :50151, Atelier :50251, Hermes :50651, Metabase :50451) register the same Engine face beside their native service.

YuniKorn queue management (engine-first)

Signals owns queue configuration and operational control as a platform capability. Thin clients never call YuniKorn REST as the product path.

Architecture

signals-ui · MCP · CLI
        │  gRPC only
        ▼
Signals engine :50551
  zndx.engine.v1.Engine          (Status — capability=scheduler, model=yunikorn)
  zndx.scheduler.v1.Scheduler    (ops + projection; vendor-agnostic)
        │ private HTTP / kube
        ▼
YK REST :30080  ·  ConfigMap apply (promote)   ← lab backend

Protocol: components/signals-protocol → zndx.scheduler.v1. YuniKorn is the lab backend, not the service name.

Official YK references:

• Queue config
• Scheduler REST

Projection roots (Aegir-shaped)

Under build/dev/ (or SIGNALS_YK_PROJECTION_ROOT):

API-symmetric clients

Queues lineup (Aegir geometry)

Full-height shell under chrome:

Legacy /nodes and ?panel=nodes redirect to /queues?open=ops/nodes.

export SIGNALS_ENGINE_TARGET=127.0.0.1:50551
export KUBECONFIG=~/.kube/config   # for promote ConfigMap apply

uv run python -m signals.engine          # server
uv run python -m signals.cli.yk health
uv run python -m signals.cli.yk queues
uv run python -m signals.cli.yk config
uv run python -m signals.cli.yk sync

# policy lifecycle
uv run python -m signals.cli.yk write-scratch path/to/queues.yaml
uv run python -m signals.cli.yk diff --live
uv run python -m signals.cli.yk promote --dry-run
uv run python -m signals.cli.yk promote
uv run python -m signals.cli.yk archives
uv run python -m signals.cli.yk index --root current

MCP (engine gRPC only; tools mirror CLI):

export SIGNALS_ENGINE_TARGET=127.0.0.1:50551
uv run python -m signals.mcp.yk
# tools: yk_health, yk_queues, yk_config, yk_validate, yk_sync,
#        yk_write_scratch, yk_diff, yk_promote, yk_archives, yk_restore, yk_index

Promote

PromoteScratch order:

1. Read scratch queues.yaml
2. Validate via YK POST /ws/v1/validate-conf (engine-private)
3. dry_run: optional kubectl apply --dry-run=server of ConfigMap patch; no projection write
4. live: apply ConfigMap → archive current → write scratch → current → best-effort SyncProjection

ConfigMap adapter (engine-private)

Lab/RKE2: yunikorn-configs overrides yunikorn-defaults at runtime (zarf values). Apply preserves other CM keys (e.g. admission controller flags).

On apply failure, projection is not mutated (applied=false).

Resource-class tree

Queue path is scarcity, not project. Project stays identity (federation.project, C2, Engine/Yield). Aegir / Atelier / Gaius / Signals share the same internal.inference.* leaves.

Canonical policy: config/scheduler/federation-queues.yaml. Catalog prose: config/scheduler/resource-classes.md.

root.default                         leftover / unannotated (no GPU)
root.platform                        Metaflow, Airflow, Eventing
root.internal.compute                CPU burst / idle sentinels
root.internal.inference.{reasoning,coding,orchestration,instruct,embedding,extract,agent-rtc}
root.external.token-metered          pay-per-token (Cerebras thinking, OpenAI-class)
root.external.rate-metered           peer meters RPM
root.external.subscription.rate-limited   Grok ACP / Bytez / xAI subscription only

Placement (without provided, annotations are ignored and everything lands on root.default):

placementrules:
  - name: provided          # yunikorn.apache.org/queue
    create: false
  - name: fixed
    value: root.default
    create: false

Queue share (WRK occupancy). Peers send Scheduler/RequestQueueShare (uuidv7 request_id, workloads[], shares[] guaranteed/max, validity window). Signals persists the time series, merges overlapping leftover floors against parent GPU max 6 (extract=1 vs light/CLT or medium/SAE), and PromoteScratch applies yunikorn-configs. Peers never write queues.yaml. Persist failure is #YK.00000007.SHAREFAIL.

Idle occupancy. Extended inactivity on a model (e.g. Qwen3.8) is Knative KPA scale-to-zero of the proxy sentinel Application, then MiNiFi C2 last-gasp → Engine/Yield on the federated engine that owns that vLLM. Tokens (federation.zndx.org/gpu) free only when the Application completes. Knative does not stop host CUDA. Serving / loading keeps min-scale: 1. See MiNiFi sentinels.

GPU occupancy is the Application claim key federation.zndx.org/gpu. Advertise node capacity with scripts/advertise_federation_gpu.sh. Host-engine sentinels request that token only — never nvidia.com/gpu on a CPU-only pod (that binds the card into the empty container). In-cluster CUDA pods request both.

Each leaf carries properties.federation.* (class, yk_enforces, peer_meters, examples) so GetQueueTree is the discovery SoR. MCP catalog tools come after this tree is live; protocol SuggestQueue later still.

An Application is placed once, when its app-id is first submitted. Re-annotating a later pod with the same id does not move it; use a new id or let the old Application complete. root.default holds unannotated leftovers (including earlier autogen).

Turnkey refresh: just redeploy walks the k8s.product-redeploy FSM (IT-ops FSM). Probes are Brier-scored forecasts. Completing still owns the app-id; timeout fails the procedure. YK and Knative substrate apply only when missing or those manifests changed. Not signals.target (host engines). Not Metabase (host peer). Not a scorched-earth rebuild.

uv run python -m signals.cli.yk write-scratch config/scheduler/federation-queues.yaml
uv run python -m signals.cli.yk promote --dry-run
uv run python -m signals.cli.yk promote

Sentinel Applications and Yield

A MiNiFi C++ sentinel is the YuniKorn Application: 1:1 with a schedulable process. signals-ui Applications lists only what YK admitted.

Closed loop

YK ends Application  →  last-gasp / heartbeat (C2 HTTP)
                     →  Signals C2
                     →  gRPC zndx.engine.v1.Engine/Yield
                     →  owning engine ends the local process

Two wires on the C2 process: HTTP toward sentinels; gRPC Yield toward the lattice engine from peer-contract.json. Sentinels never dial gRPC.

Unknown workload_id is idempotent (ok=true, process_ended=false).

Resource claim

The Application’s resource vector is the claim. GPU work asks for federation.zndx.org/gpu on the sentinel (or on an in-cluster pod). That is how YK sees occupancy and how preemption frees a card. CUDA still runs on the host engine unless the work is an in-cluster GPU pod (nvidia.com/gpu and the federation token). Do not request nvidia.com/gpu on a CPU-only MiNiFi container.

Queue path is the resource class (root.internal.inference.extract, root.internal.compute, …), not root.{project}. Project stays on federation.project for C2 and Yield.

Platform warehouse upkeep (data-product.tier-upkeep) uses a flow-level proxy sentinel on root.platform (signals-dataproduct-tier-upkeep) when the Metaflow run is not itself a YK Application. A @kubernetes step is an Application — same app-id / workload_id.

What this cut is not

• NiFi proper / canvas (optional later K8s fielding for OTel wiring; not signals-ui)
• Per-engine C2 servers
• Gaius/Aegir/Atelier Yield bodies (they implement the same RPC; Gaius maps onto CompleteWorkload / restore)
• Moving host vLLM into the sentinel container

Unit

signals-c2.service is foundation (with the engine): signals.target Wants it; it After= signals-engine.service. Start/stop wrappers: scripts/systemd_c2_{start,stop}.sh.

sudo systemctl start signals-c2.service   # or: start signals.target
curl -sf http://127.0.0.1:50561/healthz

just signals-ready records C2 as WARN until the unit is up (C2 starts after ready).

Lab

# engine on :50551, control :50552, C2 :50561
just sentinel-yield-lab

Protocol: components/signals-protocol Engine/Yield. Ops design: specification/operations/minifi_sentinels.md.

IT-ops FSM and Brier ledger

IT operations are methods: a named FSM, not a pile of kubectl lines. Each probe is a forecast (proposition + implied p) tagged with the FSM state it was made from. When the oracle (kubectl / YuniKorn GET) resolves the claim, we score Brier per (observer, method fsm_state, implicit object FSM + state, procedure, epoch). The method cell is where we were; the implicit cell is the K8s/YK object machine (yk.application=running vs completing, k8s.deploy=desired_positive, k8s.job=active). Mixing those hid the Helm-Job miss (method said apps_completing, object was still Running).

This is the same discipline as synth’s Deluge calibration ledger and cyberphy converge (fail loud, do not treat a holding state as done). Signals holds the K8s-first catalog so ACP agents can run a method with a risk number, not a vibe.

Law

1. Holding is not terminal. YK Completing still owns the app-id. Deployment spec.replicas still 1 still owns the ReplicaSet.
2. Ill-posed claims resolve false. Asserting yk-apps-completed from start or scaling is a demerit even if a GET happens to look empty.
3. Uncalibrated is 0.5 risk, not 0. No track record → honest ignorance.
4. Timeout fails the procedure. It does not WARN-and-apply.

First method: k8s.product-redeploy

just redeploy / uv run python -m signals.ops redeploy

start → substrate_ready → scaling → desired_zero
      → pods_draining? → pods_quiet
      → apps_completing? → apps_completed
      → applying → verifying → placed
                 ↘ failed (from any non-terminal)

Holding: scaling, desired_zero, pods_draining, apps_completing.

data-product.tier-upkeep

Weekly ADD next 168-hour Kudu range, then settle weeks ≥ 4 weeks old (copy → verify → DROP RANGE PARTITION). Holding through dropping. Do not DROP from verifying.

Metaflow flow: signals.flows.tier_upkeep.DataProductTierUpkeep. Airflow is the clock. YK: flow-level proxy sentinel signals-dataproduct-tier-upkeep on root.platform (config/platform/tier-upkeep-sentinel.yaml). A @kubernetes step is an Application — reuse that app-id.

just tier-upkeep
uv run python -m signals.ops method data-product.tier-upkeep

Agent / ACP surface

No extra flags. The method is the command.

An ACP agent should: list methods, load a method, refuse transitions the graph does not allow, consult risk before acting, and never treat a holding state as placed.

Ledger file: build/state/ops-observations.jsonl (gitignored).

Nautilus is a different machine. It stamps positions and scores Brier per (observer, call site, momentum bucket) inside spec_version + engine_build. That diagram is Nautilus FSM and the Brier ledger.

Lineage

Platform Metaflow (YK · Knative · Airflow)

Status: Architecture direction (platform-grade)
Date: 2026-08-11
Scope: Signals-owned ground-up Metaflow capability — not Gaius, not Marquez. Any federated engine that integrates via signals-protocol can consume it as a platform service.
Submodule: components/metaflow → weathership/oss-metaflow branch rch/devenv (custom line, same convention as other ASF/oss components).
References:

• Outerbounds AWS K8s deployment (eks_airflow Terraform path preferred for us)
• Metaflow Airflow scheduling · Outerbounds Airflow ops
• Outerbounds event triggering (stock stack uses Argo Events; we substitute Knative Eventing)

Thesis

Signals provides a platform Metaflow so current and future engines (Gaius, Aegir, Atelier, …) do not each stand up thrashing metadata services on shared RKE2. Engine repos stay free to keep isolated Metaflow for solo devenv; this tree does not modify Gaius or Marquez. Platform work lands here: submodule patches, federation packaging, Airflow + Knative Eventing, YuniKorn admission, signals-ui visibility.

With YuniKorn + Knative Serving already locked as the federation control plane, Metaflow is a first-class platform service:

Stock Outerbounds prefers Argo Workflows when in doubt (full event triggering). We deliberately take the eks_airflow shape and close the event gap with Knative Eventing, not Argo Events / Argo Workflows.

Submodule and customization line

Platform-specific changes to Metaflow itself (Airflow event-bridge hooks, Knative-friendly defaults, YK labels, devenv packaging) go on rch/devenv in that repo — not as one-off patches only in Signals, and not by forking work into Gaius.

Isolation elsewhere (out of scope here)

Other products may still run engine-local Metaflow (Tilt metadata, private PG/S3) for isolated devenv. That pattern is fine for those repos; Signals does not implement or maintain it. Platform Metaflow is the shared SoR when an engine opts into federation via signals-protocol.

Why platform upgrade

Official stack vs Signals adaptation

Outerbounds eks_argo (reference default)
  Metaflow service + S3 + EKS
  Argo Workflows  ←── steps
  Argo Events     ←── event triggering

Outerbounds eks_airflow (our base)
  Metaflow service + S3 + EKS
  Airflow         ←── steps as KubernetesPodOperator DAGs
  @schedule only  ←── stock Metaflow→Airflow (no native event trigger)

Signals platform (target)
  Metaflow service + RustFS + RKE2
  Airflow         ←── production DAGs (schedule + API-triggered runs)
  Knative Eventing←── event fabric (replace Argo Events role)
  YuniKorn        ←── schedule all Metaflow + sentinel pods
  Knative Serving ←── sentinels / scale-to-zero (already federation)

Critical product fact: Metaflow’s first-class event triggering is documented as Argo Workflows only today. Airflow docs state event-based triggering is not supported via Metaflow’s Airflow integration (only @schedule / cron). Gaius already showed that application-level and platform-level triggering is enough for real product capability. We institutionalize that as Knative Eventing → Airflow, rather than forking Metaflow’s Argo-only path.

Target architecture

                    ┌─────────────────────────────────────┐
                    │  Producers                          │
                    │  CLI · MCP · Atlas OL · Sentinels · │
                    │  Engine gRPC · webhooks · cron      │
                    └──────────────┬──────────────────────┘
                                   │ CloudEvents
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  Knative Eventing                   │
                    │  Broker / Channel / Triggers        │
                    │  (types: metaflow.run.v1, …)        │
                    └──────────────┬──────────────────────┘
                                   │ Sink: Airflow REST / API
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  Apache Airflow (on RKE2, YK queue) │
                    │  DAGs from: python flow.py          │
                    │             airflow create dag.py   │
                    │  @schedule + event-triggered runs    │
                    └──────────────┬──────────────────────┘
                                   │ KubernetesPodOperator
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  RKE2 + YuniKorn                    │
                    │  Metaflow @kubernetes step pods     │
                    │  Queues root.{engine,signals,…}     │
                    └──────────────┬──────────────────────┘
                          ┌────────┴────────┐
                          ▼                 ▼
              ┌───────────────────┐  ┌──────────────────┐
              │ Metaflow metadata │  │ RustFS (S3)      │
              │ service + PG      │  │ artifacts/cards  │
              └───────────────────┘  └──────────────────┘
                          │
                          ▼
              ┌───────────────────┐
              │ signals-ui        │
              │ Applications ·    │
              │ Queues · Status   │
              └───────────────────┘

Component notes

Artifact store — RustFS only

Metaflow’s product property is that every run retains an immutable snapshot of code, data, and dependencies. That snapshot is the first Signals data product (signals.metaflow.snapshots). Mapping and ACP brief: Data Products History.

signals.ops.metaflow_store.require_rustfs fails closed if any of those drift. Do not point SYSROOT at /tmp, ~/.metaflow, or a foreign S3.

Metaflow metadata service

• Deploy as platform Deployment/Service (not engine Tilt thrash).
• Postgres: platform instance (signals or dedicated metaflow DB on lab PG 5455 / production SoR) — not a second unmanaged store per engine.
• Expose: ClusterIP + NodePort or signals-ui/proxy for lab (METAFLOW_SERVICE_URL).
• Config profile (shared ~/.metaflowconfig shape):

{
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_SERVICE_URL": "http://metaflow-service.platform.svc:8080",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DATASTORE_SYSROOT_S3": "s3://metaflow/metaflow",
  "METAFLOW_S3_ENDPOINT_URL": "http://rustfs.platform.svc:9010",
  "METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
  "METAFLOW_AIRFLOW_KUBERNETES_CONN_ID": "kubernetes_default"
}

Airflow

• Run on RKE2 under YK (queue root.platform for control-plane services; inference steps use root.internal.inference.*, not a project parent).
• Lab/platform pin: Airflow 3.1.7 (components/airflow chart); cncf.kubernetes provider for KPO.
• Metaflow steps → KubernetesPodOperator only (@kubernetes, not @batch).
• Deploy path: python flow.py --with retry airflow create flow_dag.py then ship DAG into Airflow dags/ (Git-sync or ConfigMap pipeline).
• Time schedules: Metaflow @schedule as today.
• Event runs: Airflow REST POST /api/v1/dags/{dag_id}/dagRuns (or stable equivalent) invoked by Knative SinkBinding / Service — this is the platform “bring Airflow up to event-triggering spec.”

Knative Eventing (not Argo Events)

Example event types (illustrative):

• dev.metaflow.flow.finished — inter-flow chaining (@trigger_on_finish analogue)
• dev.signals.sentinel.activated — sentinel woke; start inspection flow
• dev.gaius.article.curate.requested — product CLI/MCP publish
• dev.atlas.openlineage.runEvent — optional OL-driven reactions

Emitter libraries: small platform helper (replace ArgoEvent.publish) that POSTs CloudEvents to the Broker ingress. Gaius/CLI keep product APIs; they publish platform events instead of talking Argo.

YuniKorn

• All Metaflow task pods and Airflow workers labeled into federation queues.
• signals-ui Queues flow already shows root.* capacity; Applications shows admitted apps (including Metaflow steps once scheduled).
• Avoid default-scheduler side channels for production steps.

Sentinels

• Remain Knative Serving (scale-to-zero).
• Eventing can wake work and start Metaflow DAGs when a sentinel signals readiness or C2 heartbeat thresholds — without conflating Serving and Workflow engines.

Isolation vs platform (policy)

Engines opt into platform by setting Metaflow config profile to platform URLs and deploying DAGs to platform Airflow. Isolation remains for unit hermeticity.

Delivery phases

M0 — Document + guardrails (this page)

• Architecture + non-goals; signals-ui Applications already frames YK workloads.

M1 — Platform Metaflow service (landed)

Host PG (:5455) and RustFS (:9010) are bridged into ns/metaflow via Service + Endpoints (signals-postgres, signals-rustfs).

M2 — Airflow 3 on RKE2 (YK) (landed)

Shape: LocalExecutor, external PG (airflow DB on host :5455 via hostNetwork socat proxy — devenv PG is loopback-only), no in-cluster Bitnami Postgres/Redis. Fernet/JWT/API secrets persist under build/config/airflow-secrets.env. CI DAG signals_ci is live; Metaflow airflow create DAGs ship next (same ConfigMap / dags path). REST: AF3 /api/v2 with JWT from POST /auth/token.

M3 — Knative Eventing bridge (landed)

No Argo. Engines (Gaius/Aegir/…) and Metaflow finish hooks publish CloudEvents to the Broker; Triggers filter by type and invoke Airflow. Zarf agent ignore until Eventing images are packaged into signals-federation.

M4 — Multi-engine consumers

• Gaius/Aegir/Atelier profiles point at platform Metaflow
• signals-ui Applications: show Metaflow run correlation (run id ↔ YK app id)
• OL: keep Atlas SoR; Metaflow cards remain UX, not second lineage SoR

Critical plane membership

Metaflow (with YK, Knative, Airflow, RustFS, and host data services) is part of the uniform critical plane for Signals — enforced by just stack-ready / just signals-ready / signals:stack-ready. See Critical plane.

Non-goals

• Touching Gaius, Marquez, or engine repos from this workstream — engines adopt platform Metaflow on their schedule via config + protocol
• Argo Workflows / Argo Events as SoR — rejected for this platform path
• Nested Metaflow foreaches on Airflow (Airflow limitation) — avoid in platform DAGs or gate on Airflow version + design
• Conditional/recursive steps (Metaflow 2.18+) until Airflow support exists
• Marquez DB as Metaflow metadata (Atlas OL remains lineage SoR)

Open design questions

1. Airflow executor: start LocalExecutor / CeleryExecutor vs full KubernetesExecutor (DAG volume sync cost on single-node lab).
2. Event exactly-once: Broker + Airflow dagRun dag_run_id idempotency keys from CloudEvent id.
3. Namespace layout: one metaflow ns vs per-engine namespaces for pods with YK queue labels only.
4. UI: Metaflow UI (Outerbounds) optional lab validation vs signals-ui only.

Related

• signals-federation Zarf — YK + Knative Serving
• Airflow component stub
• signals-ui Applications — workloads surface
• Gaius Metaflow service (isolation reference)

Signals critical plane

Status: Policy (enforced via preflight)
Date: 2026-08-11

A Signals deployment is not “Postgres + a few daemons.” devenv up / devenv up -d is the uniform entry point for a critical plane that spans host processes and RKE2 platform services. Engines integrate via signals-protocol; they do not each re-host thrashing control-plane copies of these services.

Critical plane (uniform)

These are uniformly critical: missing any of them is a degraded Signals deployment, not a supported “minimal profile” for product work. Hermetic unit tests may still mock subsystems; the live stack does not.

Not critical (by design)

Enforcement

devenv up -d                        # preferred — full validate + bootstrap
just stack-ready                    # preflight (may bootstrap missing pieces)
just signals-ready                  # check-only oneshot for peers / systemd
just data-plane-ci                  # Kudu/Impala ports after up (elevated CI gate)

Naming: ready vs CI vs smoke

Do not elevate “smoke” into system recipe / package / CloudEvent API names — that role is CI.

Topology

devenv up [-d]
  ├── host critical: KDC, PG, RustFS, Atlas, Ranger, Kudu, Impala, signals-ui, …
  └── RKE2 critical: YuniKorn, Knative, Metaflow service, Airflow 3
            │
            ▼
     Federated engines (signals-protocol) — optional consumers

Related

• Platform Metaflow
• Peer integration — systemd group + external engines
• signals-federation Zarf
• DCGM, the warehouse, and agent exchange — cluster flow; the live yield is not the store
• Development environment
• Sample units: infra/systemd/ · contract: config/platform/peer-contract.json

Query Engine & Catalog Stack

Transparent hierarchical storage in Percy’s sense: Kudu for hot mutable rows, Impala for SQL, cooler data as Iceberg files on RustFS. PostgreSQL is the application front door; impala_fdw carries one Kerberos principal through to Impala and Kudu.

Table metadata lives in a PostgreSQL catalog registry (KuduMetaProvider, SignalsDdlExecutor) and is loaded from the Kudu master. Impala talks to Kudu (STORED AS KUDU) and to Iceberg via the Polaris REST catalog. Applications use Postgres foreign tables: kudu_scan for closed hot-tier shapes, impala_sql for Iceberg and UNION ALL views of hot ∪ warm. The cluster path from DCGM into that union, and the agent-exchange join back to those samples, is the appendix diagram.

Stack

HMS-Free Mode

When -Dsignals.hms_free_mode=true is set, the catalog server bypasses HMS entirely:

Iceberg DDL/DML through Impala is PENDING — in-fork, gradual, with purpose. The Phase-2 IcebergRESTCatalog.createTable() path exists but its created metadata cannot yet be loaded back by the MultiMetaProvider (2026-08-30 phantom-table incident, #SL.00000027.SCHEMA2), and upstream’s REST-catalog support marks tables ACCESSTYPE_READ (“read-only … not supported yet”, IMPALA-13586) so DML is gated at analysis. Doctrine: this fork will implement the Transparent Hierarchical Storage write path over Kudu and Iceberg through our own efforts. Until then the bridges are signals.ops.iceberg_register (PyIceberg → Polaris) for tier1 DDL and out-of-band settle (HDF5 + IcebergHdf5Register) for tier1 writes — bridges, not the architecture.

Modified Components

Eight Java files implement HMS-free mode in the Impala fork (rch/asf-impala on the rch/devenv line):

Catalog Registry Schema

The PostgreSQL signals_catalog database stores table registrations:

-- config/impala/catalog_schema.sql
CREATE TABLE catalog_databases (
    db_name TEXT PRIMARY KEY,
    comment TEXT,
    location TEXT,
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE catalog_tables (
    db_name TEXT NOT NULL,
    table_name TEXT NOT NULL,
    kudu_table_name TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (db_name, table_name)
);

Current Status

Working (validated end-to-end):

• CREATE/DROP DATABASE
• CREATE/DROP TABLE (Kudu, STORED AS KUDU)
• INSERT INTO (single and multi-row)
• SELECT with WHERE, ORDER BY, JOINs, GROUP BY, HAVING, subqueries
• Multiple data types: INT, STRING, DOUBLE, BOOLEAN, BIGINT

Near-term:

• Iceberg tables via Polaris REST catalog
• Atlas integration for metadata visibility and tagging

Running the Stack

All services start via devenv or manual scripts:

Statestore  → port 24000  (pure C++, no JVM)
Catalogd    → port 26000  (JVM, HMS-free mode)
Impalad     → port 21050  (JVM, HS2 protocol)
Kudu Master → port 7051
Kudu TServer→ port 7050
PostgreSQL  → port 5455   (signals_catalog DB)

Connect via JDBC:

jdbc:hive2://localhost:21050/default;auth=noSasl

Data Products History

Top-line History is the federation inventory of autonomous data products, viewed as agents understand them. OpenLineage remains the provenance facet (Atlas /api/v1); it is not the nav label.

Sole SoR: Iceberg on RustFS

Logical names stay details / tx / hx. Physical storage is tiered:

Writers insert tier0. Time grain is UTC hour; each Kudu range is one week (168 hours). After 4 weeks, copy that week to tier1, verify, then DROP RANGE PARTITION — never DELETE rows (Kudu schema design). Hour is the range unit, not a second hash/range level. Iceberg is not hour-partitioned (file explosion); details_tier1 / tx / hx use product.

Postgres :5455 (devenv PostgreSQL 16 + AGE) is administrative only (Atlas graph, Ranger). It is not pglite. No product copies there. impala_fdw stays Kudu-only and is not the warehouse reader.

Schema:

• data-products-kudu.sql
• data-products-iceberg.sql
• data-products-views.sql

Aspects are SHACL Core NodeShapes, not products. Instances claim a spec (signals.spec.warehouse_product) and carry aspect.<id> bindings. The shapes graph is signals.aspects.catalog (a product that describes aspects). Wire: ServerQuery kind=ASPECTS. Contract: data_products.md.

details is a fact log (e, a, v, t, op) plus epoch_hour (range unit). t / tx_id is RFC 9562 UUIDv7 (time-ordered). Non-v7 ids are refused and surfaced to the source via Engine/Remediate (TX_ID_NOT_UUIDV7). Current inventory = latest assert per (e, a) over the view.

erDiagram
    PRODUCT ||--o{ TX : "emits"
    PRODUCT ||--o{ DETAILS : "described by"
    TX ||--o{ DETAILS : "asserts at t"
    TX ||--o{ HX_EXCHANGE : "keyed by tx_id"
    TX ||--o{ HX_REASONING : "keyed by tx_id"

    PRODUCT {
        string product_id PK "logical — no Iceberg table"
    }
    TX {
        string tx_id PK
        string product_id FK
        long ts_ns
        string kind
        string summary
        string source
        string ce_type
    }
    DETAILS {
        int epoch_hour "Kudu RANGE unit; week-wide tablets"
        string e FK "product_id"
        string a "peer title kind leaf agent_focus spec aspect.* + snapshot facts"
        string v
        string t FK "tx_id"
        boolean op
    }
    HX_EXCHANGE {
        string tx_id FK
        string product_id FK
        string agent
        string actor
        string message
    }
    HX_REASONING {
        string tx_id FK
        string product_id FK
        string quality
        string lineage
        string delta
        string trace
    }

config/platform/data-products.json is a bootstrap seed for a first tier0 tx + details assert, then retired as SoR. Local JSONL under build/state/ is lab scratch only — never a copy of the warehouse.

Object IO is RustFS (/raid/signals/rustfs, S3 :9010). Impala core-site.xml uses s3a://signals-dataproducts/ — not /tmp.

First product: Metaflow run snapshots

signals.metaflow.snapshots is the first Signals-owned data product. Metaflow already retains an immutable code + data + deps triple per run; we do not re-snapshot. We map that triple into details and refuse any datastore that is not RustFS.

Each run is a tx (kind=snapshot, source=metaflow). Latest-wins on latest_* shows the current run; run-qualified keys run.{flow}/{run_id}.* keep every retained snapshot in the projection so ACP can see the net set, not only the last write.

Fail closed: METAFLOW_DEFAULT_DATASTORE=local (or unset, or a non-RustFS endpoint / bucket) is a warehouse error. Profile: config/metaflow/platform.json (s3://metaflow/metaflow @ RustFS :9010).

just record-snapshot DataProductTierUpkeep 42 \
  --code-package s3://metaflow/metaflow/DataProductTierUpkeep/data/abc \
  --code-sha abc
uv run python -m signals.ops review-product signals.metaflow.snapshots

The product includes the ACP assessment, not only the object URIs. An observer (acp-observer) walks data-product.history-review and writes quality / lineage / delta on hx_reasoning plus details facts assessment, upkeep_nominal, upkeep_fsm. The job is narrow: observe data-product.tier-upkeep and confirm it is proceeding nominally (legal FSM — holding is neither failed nor done — ADD next week, settle only weeks ≥ 4 weeks old via DROP RANGE PARTITION, snapshot on RustFS). Off-nominal is recorded, then fail-closed.

DataProductTierUpkeep.end passes the walk doc into that assessment.

Nascent products (seed)

Events → agent

CloudEvent type: dev.signals.dataproduct.updated.

On create / update / maintain, walk method data-product.history-review:

1. event_received → reviewing → understood
2. Activate ACP: Grok-Subscription (root.external.subscription.rate-limited) or Grok-Local with Qwen3.8 capability=[thinking,vision]
3. Brief must cover quality, lineage, and delta (and nominal for signals.metaflow.snapshots)
4. Persist a tx + details asserts; persist agent work on hx_* keyed to tx_id

uv run python -m signals.ops review-product gaius.cognition.outputs \
  --kind updated --summary 'article-curate run 18'

review-product writes tier0 (Kudu). JSON catalog is seed only. ACP spawn is the next hop; hx is how History stays agent-facing.

Partition upkeep is method data-product.tier-upkeep (Metaflow flow, Airflow schedule). It surfaces on YK root.platform as proxy sentinel signals-dataproduct-tier-upkeep, or as a real Application if a step is a K8s pod.

just rebuild              # Polarisfork assemble + restart + wait :8182
just polaris-install      # assemble only

UI: /history (nav History). /lineage redirects here. The page is a reader of the warehouse, not a catalog editor.

Peers that maintain their own product on this warehouse: Peer data products. Device samples for a tx join these hx_* rows at read time: DCGM, the warehouse, and agent exchange.

Data Flow

Data flow through the system connects user interaction, the gRPC engine, compute infrastructure, and data sources.

Instruction Path

Data Sources

Streaming Ingest

High-velocity data (click streams, telemetry) arrives via NiFi and is routed to appropriate storage layers. The engine can direct the agent to observe stream accumulation and discover ontology-grounded feature patterns.

Analytical Storage

Object Storage

S3 buckets store raw data, intermediate results, and model artifacts. In air-gap deployments, an S3-compatible endpoint (RustFS or similar) replaces AWS S3.

Response Path

Computed visualizations flow back through the engine to the web client:

1. Dask workers complete computation
2. Datashader rasterizes the result
3. HoloViews composes the view specification
4. Engine streams the update over gRPC
5. Web client renders the visualization in the top panel

OpenLineage + Atlas

Governance and runtime lineage live in one Atlas process: Apache AGE on PostgreSQL 16. Existing Atlas clients keep /api/atlas/*. An OpenLineage extension on the same host serves the Marquez REST surface. Heavy analytics scale through impala_fdw onto Kudu projections of those same entities.

What lives where

components/marquez is the UI and the contract reference. Producers (Airflow, Metaflow, Flink, Gaius, sigint) post RunEvents to Signals.

Completeness

Atlas implements the Marquez OpenLineage API so that:

1. Marquez-web renders jobs, datasets, events, search, stats, tags, and lineage graphs.
2. Marquez API contract tests can point at Atlas.
3. OpenLineage clients talk to one host.

Empty collections return 200. Every path Marquez-web calls returns Marquez-shaped JSON.

Deployment model (one SoR process; UI on adjacent port)

Producers (Flink OL, Airflow, polyglot, sigint, …)
        │
        ▼
 Atlas :21010  (single SoR process)
   ├─ /api/atlas/*     # governance — existing clients, zero changes
   ├─ /api/v1/*        # complete Marquez-compat OL API + ingest
   └─ /api/v2beta/*    # search jobs/datasets (OpenSearch-shaped; AGE-backed)
        │
        ▼
 signals Postgres :5455
   ├─ Atlas AGE graph (entities, classifications, OL Job/Run/Dataset)
   ├─ lineage_events / lineage_tags / lineage_namespaces / lineage_sources
   └─ optional Kudu projections (FDW scale path)

 marquez-web :21011  ── default stack; port = Atlas HTTP + 1; proxies /api/v1 + /api/v2beta → Atlas

Marquez-web forwards /api/v1 and /api/v2beta to Atlas. It is part of every just up. Port rule: MARQUEZ_WEB_PORT = SIGNALS_ATLAS_HTTP_PORT + 1 (defaults 21010 / 21011).

Aegir gateway/marquez.py and Gaius hx.lineage inform graph shape and contract tests.

Atlas client compatibility

Marquez API surface on Atlas

Implemented in components/atlas (OpenLineageServlet + OpenLineageStore):

Store: AGE graph signals_ol (Job / Run / Dataset) plus relational lineage_events, lineage_namespaces, lineage_tags, lineage_entity_tags, lineage_sources on the same Postgres database as Atlas.

Marquez in devenv

There is no marquez:db-init, no marquez PG database, no processes.marquez-api, no Python OL side service.

Validation loop

# After atlas:build + restart atlas process:
curl -sS http://127.0.0.1:21010/api/v1/health
curl -sS http://127.0.0.1:21010/api/v1/tags
curl -sS http://127.0.0.1:21010/api/v1/jobs
curl -sS 'http://127.0.0.1:21010/api/v1/events/lineage?limit=10'
curl -sS 'http://127.0.0.1:21010/api/v1/stats/lineage-events?period=DAY'
# UI: http://<node>:21011  (proxies to Atlas)

Seed via POST /api/v1/lineage with a standard OpenLineage RunEvent; jobs, datasets, events, and lineage should appear in Marquez-web without a Marquez DB.

Producer validation: Flink

Upstream Apache Flink is a first-class OpenLineage integration; Cloudera’s stack already proves Flink → Atlas. Dual-path lab check:

BDD @tier-1 (planned): features/platform/lineage_flink_ol.feature.

Phased delivery

M2 lives in components/atlas (OpenLineageServlet on /api/v1/* and /api/v2beta/*), not in Python. Broader federation (discovery of Atlas/Ranger, OIP, external peers such as Metabase) is documented under Signals protocol core.

Related

• Signals protocol core — submodule, discovery, OIP
• Marquez component — UI-only deploy, port rule
• Scratch: docs/scratch/2026-08-08/145328_marquez-composite-schema-direction.md
• Scratch: docs/scratch/2026-08-08/151502_no-marquez-db-composite-sor.md
• Aegir: gateway/marquez.py (contract/shape reference only — not deployed here)
• Gaius: gaius.hx.lineage (event + AGE labels reference)
• FDW: components/impala_fdw (scale path for projection tables)

Atlas → Kudu projection outbox contract

Status: Binding for sync implementers (freeze 2026-08-07).
SoR: Postgres + AGE (atlas_graph).
Derived store: Impala/Kudu tables in database atlas (see config/atlas/kudu_projections.sql).

Goals

• Idempotent, at-least-once materialization of typed projections.
• Per-key ordered delivery so mutable secondary keys (entity_by_qn) stay correct.
• Full edge identity on delete so dual adjacency tables do not orphan rows.

Topics / streams

Prefer Atlas notification stream (full payloads). Logical decoding is allowed only with the delete rules below.

Per-key order (hard invariant)

Consumers must process events for a given partition key in order (single consumer per key, or Kafka partition = key).

At-least-once + Impala UPSERT is safe for latest-state entity_flat / classifications only under this order.

Mutable qualifiedName (entity_by_qn)

Atlas unique attributes are type-scoped. Digest:

qn_digest = first_16_bytes(SHA-256( type_name || 0x1F || qualified_name ))

(UTF-8; separator 0x1F unit separator.)

Rename / type change on entity G:

1. DELETE FROM atlas.entity_by_qn WHERE qn_digest = digest(old_type, old_qn)
2. UPSERT atlas.entity_by_qn (digest(new_type, new_qn), G, new_type)
3. UPSERT atlas.entity_flat (... new qualified_name, type_name ...)

A retried (1) after (2) without order → missing mapping. Ordering prevents this; Kudu has no conditional version compare on UPSERT.

Entity mutations → Kudu

Hot columns on entity_flat only: guid, type_name, qualified_name, name, state, created_ts, updated_ts. All other attributes → props_json (size-capped). Never put search-critical fields only in props_json.

Timestamps: store as BIGINT microseconds since epoch UTC (UNIXTIME_MICROS-compatible).

Edge mutations → Kudu

Delete decoding prerequisite

If using PostgreSQL logical decoding on AGE label tables:

• Default REPLICA IDENTITY DEFAULT emits only pkey (graphid) on DELETE — insufficient for Kudu.
• Required before wiring: either
  • ALTER TABLE … REPLICA IDENTITY FULL on edge label tables feeding the outbox, or
  • source deletes from Atlas notifications with full edge payload.

Do not ship sync without one of these.

Classification mutations

No guid-leading mirror table by default (single-entity tag UI stays on AGE).

Audit

Append-only: UPSERT/INSERT into entity_audit with (guid, event_ts, seq).
event_ts immutable monotonic; do not update. Retention via future RANGE partitions + DROP RANGE PARTITION (prod).

Writers

FDW consumers

Foreign tables: config/atlas/kudu_projections_fdw.sql.
Traversal uses frontier loop + src = ANY($1::bytea[]) pushdown — not recursive CTE over foreign tables.

Governance scale plane (Atlas / Ranger → Postgres AGE → FDW → Kudu)

Why this exists: Federated engine services (Ægir, Atelier, Gaius, Hermes) must be able to run simultaneously without overwhelming the lab governance database. This lab uses devenv PostgreSQL 16 on :5455 (services.postgres in devenv.nix) with Apache AGE. That is not pglite. pglite is Atelier’s AMP embedded Postgres and is only intended for AMP-style deployments, not this Signals core.

Scale is achieved by engineering Atlas and Ranger to keep connecting to Postgres, while Postgres uses impala_fdw to reach Kudu for bulk governance tables. RustFS holds durable objects on $SIGNALS_DATA_ROOT/rustfs.

This is a prerequisite for multi-engine operation and for signals-protocol.

Primary path (remember this)

  Atlas JVM  ── JDBC ──►  Postgres :5455  (DB signals + AGE atlas_graph / signals_ol)
  Ranger JVM ── JDBC ──►  Postgres :5455  (DB ranger — admin policies)
                              │
                              │  foreign tables (impala_fdw, Kerberos HS2 / kudu_scan)
                              │  atlas_entity_flat, ranger_tag_resource, …
                              ▼
                         Kudu  (atlas.* / ranger.* projection tables)
                              ▲
                              │ outbox / notification UPSERT (writers)
                         Impala HS2

Hard rule: Do not host AGE openCypher topology on Kudu. Do not make engines bypass Postgres for Atlas/Ranger identity — they use Atlas/Ranger APIs (or SQL against Postgres :5455, which FDWs to Kudu). Historical audit: docs/scratch/2026-08-07/204458_fdw-atlas-ranger-scale-audit.md.

Federated engines / Hermes
        │ discovery + REST / gRPC
        ▼
  Atlas :21010  /  Ranger :6080  /  S3 :9010
        │                │              │
        │ JDBC           │ JDBC         │ objects
        ▼                ▼              ▼
     Postgres :5455 (AGE)            RustFS
        │  AGE SoR + Ranger admin    ($SIGNALS_DATA_ROOT/rustfs)
        │  + foreign tables
        ▼
     Kudu (scale projections via FDW)

Bulk governance SQL from tools that sit on this Postgres (Aegir, analytics, Weathership helpers) should hit foreign tables → Kudu, not sequential openCypher over the whole estate.

Split of authority

Atlas → Kudu (via Postgres FDW)

Atlas continues to use AGE on Postgres as SoR. Projections are derived; consumers on Postgres :5455 query foreign tables. Continuous fill needs the outbox worker (notifications → Impala UPSERT).

Still required for “fully leverage”:

1. Outbox worker (per-key order, at-least-once)
2. True kudu_scan / libkudu_client for pk_lookup latency
3. Atlas/AGE and app SQL prefer foreign tables for bulk entity/tag joins

Ranger → Kudu (via Postgres FDW)

Ranger admin remains on Postgres. Tag/resource denorm lives on Kudu and is read through FDW so multi-engine load does not thrash x_tag* heaps. TagSync / outbox → UPSERT is near-term wiring.

RustFS (object store)

Implications for signals-protocol

Discovery should advertise Atlas, Ranger, OL, OBJECT_STORE, and that governance bulk SQL is available via Postgres :5455 + FDW (not “raw Kudu only”).

Engines leave SoR writes to Atlas/Ranger APIs; bulk analytics and tag joins go through the scale plane above.

Ops checklist

just bootstrap
devenv up -d
just atlas-kudu-projections-seed   # Kudu tables + foreign tables on Postgres :5455
just ranger-kudu-projections-seed
just gov-kudu-projections-seed     # both
mc ls local/

Related

• Atlas → Kudu outbox
• Storage and backup
• Signals protocol core
• OpenLineage + Atlas

Signals Control Plane UI (signals-ui)

Status: Architecture plan (repo + stack decisions locked)
Date: 2026-08-10
Repo: git@github.com:weathership/signals-ui.git
Signals path: components/signals-ui (git submodule)
Related: OpenLineage + Atlas, YuniKorn, MiNiFi sentinels, signals-federation Zarf

Thesis

signals-ui is the primary backplane UI for Signals. Admission, queues, and process visibility go through YuniKorn; operators open :9889.

1. Superset of stock yunikorn-web — YK REST used in rch-yunikorn-web, plus Signals views (lineage, sentinels, federated surfaces).
2. Isolated submodule — developed in weathership/signals-ui, vendored into Signals as components/signals-ui (same pattern as other components).
3. Idiomatic Rust for the service — no Node.js runtime in production. Axum (tokio-rs/axum); Askama HTML. Visual law: Keiretsu (Atelier) + Cloudera logo for brand continuity with Atelier/Aegir; dark + light.
4. Federation overwatch — MiNiFi sentinels, OTel, Atlas OL lineage, and engine discovery sit beside YK views in one process.
5. Engine required — /readyz is zndx.engine.v1.Engine/Status with capability=scheduler healthy. SIGNALS_UI_ALLOW_NO_YK=1 is chrome-only lab (skip Status). YuniKorn REST is private to the engine.

                    ┌──────────────────────────────────────────────┐
                    │  signals-ui  (Rust service · Keiretsu UI)    │
                    │  = yk-web capabilities  ∪  Signals value-add │
                    └────────┬──────────┬──────────┬───────────────┘
           YK /ws/v1/*       │  C2/OTel │  Atlas   │  discovery
                             ▼          ▼          ▼
                      YuniKorn     MiNiFi     Atlas :21010
                      (admission)  sentinels  /api/v1 + /api/atlas

0. Repository and process

Submodule lifecycle (Signals monorepo)

# once repo exists and has an initial commit
git submodule add git@github.com:weathership/signals-ui.git components/signals-ui
# devenv: process + task to build release binary
# cargo build --release -p signals-ui

Do not embed a large Node monorepo under signals root for this product.

1. Strict yunikorn-web superset

1.1 Capability catalog (must implement — parity bar)

Reference: ~/local/src/asf/rch-yunikorn-web/ (Angular; port 9889; hash routes).

Parity means: an operator who today uses yk-web can complete the same tasks against the same YuniKorn without opening Angular. Layout may differ (Keiretsu); data and actions may not.

1.2 YK REST surface (must cover — contract suite)

From yk-web SchedulerService + health/utilization:

Contract tests live in signals-ui: golden JSON fixtures captured from yk-web’s json-server (json-db.json) and/or live YK; Rust client deserializes every field the Angular models used (see src/app/models/* in yk-web).

1.3 Superset (Signals value-add — after / with parity)

Rule: YK parity is not optional “phase debt.” Value-add routes may ship behind flags, but YK surface area ships complete before declaring yk-web replaced.

1.4 Deliberately discarded (from stock yk-web)

2. Rust service architecture

2.1 Crate layout (proposed)

signals-ui/                     # weathership/signals-ui
  Cargo.toml                    # workspace
  crates/
    signals-ui/                 # binary: axum server
    signals-ui-yk/              # YK REST client + models (parity with yk-web models)
    signals-ui-atlas/           # Atlas /api/v1 + /api/atlas clients
    signals-ui-otel/            # OTel query helpers
    signals-ui-core/            # config, error, correlation ids
  assets/                       # keiretsu.css, kumo tokens, static
  ui/                           # Leptos/Dioxus app or templates (if split)
  tests/
    yk_parity/                  # contract tests vs fixtures + live YK
  README.md

2.2 Runtime responsibilities

2.3 Presentation options (choose in Phase 0 spike)

Recommendation: spike A or C in week one; pick on binary size, charting story, and team velocity. Charts: prefer Rust-friendly (e.g. plotters → SVG, or lightweight canvas) over pulling a Node chart stack.

Keiretsu/Kumo remain CSS tokens, not a JS theme runtime.

3. UI standards (Keiretsu + Kumo)

Canonical references (do not invent a third palette):

Laws

1. Elevation is lightness — canvas < base < elevated
2. Accents on accent duty only — brand for primary/link/focus
3. Contrast band — text 10–13:1 on surfaces

Default dark; full light parity required.

4. Atlas backplane (feeds superset lineage)

Closing Marquez-native gaps remains Atlas enrichment; signals-ui is a client. See openlineage-atlas.md.

Native Atlas /api/atlas/v2/lineage remains available for governance graphs.

5. Sentinels + YK as uniform process proxy

duration_display = coalesce(duration_ol, duration_otel, duration_yk)
correlation: openlineage.runId · yunikorn.applicationId · otel.trace_id · signals.workload_id

Standardize keys in signals-protocol; signals-ui BFF implements merge.

6. Phased delivery

Phase 0 — Repo + Rust skeleton

• Create weathership/signals-ui (Apache-2.0 or project license as decided)
• Workspace: axum binary, signals-ui-yk client, health, config
• Vendor Keiretsu CSS; dark/light shell page
• Add submodule components/signals-ui in Signals
• devenv process + cargo task; document port
• Spike: Leptos vs Askama+HTMX decision recorded in signals-ui README

Phase 1 — YK-web complete parity (superset base)

• All §1.1 routes/capabilities
• All §1.2 REST clients + deserialization of yk-web model fields
• Contract tests vs json-db.json fixtures from rch-yunikorn-web
• Live YK e2e in lab when scheduler is up
• Acceptance: side-by-side checklist vs Angular yk-web; no missing operator task

Phase 2 — Superset: process proxy + Atlas lineage

• Correlation + merged duration API
• Sentinel + OTel modules
• Lineage (table; column when Atlas ready)
• Jobs/datasets/events/sources views (Atlas /api/v1)

Phase 3 — Atlas depth (backplane; may be parallel in components/atlas)

• Column lineage, fields, versions, sources enrichment
• OL duration last-N from event pairs
• signals-ui consumes enriched APIs

Phase 4 — Production federation

• ZT auth, namespace tenancy
• OL ↔ Atlas entity projection
• Zarf/package packaging for air-gap control plane

7. Non-goals

8. Acceptance criteria

YK-web replacement (hard gate)

• Every yk-web operator task works in signals-ui against live YK
• Contract suite green for all §1.2 endpoints
• Documented mapping: yk-web screen → signals-ui route

Superset

• At least one Signals-only module (sentinels or lineage) shippable
• Merged duration for a process with YK+OTel and optional OL

Platform

• Single Rust binary (or small set of crates) in release builds
• Keiretsu dark + light
• Submodule builds under Signals devenv/CI

9. Immediate next actions

1. Create weathership/signals-ui empty repo + Apache-2.0 (or project) license
2. Scaffold Axum hello + YK client crate + Keiretsu static shell
3. Import yk-web json-db.json as fixtures; first contract test (/ws/v1/partitions)
4. Submodule add into Signals when skeleton builds
5. Presentation spike (Leptos vs HTMX) ≤ 1 week, then freeze
6. Parallel in Atlas: column-lineage facet spike (feeds Phase 2–3)

Related

• OpenLineage SoR: openlineage-atlas.md
• Federation Zarf: signals-federation-zarf.md
• Sentinels: components/signals-protocol/specification/operations/minifi_sentinels.md
• YK web reference (capability only): ~/local/src/asf/rch-yunikorn-web/
• Theme: Atelier ui/src/styles/theme-keiretsu.css, ui/src/theme/kumo.ts

Identity and access (authn → authz)

Signals (and the broader ZNDX constellation) separates who you are from what you may touch. There is no product path that skips either layer.

Layers

Zero Trust is the outer door. Kerberos (or equivalent workload identity in cloud) is the inner door for engines. Atlas + Ranger turn “authenticated” into “authorized.”

Precedent: Gaius

Gaius already puts Cloudflare at the edge:

• Workers on gaius.zndx.org — OAuth callbacks, A2UI surfaces, KV sessions (gaius/infra, gaius-ui/apps/worker).
• IdP direction (gaius-kb-interop roadmap): GitHub OAuth (developers), Okta OIDC (enterprise), unified under Cloudflare Zero Trust Access policies (group membership, risk signals, SCIM).
• WebRTC called out next to the SPA: access-controlled edge first, then media client.

That is the house pattern: edge authn (CF Access + IdP) → session/JWT → app and media planes — not browser → bare port on the lab network.

Precedent: synth (early HTTPS / secure context)

Synth’s multi-client observer work already hits the hard gate:

• getUserMedia (mic/camera) requires a secure context (HTTPS) off localhost — WebKit/iPad has no exception.
• Today: Caddy TLS in devenv for LAN secure context (internal CA).
• Explicit alternative called out: cloudflared (atelier-style tunnel) — good for remote reach, weaker as a pure LAN story unless paired with Access.
• WebRTC is the upgrade path for lower-latency audio/video once the observer chain is multi-client.

Early synth adoption is “TLS for observations.” Full product federation of audio + video agent observations (and later omni models) needs ZT + HTTPS as the default, not a late bolt-on.

WebRTC, omni models, federation

Federation of multi-modal agent features should assume:

1. Cloudflare Zero Trust (or equivalent) in front of every human/agent web surface.
2. Okta / GitHub as IdPs into Access policies.
3. Kerberos (or cloud IAM) for service-to-service data plane.
4. Atlas classifications → Ranger policies for data authorization.

Signals today (lab)

Non-goals / rejected

• NOSASL or “auth optional” for Impala/Kudu in devenv
• Shipping multi-modal agent UIs over plain HTTP outside localhost
• Treating Cloudflare only as CDN without Access policies for app routes

Related

• Lab Kerberos: docs/current/src/operations/storage-and-backup.md,
  scripts/signals_kerberos.sh, just bootstrap
• Gaius edge: gaius/infra, gaius-ui/apps/worker
• Gaius IdP roadmap: gaius-kb-interop/design (Okta + Zero Trust + WebRTC)
• Synth secure context: synth docs on multi-client observers + Caddy TLS
• Arclight GTM: frictionless zero-trust access as adoption requirement

Deployment Modes

Signals supports four deployment modes, from local development to full cloud production.

Laptop Development

Local-only with devenv services (PostgreSQL, KDC). Optional k3d cluster for Kubernetes workloads.

devenv up                              # Start PostgreSQL + KDC
devenv tasks run k8s:provision         # Optional: create k3d cluster
devenv tasks run k8s:deploy-dask       # Optional: deploy Dask on k3d

Best for: iterating on engine code, feature development, BDD scenarios at Tier 0-1.

Workstation

Local devenv with optional RKE2 for production-grade Kubernetes. Suitable for GPU-accelerated workloads.

devenv up                              # Start core services
devenv tasks run k8s:deploy-dask       # Deploy Dask on RKE2

Best for: integration testing, GPU workloads, running Tier 2 scenarios.

Hybrid (Workstation + AWS)

Local development environment with AWS RKE2 cluster for distributed compute. The engine and visualization run locally; heavy computation is offloaded to AWS.

devenv up                              # Local services
devenv tasks run aws:provision         # Provision AWS infrastructure
devenv tasks run aws:deploy            # Deploy RKE2 + Dask on AWS

Best for: distributed workloads that exceed local capacity, Tier 3 scenarios.

Full AWS

Everything runs on AWS EC2 with RKE2. Access via Cloudflare Zero Trust (WARP device posture).

devenv tasks run aws:provision         # VPC, EC2, S3
devenv tasks run aws:inventory         # Generate Ansible inventory
devenv tasks run aws:deploy            # Full stack deployment
devenv tasks run aws:verify            # Post-deployment checks

Best for: production deployments, air-gap environments, Tier 3 full-stack validation.

Air-Gap Variant

Any mode with Kubernetes can use Zarf for disconnected deployment. The air-gap variant removes the NAT gateway (airgap_mode = true in OpenTofu), relying on VPC endpoints for S3 access and the Zarf internal registry for container images.

devenv tasks run aws:provision         # With airgap_mode = true
ansible-playbook playbooks/airgap-deploy.yml

See Air-Gap (Zarf) for details.

Infrastructure Layers

Each deployment mode uses a subset of the infrastructure stack:

See the Infrastructure section for detailed documentation of each layer.

Metadata Tagging

sigint classifies Impala columns against the SIGDG ontology and writes those tags to Atlas. Ranger consumes the tags. Discoverability and sensitivity labels come from that loop.

Architecture

How It Works

1. Tables created in Impala become visible in Atlas as entities with columns, types, and ownership metadata.

2. The tagging service watches Atlas for new or changed entities (tables, columns). When a new table appears, the service reads its name, column names, column types, and any existing metadata.

3. The classification model maps each name to the SIGDG ontology, assigning both a semantic category and a sensitivity quality. For example:

   • Column email → SIGDG:0025 ContactInformation with SIGDG:1030 Confidential
   • Column ssn → SIGDG:0011 GovernmentIdentifier with SIGDG:1040 Restricted
   • Column total_amount → SIGDG:0021 FinancialInformation with SIGDG:1020 Internal
   • Table transactions → SIGDG:0050 TransactionInformation

4. Classifications are written back to Atlas as tags on the entity, making them visible in the catalog UI and available for policy enforcement via Ranger.

The classification model uses a context engineering approach — each column is represented as a structured feature vector that can be measured and optimized using SAGE feature importance analysis. The pipeline outputs Dempster-Shafer belief intervals at every hierarchy level via the evidence fusion layer.

Atlas Integration

Atlas provides the metadata catalog that makes Impala-managed Kudu tables and columns visible as governed entities:

• Entity types: rdbms_table, rdbms_column, rdbms_db, rdbms_instance — Atlas stock RDBMS model (2000-RDBMS), same family as Aegir (catalog bridge cut over from interim hive_*)
• Classifications: Applied as Atlas tags using SIGDG CURIEs (e.g., SIGDG_0025_ContactInformation)
• Lineage: Atlas tracks table-level lineage from INSERT…SELECT and CTAS operations
• AGE graph queries: Atlas metadata stored in PostgreSQL with AGE enables graph traversal of classification relationships, lineage paths, and impact analysis

Atlas + Ranger Policy Flow

Once entities are tagged in Atlas:

1. Ranger policies reference SIGDG classifications (e.g., “mask all columns tagged SIGDG:0022 PaymentInformation”)
2. Impala enforces Ranger policies at query time
3. The result: automated column masking, row filtering, and access control driven by ontology-grounded metadata tags

Validated Pipeline

The end-to-end tagging pipeline has been implemented and validated with 8 BDD scenarios (6 meta-tagging + 2 pipeline tagging):

1. Atlas catalog bridge — Python function registers Impala-managed Kudu tables in Atlas via REST API. Validated by 4 catalog sync scenarios.
2. Tagger service — Tagger class (src/sigint/tagger.py) orchestrates sample → classify → tag:
   • ImpalaSampler reads column names, types, and sample values from Impala
   • EmbeddingClassifier classifies against the SIGDG taxonomy using sentence-transformer embeddings
   • AtlasClient writes classifications back as SIGDG tags with confidence scores and evidence
3. Air-gap isolation — Shared HF caches (HF_HOME / HF_HUB_CACHE / SENTENCE_TRANSFORMERS_HOME, lab RAID), HF_HUB_OFFLINE=1, zero external network calls. See Air-Gap (Zarf).
4. HOCON config — Single source of truth (config/base.conf) drives both application code and BDD test helpers. All connection parameters (Impala host/port, Atlas URL/credentials, model cache) flow from config.

Running the Pipeline

# Pre-cache model (once)
just cache-models

# Dry-run: classify without writing to Atlas
just tag-dry-run default.my_table

# Live: classify and write SIGDG tags to Atlas
just tag default.my_table

# Or via Python module
uv run python -m sigint --tables default.my_table --dry-run

Next Steps

1. Event-driven tagging: Watch Atlas for new entities and trigger classification automatically (replace manual just tag invocation)
2. Evidence fusion calibration: Calibrate DST discount factors and mass constants against held-out data; resolve source independence assumption (see Research Roadmap)
3. OWL formalization: Publish SIGDG as OWL ontology with BFO imports, verify via RASE 5-gate pipeline
4. Ranger tag-based policies: Column masking and access control driven by Atlas SIGDG classifications
5. Feedback loop: Analyst corrections to classifications improve the model over time
6. Data lifecycle integration: Preserve classifications across Kudu→Iceberg CTAS migrations

Context Engineering

Context engineering is the discipline of deliberately constructing, measuring, and optimizing the input features fed to an embedding classifier. Rather than treating the text assembled from a column’s metadata as an ad-hoc string, we decompose it into 12 discrete, ablatable features and use SAGE (Shapley Additive Global importancE) to quantify each feature’s contribution to classification accuracy.

The goal: replace intuition about “what information helps the classifier” with measured Shapley values that show exactly which features drive predictions and which are noise.

The Problem

An embedding classifier that encodes column metadata as free-form text achieves high accuracy on well-named columns — but most correct predictions are assisted by string-matching heuristics (name boost). Columns with opaque or generic names (e.g., col0, field_42) expose the model’s dependence on column names: cosine similarity drops dramatically on these columns while maintaining high accuracy on semantically named columns.

To build a classifier that generalizes beyond known column names, we need to:

1. Know which features actually contribute to correct predictions
2. Measure how much accuracy degrades when each feature is removed
3. Systematically reduce reliance on name matching
4. Compare multiple classification methods on the same feature set

Feature Decomposition

Each column is represented by 12 named features extracted from its metadata:

Features are composed into pipe-separated embedding text via ColumnFeatures.to_embedding_text(mask), where the mask controls which features are included. When a column has a generic name (col0, field_1, Unnamed), the value_description feature substitutes for the uninformative name in the embedding text; otherwise it is appended as complementary information. See Heuristic Elucidation for the methodology behind this feature.

With all features enabled, a column might produce:

payment card number | int | 4111111111111111, 5500000000000004 | cardinality=42 |
  patterns: credit_card_pattern | avg_len=16.0 | numeric=1.00 |
  siblings: first name, last name, expiry date | table=personal_data

Pattern Detectors

Eight regex-based pattern detectors scan sample values for structural signals:

A pattern is flagged when at least 1/3 of sample values match. This threshold prevents spurious matches from a single outlier value.

SAGE Feature Importance

SAGE (Shapley Additive Global importancE) measures the global importance of each feature by estimating how much each feature contributes to reducing classification loss. Unlike local explanation methods (LIME, SHAP per-instance), SAGE provides a single importance score per feature across the entire dataset.

How It Works

SAGE marginalizes each feature by substituting values from other samples in the dataset. For example, when marginalizing column_name for column email, SAGE asks: “what if this column had the name trade_secrets but the same sample values, cardinality, and patterns?”

The implementation:

1. Feature index matrix: X has shape (N, 12) where X[i, j] = i — each sample initially uses its own value for every feature.

2. Value lookup tables: For each feature j, a table maps sample index → feature text. When SAGE permutes X[i, j] to index k, the model uses sample k’s value for feature j instead.

3. Model wrapper (FeatureMaskModel): Looks up text from the per-feature tables according to X, assembles embedding text, encodes via sentence-transformers, and returns softmax’d cosine similarities as class probabilities.

4. SAGE estimation: MarginalImputer + PermutationEstimator with cross-entropy loss estimates each feature’s Shapley value across n_permutations (default 512) random orderings.

The result is a vector of 12 importance values with standard deviations — a quantitative answer to “which features matter?”

Pipeline

The classification pipeline runs in four stages, producing a parquet with three independent signals per column:

Three-Signal Comparison

When ground truth is provided (--ground-truth), the pipeline produces three independent classification signals per column:

The CatBoost CV baseline uses category reference embedding augmentation to overcome extreme low-data regimes. Each fold’s training set includes all category reference embeddings (the same texts cosine uses as targets), giving at least 2 training points per class even in held-out folds.

The train→eval pipeline replaces k-fold CV with synthetic training data and several additional techniques. See Classification Training for the full methodology and accuracy progression.

The pipeline always produces Dempster-Shafer belief intervals at every hierarchy level. See Evidence Fusion for the full DST architecture.

Column Kinds

The pipeline classifies each column into one of three kinds based on naming patterns:

• data: Semantically named columns (email, first_name, amount) — cosine excels here
• opaque: Generic or positional names (col0, field_42) — CatBoost train→eval is needed
• row_id: Row identifiers — excluded from GT evaluation

Ablation Runs

The pipeline supports controlled ablation experiments:

# Three-signal comparison with ground truth
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --ground-truth <ground-truth.json> \
    --output build/sigint_embeddings.parquet

# Feature ablation (disable specific features)
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --disable-features sample_values sibling_context \
    --output build/sigint_ablation.parquet

# With SAGE feature importance
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --sage-permutations 512 \
    --output build/sigint_embeddings.parquet

# Visualize with embedding-atlas
embedding-atlas build/sigint_embeddings.parquet --text embedding_text

Output Schema

Each run produces a parquet (38 columns) and companion report JSON:

Current Status

The train→eval pipeline closes the opaque-name accuracy gap. Remaining errors are inherently confusable category pairs — types whose values are structurally identical (e.g., two address subtypes, two hex identifier formats). See Classification Training for the full methodology.

What remains:

1. Confusable pair resolution — remaining errors cluster in category pairs that share identical value patterns. Resolving these requires either richer context (e.g., table-level schema hints) or category consolidation
2. Feature refinement — richer pattern detectors (date formats, currency symbols, statistical distributions), deeper sample analysis (value distributions, min/max/mode)
3. Cross-dataset validation — accuracy on datasets with different naming conventions and taxonomies (see GitTables CTA benchmark)
4. Feedback integration — analyst corrections feed back as training signal, with SAGE tracking whether corrections improve non-name features

Classification Training

The classification pipeline’s CatBoost train→eval mode trains on synthetic data and evaluates on real data. This page documents the synthetic data generator, the training pipeline, and the techniques that bridge the domain shift between synthetic and real columns.

Note: CatBoost replaced XGBoost in March 2026. CatBoost’s ordered boosting (posterior_sampling=True) eliminates the self-training target leakage identified in the integrity audit. See the CatBoost migration section for details.

Why Synthetic Training

Real datasets typically exhibit extreme data scarcity: many leaf categories with only a few samples each. K-fold cross-validation in this regime produces poor accuracy because each fold’s training set has at most 1-2 examples per class.

A second challenge is the opaque name problem. Columns with generic or positional names (e.g., col0, field_42) carry no semantic signal. Cosine similarity achieves high accuracy on semantically named columns but fails on opaque names — the classifier depends on column names, not value patterns.

Synthetic training data addresses both problems: it generates thousands of columns per category with controlled name diversity, and half the synthetic columns use opaque names, forcing CatBoost to learn from value patterns rather than column names.

Synthetic Data Generator

scripts/generate_meta_tagging_train.py generates wide-format CSV training data for all leaf categories in the target taxonomy.

Name Generation

Each category produces two kinds of synthetic column names:

Semantic names use a synonym table (~60 terms with abbreviations and variants) to generate diverse human-readable names. Techniques include:

• Case variants: snake_case, camelCase, UPPER_SNAKE_CASE
• Synonym expansion: phone → tel, ph, phn
• Word-dropping: payment_card_number → payment_number, card_number
• Prefixed variants: user_, primary_, customer_, acct_, src_, raw_

Opaque names use 13 prefix templates (field_, col_, meta_, var_, etc.) with random numeric/alphabetic suffixes, mimicking generic column naming patterns found in real databases.

Each category generates a 50/50 split: half semantic-name columns, half opaque-name columns.

Value Generators

70+ generator functions cover all SIGDG leaf categories across the full taxonomy:

• Payment card data — PANs (Visa/MC/Amex/Discover prefixes), CVVs, magstripe tracks, BINs, last-4, expiration dates, masked PANs, PINs
• Identifiers — SSNs, passports, driver’s licenses, CPF (Brazil), PAN (India), VATIN, UUIDs, device IDs, serial numbers, MAC addresses
• Contact — full names, first/middle/last/nickname, emails, phone numbers (full and split: country code, area code, subscriber, extension), social media handles
• Addresses — full addresses, split components (street, city, state, postal, country), billing/shipping variants, coarse/precise geolocation
• Demographics — age, birthdate, gender, race, education, employment, income, benefits, background checks
• Health — mental health conditions, physical conditions, genetic markers, biometric IDs
• Product usage — login events, crash types, security references, forum posts, photos, videos, user activity, device metrics, app bundles, user agents
• Authentication — passwords (masked), security Q&A, e-signatures, key material, key digests
• System — IP addresses, LDAP groups, permissions, session data, URLs, cluster nodes, file paths, executables, runtime data
• Business — document references, SEC filings, trade secrets

Output Format

build/datasets/sigint_train/
  ├── synth_001.csv ... synth_NNN.csv   (wide-format, 100 rows each, 50 cols per file)
  ├── ground_truth.json                  (column → SIGDG code)
  └── taxonomy.csv                       (copy of controlled vocabulary)

CSV headers use table.column format (e.g., synth_001.credit_card_number) matching the eval data’s convention.

Train/Eval Pipeline

The --train-dir flag in build_sigint_embeddings.py activates train→eval mode. The pipeline applies four techniques that collectively raise accuracy from a 33.4% naive baseline to 83.1%.

Feature Vector

Each column is represented by a 992-dimensional feature vector:

[full_emb(384) | value_only_emb(384) | discrete(12) | cosine_sims(212)]

Dual Embedding

The full embedding encodes everything including column name and table context — strong for columns with semantic names. The value-only embedding strips column_name, source_table, and sibling_context, encoding only value patterns, cardinality, entropy, and detected patterns. This bridges the domain shift for columns with opaque or generic names.

Category Reference Augmentation

All taxonomy reference embeddings (the same texts cosine similarity uses as targets) are included as training samples. These anchor embeddings exist in the same embedding space as the real eval data, teaching CatBoost the mapping from embedding regions to categories even when synthetic embeddings differ from real ones.

Cosine Similarity Features

For each column, the pipeline computes cosine similarity between the column’s value-only embedding and each category reference embedding. These per-category similarity features encode the cosine classifier’s knowledge in a domain-invariant way (category references are identical for training and evaluation).

Discrete Feature Scaling

The 12 discrete features (cardinality, null ratio, entropy, 8 pattern flags, value description) are multiplied by \(\sqrt{384/12} \approx 5.7\) so their magnitude competes with the 384-dimensional embedding vectors. Without scaling, gradient boosting ignores the discrete features because individual embedding dimensions dominate split gain.

Paired Column Propagation

A post-prediction pass exploits structural column relationships in the dataset. When CatBoost is uncertain about one column (confidence < 0.50) but confident about a structurally related column (confidence ≥ 0.35), the confident prediction propagates to the uncertain one. This is a dataset-specific heuristic — the particular structural relationship varies by dataset (see Heuristic Elucidation).

StandardScaler

The full feature vector is normalized with StandardScaler before CatBoost training for stable gradient boosting across features with different scales.

Results

Accuracy progression on a GT-labeled evaluation set, showing each technique’s marginal contribution:

The naive k-fold CV (48.0%) cannot learn with ~2 examples per class. Synthetic training data + dual embedding (full + value-only) provides the largest gain. Column propagation corrects 43 annotation columns where CatBoost is uncertain but the preceding data column is confident. Self-training injects GT-labeled evaluation columns into the CatBoost training set, raising accuracy from 83.1% to 99.4% — legitimate for the LLM annotation reproduction workflow (see Self-Training Mode).

Validated numbers (2026-03-24, GPU-accelerated): CatBoost standalone (dual embedding + cosine sims + ordered boosting + propagation) achieves 83.1% (291/350, data=88.0%, ann=78.3%). CatBoost with self-training: 99.4% (348/350, data=99.4%, ann=99.4%) — only 2 residual errors, both in the Masked PAN category. SVM standalone: 84.6% (296/350). SAGE analysis: 489s on GPU vs 13,278s on CPU (27x speedup). CatBoost requires CPU for posterior_sampling=True and rsm=0.6 (not supported on CatBoost GPU for multiclass).

SVM text format fix (2026-03-24): The SVM text format mismatch (SVM trained on short name | type | values but evaluated on full 12-feature text) has been fixed. EmbeddingClassifier._build_svm_text() now ensures consistent short text format between training and inference. 5-source DST accuracy corrected from 60.6% to 84.6%.

Discovery Methodology

Each technique in the accuracy progression was discovered through benchmark observation, not designed a priori. Dual embedding emerged from observing that opaque-name columns fail because names are uninformative. Category reference augmentation emerged from the observation that many classes with few samples each is insufficient for gradient boosting. Column propagation emerged from noticing exploitable structural relationships between columns in the dataset.

This pattern — observe a phenomenon, hypothesize a mechanism, implement it as a feature, and measure its contribution with SAGE — is the heuristic elucidation methodology. SAGE Shapley values quantify each technique’s marginal contribution, and cross-benchmark validation (against GitTables CTA) prevents overfitting to a single dataset.

CatBoost Migration

CatBoost replaced XGBoost in March 2026 for two reasons:

1. Self-Training Redesign

The XGBoost pipeline included an implicit self-training step that injected cosine pseudo-labels (confidence ≥ 0.50) from evaluation data into the training set. For benchmark evaluation (where test/train splits matter), this constituted target leakage. The implicit self-training was removed and replaced with an explicit, opt-in --self-train mode (see Self-Training Mode) for the LLM annotation reproduction workflow where it is legitimate.

2. Ordered Boosting

CatBoost’s posterior_sampling=True enables ordered boosting — a method that constructs each tree using only “historical” data points, preventing the prediction shift that occurs when a model is trained and evaluated on the same data distribution. This provides a principled alternative to self-training without leakage.

Hyperparameter Mapping

Evidence Fusion Integration

CatBoost’s predict_proba() output is now available as a mass function source for evidence fusion. When a trained CatBoost model is loaded, its class probabilities are converted to a BeliefAssignment via catboost_to_mass(), contributing independent evidence alongside cosine similarity, pattern detection, and name matching. CatBoost’s virtual ensembles can additionally provide per-class variance estimates, enabling adaptive discounting: high model uncertainty → more mass allocated to \(\Theta\) (total ignorance).

SHAP Explanations

The pipeline computes per-item feature explanations using CatBoost’s built-in TreeSHAP (\(O(TLD)\) complexity). For each classified column, the top 3 most influential feature groups are reported:

CatBoost’s 992-dimensional feature space is grouped into interpretable categories: full_embedding (384 dims), value_only_embedding (384 dims), and 11 discrete features (column_name, sample_values, cardinality, null_ratio, entropy, pattern_*, value_description, sibling_context) plus cosine similarities. SHAP values within each group are summed to produce a single importance score.

Disable SHAP with --no-shap for faster runs that skip the TreeSHAP computation.

Remaining Errors

Remaining errors cluster in confusable category pairs where values are structurally identical — for example, two hex identifier types or two address subtypes that share the same format. These are registered as confusable pairs in the DST frame so that mass flows to the pair rather than forcing an arbitrary leaf choice.

These pairs cannot be distinguished by value patterns alone — resolution requires richer context (table-level schema, data dictionary, or category consolidation).

Self-Training Mode

The --self-train flag activates self-training, where GT-labeled evaluation data is injected into the CatBoost training set. Off by default.

When Self-Training is Legitimate

The primary workflow for this pipeline is LLM annotation reproduction:

1. An LLM provides initial column classifications (the “ground truth”)
2. ML models learn to reproduce those LLM annotations transparently
3. The ML pipeline provides explainability, reproducibility, and measured feature importance

In this workflow, self-training is not target leakage — the LLM annotations ARE the ground truth to learn. The ML pipeline’s goal is highly accurate recreation of the LLM-provided mappings.

Validated Results

Self-training (1 round, direct GT injection) achieves 99.4% (348/350) — only 2 residual errors, both in the Masked PAN category (one opaque annotation column confused with Security Question at 0.039 confidence, one data column confused with PAN India at 0.050 confidence). Training set: 7,684 synthetic + 212 reference + 348 GT-injected = 8,244 samples. Pipeline wall clock: 68 minutes (CatBoost CPU training dominates).

When Self-Training is Illegitimate

For benchmark evaluation (test/train splits measuring generalization to unseen data):

• Do NOT use --self-train — this inflates accuracy metrics
• The default mode (no self-training) maintains strict train/test separation
• All published benchmark accuracy numbers use the default mode

Multi-Round Refinement

With --self-train-rounds N (N > 1), the pipeline iterates:

1. Round 1: Inject all GT-labeled eval columns with their LLM labels
2. Round 2+: Inject columns whose CatBoost confidence exceeds --self-train-threshold (default 0.80) using CatBoost’s predicted labels as pseudo-labels
3. Repeat until no new columns qualify or round limit reached

Embedding encoding happens once before the loop — only CatBoost training repeats per round.

Commands

Single-command workflow (recommended)

The --auto-generate flag inlines the synthetic data generator, producing a single command that generates training data, trains CatBoost, classifies, computes SHAP explanations, and writes the output parquet:

uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --ground-truth <ground-truth.json> \
    --auto-generate --variants-per-category 50 \
    --output build/sigint_shap_eval.parquet

The output parquet includes all classification columns plus item-wise SHAP explanations (shap_top{1,2,3}_{name,value}), suitable for visualization with embedding-atlas.

Two-step workflow

For iterating on the synthetic generator separately:

# Step 1: Generate synthetic training data
uv run python scripts/generate_meta_tagging_train.py \
    --data-dir <data-dir> \
    --output-dir build/datasets/sigint_train/ \
    --variants-per-category 50

# Step 2: Train→eval pipeline
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --ground-truth <ground-truth.json> \
    --train-dir build/datasets/sigint_train/ \
    --output build/sigint_embeddings.parquet

Self-training (LLM reproduction mode)

# Direct GT injection — learn the LLM's mapping
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> --taxonomy <taxonomy> \
    --ground-truth <ground-truth.json> \
    --auto-generate --self-train \
    --output build/sigint_self_train.parquet

# Multi-round refinement (3 rounds, 85% confidence threshold)
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> --taxonomy <taxonomy> \
    --ground-truth <ground-truth.json> \
    --auto-generate --self-train \
    --self-train-rounds 3 --self-train-threshold 0.85 \
    --output build/sigint_self_train_3r.parquet

Other commands

# Disable SHAP (faster, skips TreeSHAP computation)
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> --taxonomy <taxonomy> \
    --ground-truth <ground-truth.json> \
    --auto-generate --no-shap \
    --output build/sigint_eval.parquet

# Run tests
uv run pytest tests/sigint/test_generate_train.py -v

Heuristic Elucidation

Each classification dataset presents unique properties — naming conventions, column structure, value distributions — that affect classifier performance in dataset-specific ways. Rather than treating techniques as ad-hoc fixes, the pipeline follows a systematic methodology: observe a phenomenon, hypothesize a mechanism, implement it as a feature or technique, quantify its contribution with SAGE, and validate across benchmarks.

We call this cycle de novo heuristic elucidation: the disciplined reduction of observed dataset-specific phenomena to practice, with measured contribution to overall accuracy.

This page ties together the three classification architecture pages — Context Engineering, Classification Training, and Evidence Fusion — by documenting the methodology that produces the features, techniques, and evidence sources they describe.

The Methodology Cycle

Each step is concrete:

1. Observe: Run the pipeline on a benchmark dataset. Examine accuracy by column kind (data vs. annotation), by category group, and by naming convention. Identify where accuracy drops below baseline expectations.

2. Hypothesize: Formulate the failure mode as a testable phenomenon. “Annotation columns fail because the column name carries no semantic signal” is a hypothesis. “The training set has only 2 samples per class” is another.

3. Implement: Reduce the hypothesis to a feature (added to the CatBoost feature vector) or a technique (added to the pipeline). Every implementation must be expressible as a SAGE-ablatable feature or a measurable pipeline stage.

4. Quantify: Run SAGE to estimate the new feature’s Shapley value — its marginal contribution to accuracy across all columns. Features with near-zero SAGE importance are noise; features with high importance validate the hypothesis.

5. Validate: Test the technique on a second benchmark with different properties. A technique that helps on the meta-tagging dataset but hurts on GitTables reveals a dataset-specific assumption that needs gating.

Heuristic Catalog

Seven heuristics have been discovered through this cycle. Each row links to the architecture page where the technique is described in detail.

Generalization Status

Heuristics 1-3, 5-6 are universally applicable — they address structural properties (class imbalance, feature scale mismatch, uninformative names) that arise in any column classification task.

Heuristic 4 (column propagation) is dataset-specific: it exploits a structural property where columns are related and share category assignments. Other datasets may have different column relationships that warrant different propagation heuristics — but the methodology of exploiting structural relationships is universal.

Heuristic 7 (synthetic data generation) applies when the evaluation data’s generation process is known or inferable. For production data with unknown provenance, the role of synthetic generation shifts to LLM bootstrapping — the LLM classifies a sample of real columns to produce training signal.

Cross-Benchmark Validation

Two benchmarks with complementary properties test whether heuristics generalize:

Method Accuracy by Benchmark

The cross-benchmark comparison reveals three regimes:

1. Cosine dominates (semantic names): Semantic column names directly match category labels. Cosine similarity achieves 99.4% accuracy. CatBoost adds noise.

2. CatBoost dominates (generic/opaque names): Column names are uninformative. CatBoost trained on value patterns achieves 81.6% on GitTables. Cosine adds pure conflict (K > 0.5 on 100% of GitTables columns).

3. Methods are complementary (mixed names): The union of correct answers reaches 66.0% — 12 points above either method alone. A well-tuned DST fusion captures both.

Confidence-Gated Fusion

The regimes map to a principled DST integration rule: use cosine evidence’s own confidence to decide how to weight it against CatBoost.

This regime analysis motivates a planned confidence-gated adaptive discounting refinement (tracked in R-02). The current implementation uses fixed discounts, but the empirical observation is that cosine and CatBoost, despite sharing the embedding space, contribute discriminatively in different regimes — their functional orthogonality mitigates (but does not eliminate) the source independence concern (R-01).

The DST conflict metric \(K\) provides the runtime diagnostic: when \(K > 0.5\), the sources disagree enough that one should be discounted. On GitTables, 100% of columns have \(K > 0.5\) — a clear signal that cosine should be discounted. On meta-tagging data columns, \(K\) is moderate (mean 0.47) — the sources agree enough for standard combination.

Agentic Integration

The heuristic elucidation cycle maps to an agentic workflow where frontier LLMs play the roles of observer, hypothesis generator, and training-signal bootstrapper:

Role of Each Component

LLM as observer: Given a new dataset, the LLM examines column names, value distributions, table structure, and identifies phenomena. For example: “these columns have generic positional names — value patterns are the only discriminative signal.”

LLM as bootstrapper: The LLM classifies a sample of columns to generate training signal for CatBoost, replacing manual ground-truth labeling. This is the target design for production deployment — the LLM runs once to create training data, then the transparent CatBoost+DST pipeline handles ongoing classification without LLM dependency.

SAGE as quality gate: After implementing a new feature, SAGE automatically measures whether it improved accuracy. Features with near-zero Shapley values are noise — they should be discarded regardless of how plausible the hypothesis seemed. This prevents accumulation of heuristics that seemed helpful on one dataset but don’t generalize.

DST conflict as disagreement detector: When a new evidence source conflicts with existing sources (\(K > 0.5\)), the conflict metric flags it for review. High conflict doesn’t necessarily mean the new source is wrong — it may mean the confidence gate needs adjustment. But it does mean the sources should not be naively combined.

Self-Improving Pipeline

The agentic workflow creates a self-improving classification pipeline: each new dataset encounter produces new heuristic candidates, which are automatically validated by SAGE and incorporated (or rejected) based on measured contribution. The LLM’s role shifts from classifier to methodology driver — it doesn’t classify columns directly in production, but it discovers the features and training signal that make CatBoost+DST effective.

Relationship to Research Roadmap

Several items on the research roadmap are consequences of the heuristic elucidation methodology:

The research roadmap is not a static list but a backlog of unresolved observations from previous heuristic elucidation cycles. Each completed research item either produces a new heuristic or validates that a phenomenon is irreducible (e.g., ADID and GUID values are structurally identical — no embedding-based feature can distinguish them without external context).

Evidence Fusion

The evidence fusion layer replaces single-point confidence scores with belief intervals derived from Dempster-Shafer Theory (DST). Each evidence source — cosine similarity, CatBoost prediction, pattern detection, name matching, and SVM short-text classification — produces an independent mass function. These are combined via Dempster’s rule of combination to yield belief intervals \([Bel(A), Pl(A)]\) at every level of the category hierarchy.

The key insight: a flat confidence of 0.85 conflates “definitely Payment Card Number” with “definitely some kind of Payment Information but unsure which sub-type.” Belief intervals expose this distinction. When \(Bel(\text{PaymentCardData}) = 0.6\) but \(Pl(\text{PaymentCardData}) = 0.95\), the gap signals that evidence supports the broader category but is ambiguous at the leaf level — a meaningful signal for downstream consumers and human reviewers.

Theoretical Foundation

Dempster-Shafer Theory

DST generalizes Bayesian probability by assigning mass not only to singleton hypotheses but to subsets of the frame of discernment \(\Theta\). A mass function \(m: 2^\Theta \to [0,1]\) satisfies \(m(\emptyset) = 0\) and \(\sum_{A \subseteq \Theta} m(A) = 1\). The belief function \(Bel(A) = \sum_{B \subseteq A} m(B)\) gives a lower bound on the evidence supporting \(A\), while the plausibility function \(Pl(A) = \sum_{B \cap A \neq \emptyset} m(B)\) gives an upper bound. The interval \([Bel(A), Pl(A)]\) represents epistemic uncertainty: what the evidence commits to versus what it does not rule out.

For decision-making, the pignistic probability transform [Smets & Kennes, 1994] redistributes mass from composite focal elements to singletons:

\[BetP(\{x\}) = \sum_{A \ni x} \frac{m(A)}{|A|}\]

This converts belief intervals into point probabilities suitable for classification decisions while preserving uncertainty information in the intervals themselves.

Dempster’s Rule of Combination

Given two independent mass functions \(m_1\) and \(m_2\), Dempster’s rule produces:

\[m_{1 \oplus 2}(A) = \frac{1}{1-K} \sum_{B \cap C = A} m_1(B) \cdot m_2(C)\]

where \(K = \sum_{B \cap C = \emptyset} m_1(B) \cdot m_2(C)\) measures conflict between sources. High conflict (\(K > 0.2\)) signals that sources disagree — a diagnostic the flat-confidence approach discards entirely.

Closed-World Assumption

The implementation enforces \(m(\emptyset) = 0\) (closed-world assumption), appropriate for a classification task with an exhaustive taxonomy. Mass on \(\Theta\) represents ignorance (“one of these categories, but I don’t know which”), not rejection. The Transferable Belief Model’s open-world extension [Smets, 1990] would allow \(m(\emptyset) > 0\) for unknown-category detection — a potential future direction.

Architecture

Restricted Focal Set

For a taxonomy with \(N\) leaves, the power set \(2^N\) is computationally intractable for any non-trivial taxonomy. The implementation uses a restricted focal set [Denoeux, 2008] containing only semantically meaningful subsets:

This exploits the category hierarchy: rather than tracking arbitrary subsets, focal elements correspond to nodes in the taxonomy tree. An internal node like IdentityInformation (0010) maps to the focal element \(\{0011, 0012, 0013\}\) — the set of its descendant leaves.

Complexity

Each Dempster combination computes pairwise intersections over focal elements. With \(F\) focal elements and \(S\) sources requiring \(S-1\) combinations:

\[\text{Cost per column} = (S-1) \times F^2\]

With 5 sources and \(F \approx 53\) (SIGDG taxonomy), this is ~11,200 operations per column — negligible compared to the ~50ms sentence-transformer encode step. Even for larger taxonomies with hundreds of focal elements, DST overhead remains sub-millisecond.

Mass Function Converters

Each evidence source implements a converter from raw signal to BeliefAssignment (mass function). All converters produce valid mass functions: non-negative masses summing to 1.0, with a fraction allocated to \(\Theta\) representing the source’s inherent uncertainty.

Cosine Similarity → Mass

cosine_to_mass(similarities, frame, discount=0.3)

Applies softmax to cosine similarities across all leaf categories, then discounts by allocating 30% of mass to \(\Theta\):

\[m(\{c_i\}) = \text{softmax}(s_i) \times (1 - d), \quad m(\Theta) = d\]

The softmax transform converts raw cosine similarities (which may be negative or arbitrarily scaled) into a probability distribution. The discount parameter \(d\) controls how much trust is placed in cosine similarity as a standalone signal.

CatBoost Probability → Mass

catboost_to_mass(proba, frame, virtual_ensembles_variance=None)

Maps predict_proba() output directly to singleton masses. When CatBoost’s virtual ensembles provide per-class variance estimates, high variance increases the discount (more mass to \(\Theta\)):

\[d = \min(0.5,\ 0.1 + \bar{\sigma}^2 \times 1.6)\]

This adaptive discounting means CatBoost contributes less evidence when its own uncertainty is high — a principled integration of aleatoric uncertainty from the model into the epistemic uncertainty framework.

Pattern Detection → Mass

pattern_to_mass(pattern_signals, frame, pattern_category_map=None)

Binary pattern signals (email regex, SSN regex, credit card regex, etc.) map to known category codes. When patterns are detected, mass is concentrated on matched categories with high confidence (0.9) because regex patterns have near-zero false positive rates:

When no patterns are detected, the function returns a vacuous mass function (all mass on \(\Theta\)), contributing no evidence rather than misleading evidence.

Name Match → Mass

name_match_to_mass(column_name, frame, category_set)

Replaces the previous additive boost heuristic with a proper evidence source. Three matching tiers:

This formalization means name matching no longer inflates confidence scores. Instead, its evidence is combined with other sources via Dempster’s rule, where agreement reinforces and disagreement raises the conflict diagnostic.

SVM Short-Text Classification → Mass

svm_to_mass(proba, frame, discount=0.20)

Converts calibrated probability estimates from a TF-IDF + LinearSVC classifier into a mass function. The SVM operates on sparse lexical features — character n-grams (3–6) and word bigrams — making it architecturally independent from the dense sentence-transformer embedding shared by the cosine and CatBoost sources. This directly addresses the source independence concern (R-01): correlation between TF-IDF features and dense embeddings is measurably lower than the correlation between two dense-embedding-derived sources.

The classifier uses scikit-learn’s CalibratedClassifierCV (Platt scaling) to produce well-calibrated probability estimates from the LinearSVC decision function. These probabilities map directly to singleton masses:

\[m(\{c_i\}) = p_i \times (1 - d), \quad m(\Theta) = d\]

The default discount (\(d = 0.20\)) is lower than cosine (0.30) because Platt-scaled SVM probabilities tend to be well-concentrated on the correct class for short-text classification. Like other sources, when the SVM has no meaningful prediction, it can be omitted entirely — classify() accepts svm_proba as an optional parameter, and the combination proceeds with only the non-vacuous sources.

The SVM source is always active when training data is available. The main pipeline (build_sigint_embeddings.py) trains SVM inline on synthetic data before the classification loop, making it the default 5th evidence source. Standalone accuracy: 84.6% on the meta-tagging evaluation set. A validation script (scripts/svm_pilot.py) measures source correlation and the marginal impact on fused belief intervals and conflict \(K\).

Training/evaluation text format. The SVM is trained on short text (name | type | values) and must receive the same format at evaluation time. The EmbeddingClassifier._build_svm_text() method ensures format consistency between training and inference, regardless of the full embedding text used by other evidence sources.

Cross-Benchmark Analysis: Cosine Reliability Regimes

Cross-benchmark experiments reveal that cosine similarity evidence is not uniformly helpful — it ranges from near-perfect (99.4% on semantically named columns) to destructive (1.6% on generic names, where it adds pure conflict to CatBoost’s 81.6% accuracy).

Empirical Evidence

On GitTables (2517 columns, all generic names): CatBoost standalone achieves 81.6%, but DST fusion with cosine drops to 71.4%. Mean Dempster conflict \(K = 0.65\), with 100% of columns exceeding \(K > 0.5\). Cosine evidence is near-random and creates systematic conflict.

On the SIGDG evaluation set (mixed semantic and opaque names): cosine achieves 99.4% on semantically named columns but only 8.0% on opaque-name columns. CatBoost adds value precisely where cosine fails (29.7% on opaque columns). The union ceiling of both methods reaches 66.0% — 12 points above either alone.

Three Regimes

The current implementation uses fixed discounts (0.3 for cosine, 0.15 for CatBoost). The regime analysis motivates a planned refinement: confidence-gated adaptive discounting, where the cosine discount increases when cosine confidence is low and decreases when it is high. This is tracked as part of the calibration experiment (R-02) in the research roadmap.

The regime structure also informs the source independence analysis — cosine and CatBoost share the embedding space, but in different regimes their contributions are nearly orthogonal: cosine dominates on semantic names (where CatBoost is redundant) and CatBoost dominates on opaque names (where cosine is noise).

See Heuristic Elucidation for the full cross-benchmark analysis.

Hierarchical Classification Output

classify_dst() returns a HierarchicalClassification that provides belief intervals at every level of the taxonomy:

result = classifier.classify_dst(sample, frame, category_set)

# Leaf-level belief interval
bel, pl = result.interval_at("0085")   # TaxIdentifier
# → (0.72, 0.91) — strong evidence, moderate uncertainty

# Parent-level belief interval
bel, pl = result.interval_at("0011")   # GovernmentIdentifier
# → (0.72, 0.95) — higher plausibility at parent level

# Root-level belief interval
bel, pl = result.interval_at("0010")   # IdentityInformation
# → (0.72, 0.98) — near-certain at this level

# Diagnostics
result.uncertainty_gap    # Pl - Bel for predicted leaf
result.conflict           # Dempster conflict K
result.needs_clarification  # gap > 0.3 or K > 0.2

Evidence String Format

When DST is active, the evidence string encodes source contributions and the belief interval for the predicted category:

dst(cosine=0.621, catboost=0.834, patterns=0.900) → TaxIdentifier [Bel=0.72, Pl=0.91, K=0.05]

Parquet Output Columns

The pipeline always includes these DST columns in the parquet output:

Relationship to Prior Pipeline

The classify() method is unchanged — existing callers see no difference. classify_dst() is a parallel path that reuses the same embedding infrastructure but wraps evidence in mass functions before combining:

Implementation

Test Coverage

86 tests cover the DST layer (400 total across the sigint package):

Combination Rule Choice & Limitations

Why Classical Dempster

The implementation uses Dempster’s conjunctive rule with normalization (the orthogonal sum) for several engineering reasons:

1. Associativity and commutativity. The combination order does not affect the result. Sources can be added or removed without restructuring the pipeline. combine_multiple() applies left-to-right sequential combination, but the final mass function is order-independent.
2. Traceability. Every combined mass can be decomposed back to source contributions. The evidence string records per-source mass summaries alongside \([Bel, Pl, K]\), making audit straightforward.
3. Conflict as diagnostic. Dempster’s \(K\) is not a flaw — it is a feature. High conflict between sources is a meaningful signal that something is wrong (ambiguous column, mismatched evidence). The needs_clarification flag (\(K > 0.2\) or gap \(> 0.3\)) routes uncertain columns to human review rather than suppressing disagreement.
4. Restricted focal set efficiency. With ~53 focal elements and 3 pairwise combinations, the \(O(F^2)\) cost per column is negligible. More complex rules (e.g., PCR6) would add implementation complexity without proportional benefit at this scale.

Known Limitations

High-conflict sensitivity (Zadeh’s paradox). Classical Dempster with normalization concentrates mass on the intersection even when conflict is extreme. Zadeh (1986) demonstrated that when two sources assign mass to disjoint singletons, the normalized result assigns all mass to a third category with negligible original support. In our setting, this scenario is mitigated by three factors: (a) all four sources operate over the same embedding-derived information, making fully disjoint evidence unlikely in practice; (b) pattern and name-match sources return vacuous mass functions when they have no signal, contributing zero conflict rather than misleading evidence; (c) the \(K > 0.2\) flag identifies problematic cases for human review. However, no formal fallback rule is triggered — this is a passive diagnostic, not an active conflict resolution mechanism.

Source independence. Dempster’s rule assumes independent evidence sources. Cosine similarity and CatBoost both consume the same sentence-transformer embedding vector — they are not independent. Name matching partially overlaps with cosine (column name is part of embedding text). The cross-benchmark regime analysis suggests their contributions are functionally orthogonal in most cases (cosine dominates on semantic names, CatBoost on opaque names), but this is an empirical observation, not a theoretical guarantee. The optional SVM source directly addresses this concern: it operates on sparse TF-IDF features (character and word n-grams) with no dependency on the sentence-transformer embedding, providing a genuinely independent evidence channel. Additional options under consideration include merging cosine and CatBoost into a single source, or adopting a cautious rule that does not require independence (Denoeux, 2008). See R-01: Source Independence Analysis.

Heuristic discounting. The fixed discount constants (cosine: 0.30, CatBoost: 0.15, pattern: 0.10, name match: 0.30/0.50/0.70) were set by engineering judgment. No sensitivity analysis, grid search, or Bayesian optimization has been published. CatBoost’s variance-adaptive discount is the one exception — it uses virtual ensemble variance to modulate trust. A systematic calibration experiment (R-02) is planned: sweep constants against held-out data, measure accuracy and Expected Calibration Error (ECE), and extract optimized values with confidence intervals.

Hierarchical propagation. Belief and plausibility at internal taxonomy nodes are computed by standard summation over subsets (\(Bel(A) = \sum_{B \subseteq A} m(B)\)), which is correct for arbitrary focal sets. The restricted focal set — singletons, internal nodes, confusable pairs, \(\Theta\) — means these sums involve only the pre-computed focal elements that are subsets of a given node’s descendant set. This is not optimized via dedicated hierarchical algorithms (e.g., Shafer-Logan or evidential network propagation). For the current focal set size (~53 elements), brute-force summation is adequate. Larger taxonomies or dense confusable-pair sets would benefit from lattice-based optimization.

Closed-world assumption. The implementation enforces \(m(\emptyset) = 0\), appropriate for classification against an exhaustive taxonomy. Mass on \(\Theta\) represents “one of these categories, but I don’t know which” — not “none of the above.” The Transferable Belief Model’s open-world extension (Smets, 1990) would allow \(m(\emptyset) > 0\) for unknown-category detection. This is a potential extension for schema-drift scenarios where new column types appear that do not map to any SIGDG category.

Decision mapping. The pignistic transform (BetP) always selects a leaf singleton for the classification decision. When evidence supports a parent node but is ambiguous among its children, the classifier still commits to a specific leaf — even when \(Bel(\text{leaf}) \ll Bel(\text{parent})\). A cautious classification approach following Denoeux & Zouhal (2001) would return the deepest hierarchy node where \(Bel(A) > \tau\), preserving the hierarchy’s value. See R-05: Cautious Hierarchical Classification.

Alternative Rules Considered

The modular architecture (belief.py + mass_functions.py) is designed to allow rule substitution — dempster_combine() can be swapped for any pairwise combination function with the same signature (BeliefAssignment, BeliefAssignment) → (BeliefAssignment, float). A planned branch will benchmark Murphy averaging and Yager’s rule against classical Dempster on high-\(K\) synthetic columns.

References

• Dubois, D. & Prade, H. (1988). Representation and combination of uncertainty with belief functions and possibility measures. Computational Intelligence, 4(3), 244-264.
• Denoeux, T. (2008). Conjunctive and disjunctive combination of belief functions induced by nondistinct bodies of evidence. Artificial Intelligence, 172(2-3), 234-264.
• Denoeux, T. & Zouhal, L.M. (2001). Handling possibilistic labels in pattern classification using evidential reasoning. Fuzzy Sets and Systems, 122(3), 409-424.
• Murphy, C.K. (2000). Combining belief functions when evidence conflicts. Decision Support Systems, 29(1), 1-9.
• Sentz, K. & Ferson, S. (2002). Combination of Evidence in Dempster-Shafer Theory. Sandia National Laboratories, SAND2002-0835.
• Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press.
• Smarandache, F. & Dezert, J. (2005). Information fusion based on new proportional conflict redistribution rules. Proceedings of Fusion 2005.
• Smets, P. (1990). The combination of evidence in the Transferable Belief Model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(5), 447-458.
• Smets, P. & Kennes, R. (1994). The Transferable Belief Model. Artificial Intelligence, 66(2), 191-234.
• Yager, R.R. (1987). On the Dempster-Shafer framework and new combination rules. Information Sciences, 41(2), 93-137.
• Zadeh, L.A. (1986). A simple view of the Dempster-Shafer theory of evidence and its implication for the rule of combination. AI Magazine, 7(2), 85-90.

Bootstrap Agent

The bootstrap agent is an LLM-driven pipeline for classifying novel databases when no ground truth exists. It uses DST conflict K as a convergence signal between LLM classifications and ML pipeline predictions, producing ground truth that feeds directly into the self-training pipeline.

Problem

When a novel database arrives — real tables with no codes in column names, no label columns, no pre-existing ground truth — the ML pipeline has no training signal. Manual classification of thousands of columns is infeasible. The bootstrap agent solves this cold-start problem by using an open-weights LLM as a zero-shot classifier, validating outputs through 5-source DST fusion, and iteratively refining disagreements until convergence.

Architecture

Three-Phase Architecture

Phase 1: LLM Sweep

The LLM classifies all columns in a single pass using table-aware batching. Each batch prompt includes:

• Table name and discovered data elements for context
• Column metadata: name, type, sample values, sibling columns, DE membership
• Full taxonomy as a markdown table (code, label, description)
• Instructions to return a JSON array with primary classification, confidence, evidence, and up to 3 alternatives

The alternatives field is critical — it feeds directly into llm_to_mass() where the primary prediction and alternatives are converted into a Dempster-Shafer mass function with calibrated uncertainty.

Batch mechanics:

Phase 2: ML Validation

After LLM labels are assigned, the ML pipeline runs as a validator. It produces independent evidence from 5 orthogonal sources and combines them via Dempster’s rule. The key output is conflict K — a rigorous measure of how much the sources disagree.

A column is flagged for revisit when both conditions hold:

1. The LLM label differs from the ML prediction: llm_code != ml_code
2. The DST conflict exceeds the threshold: K > k_threshold (default 0.2)

Phase 3: Targeted Revisit

When disagreements exist, the agent re-sends only contentious columns to the LLM with enriched context from the ML validation. The convergence loop iterates up to max_iterations (default 5) times with three exit conditions: all resolved, budget exhausted, or mean K below threshold.

Schema Discovery & Data Elements

Before the LLM sees any columns, the system performs structural analysis to discover Data Elements — composite semantic concepts that span columns across one or more tables.

Data Element Pattern Library

11 well-known patterns across 7 governance domains:

Table-Aware Batching

When table_aware_batching = true (the default), columns are grouped by source table instead of flat slices. This ensures the LLM prompt includes a ## Table: payments header and lists discovered data elements, giving the model coherent schema context.

FK Hint Detection

Columns named <entity>_id (e.g., instrument_id) are matched against table names (instruments) with singular/plural handling, identifying cross-table relationships that inform data element membership.

Post-Classification DE Refinement

After bootstrap classification, a second pass discovers data elements that naming patterns missed — using category co-occurrence within tables. Known category combinations (e.g., PAN + Name + Address → PaymentCard) trigger new data element emissions with source="co-occurrence".

Label Propagation

After LLM labels, similar unlabeled columns receive the same label via embedding cosine similarity. This exploits the SAGE finding that column_name accounts for 53% of classification signal — columns with similar names and values likely share the same category.

Propagation rules:

• Only from LLM-direct labels (no cascading)
• Cosine similarity > 0.85 (configurable)
• Skip if ML strongly disagrees (confidence > 0.5 and different code)

After all LLM phases complete, any remaining unlabeled columns with high ML confidence (>= confidence_floor) receive the ML prediction as their label. Every label carries a source_map entry tracking provenance: "llm", "llm_revisit", "propagated", or "ml".

LLM Revisit

When the LLM and ML disagree on a column AND K is high, the column is re-sent to the LLM with enriched context:

### Column 3: account_number (REVISIT)
Type: STRING | Values: ["1234567890", "9876543210"]
Siblings: [customer_name, routing_number, balance]
Data elements: [BankAccount]
ML prediction: BankAccountNumber [Bel=0.35, Pl=0.68, K=0.31]
Confusable: BankAccountNumber / PaymentAccountNumber
Your previous: PaymentAccountNumber (conf=0.65)

This gives the LLM three pieces of information it didn’t have before:

1. ML prediction with belief interval — what the evidence sources think, with calibrated uncertainty
2. Confusable pair — the specific known ambiguity between categories
3. Its own previous answer — a reminder of what it said and at what confidence

LLM Backends

The agent supports three LLM backends via the OpenAI chat completions protocol:

All backends use the same batch prompt format and JSON response parsing. The OpenAI-compatible backend includes lenient parsing (regex fallback) for smaller models.

GLM-4.7 is the default because it is an open-weights model — critical for reproducibility and on-premises deployment in air-gapped environments. The same weights can run via cloud inference APIs or locally via vLLM/TGI, ensuring identical classification behavior regardless of deployment mode. Its reasoning capability (chain-of-thought) improves classification accuracy on ambiguous columns, though it consumes ~80-85% of output tokens on reasoning, requiring llm_max_tokens >= 65536.

LLM as DST Evidence Source

The LLM’s predictions become proper mass functions via llm_to_mass(), not overrides:

The low discount (0.10) reflects the LLM’s high reliability. DST conflict K between LLM and ML sources is a genuine disagreement signal.

Configuration

All bootstrap parameters flow through HOCON (config/base.conf):

Integration with Self-Training

The bootstrap output feeds directly into the self-training pipeline:

# Step 1: Bootstrap — produce initial ground truth
uv run python scripts/bootstrap_classify.py \
    --data-dir ~/data/novel_tables/ \
    --data-elements \
    --output build/bootstrap_gt.json

# Step 2: Self-train — produce production model
uv run python scripts/build_sigint_embeddings.py \
    --data-dir ~/data/novel_tables/ \
    --ground-truth build/bootstrap_gt.json \
    --self-train --auto-generate \
    --output build/sigint_final.parquet

The --data-elements flag enables schema discovery and data element output alongside the ground truth JSON. The self-trained CatBoost model learns to reproduce LLM classifications with 99.4% fidelity — providing deterministic inference, per-item SHAP explanations, calibrated belief intervals, and zero marginal cost at scale.

Key Files

Scenarios Overview

BDD scenarios (behave) cover the hub and the in-tree classification pipeline. Active domains have passing coverage; backlog domains document later work.

Active Domains

Classification Pipeline (Tier 0 — 40 scenarios)

Offline tests that validate the sigint classification engine without infrastructure dependencies:

Platform Health (Tier 1 — 22 scenarios)

Infrastructure health checks that validate each service in the devenv stack:

Integration & Tagging (Tier 1 — 12 scenarios)

Cross-component integration validating the Impala→Kudu→Atlas→sigint pipeline:

Planned Enhancement

Data Lifecycle (Tier 1 — 5 scenarios, not yet passing)

The hot→warm data lifecycle is the keystone capability of the stack: Kudu for upsert-heavy hot-tier ingest, Iceberg for warm-tier analytics, and Impala providing transparent queries across both tiers. See Roadmap.

Backlog Domains

The following scenario domains are archived in features_archive/ as future specifications. They document capabilities that will be revisited once the core platform (query stack, metadata governance, classification pipeline) is fully validated.

Feature Set A: Agent + Visualization (S01–S03)

Archived features: features_archive/agent/visualization.feature, extension.feature, evolution.feature

Feature Set B: Analytics (S04–S06)

Archived features: features_archive/analytics/otel_investigation.feature, cybersec_investigation.feature, streaming_ontology.feature

Also archived: features_archive/grpc_engine.feature — gRPC engine health and connectivity checks.

These scenarios remain valid specifications. When the gRPC engine, Dask compute layer, or streaming pipeline is implemented, the corresponding features can be moved from features_archive/ back to features/ and connected to step definitions.

Current Status

See Test Infrastructure for the tier system, how to run scenarios, and the BDD framework architecture.

Test Infrastructure

BDD scenarios are implemented with the behave framework. Tests are organized into tiers based on what they validate. Tier-0 classification scenarios run offline (no infrastructure). Tier-1 integration scenarios require the full devenv stack (devenv up).

Test Tiers

Running Tests

# ── Tier 0: Classification (no infrastructure needed) ──────────
uv run behave features/classification/ --no-capture

# ── Tier 1: Integration (requires devenv up) ───────────────────
uv run behave features/platform/ features/tagging/ --no-capture

# ── All tiers ──────────────────────────────────────────────────
uv run behave --no-capture

# ── By tag ─────────────────────────────────────────────────────
uv run behave --tags=@tier-0 --no-capture
uv run behave --tags=@tier-1 --no-capture
uv run behave --tags=@health --no-capture
uv run behave --tags=@integration --no-capture

# ── Specific feature ──────────────────────────────────────────
uv run behave features/platform/integration_meta_tagging.feature --no-capture
uv run behave features/classification/evidence_fusion.feature --no-capture

# ── Dry run (parse only) ─────────────────────────────────────
uv run behave --dry-run

# ── Air-gap mode (no external network calls) ─────────────────
HF_HUB_OFFLINE=1 uv run behave --no-capture

Air-Gap Testing

The devenv shell sets HF_HUB_OFFLINE=1 and preserves host HF_HOME / HF_HUB_CACHE / SENTENCE_TRANSFORMERS_HOME (lab RAID under /raid/cache/*), falling back to build/models only if no shared cache exists. Models must already be on the shared cache (or cached once via just cache-models):

just cache-models                        # Download model once
# or: devenv tasks run sigint:cache-models

# All subsequent runs are fully offline
uv run behave --no-capture               # Zero external network calls

This ensures BDD scenarios validate the same execution path used in air-gap deployments. See Air-Gap (Zarf) for packaging model artifacts into Zarf archives.

Feature Tags

Features use hierarchical tags for domain, tier, and scope:

@platform @integration @catalog-sync   # Domain tags
@tier-1                                 # Tier tag (controls which scenarios run)

Domain tags: @platform, @integration, @catalog-sync, @meta-tagging, @health, @classification, @tagging, @pipeline

Scenario Coverage

Tier 0 — Classification Pipeline (40 scenarios)

Tier 1 — Component Health (22 scenarios)

Tier 1 — Cross-Component Integration (12 scenarios)

Catalog sync scenarios:

1. Impala CREATE TABLE registers in Kudu
2. Impala CRUD operations work end-to-end on Kudu
3. Atlas discovers Impala-managed Kudu tables (via catalog bridge)
4. Atlas entity lifecycle follows Impala DDL (create → register → delete)

Meta-tagging scenarios: 5. Tagger creates SIGDG classification types in Atlas 6. Tagger classifies columns and applies tags (live mode) 7. Tagger classifies columns without writing (dry-run mode) 8. Apply classification to a specific column (column-level tagging) 9. Search for entities by classification 10. Tagger setup + dry-run pipeline validates without Atlas writes

Pipeline tagging scenarios: 11. Full Tagger pipeline: sample → classify → tag on a live Impala/Kudu table 12. Dry-run pipeline: classify without Atlas writes, verify report output

Planned — Data Lifecycle (not yet passing)

Total

Feature Organization

features/
├── environment.py                              # Stack health + cleanup hooks
├── conftest.py                                 # Pytest bridge (optional)
├── classification/                             # Tier-0: offline classification tests
│   ├── steps/
│   │   └── classification_steps.py             # All classification step definitions
│   ├── benchmark_gittables.feature
│   ├── bespoke_dataset.feature
│   ├── config_lifecycle.feature
│   ├── embedding_classification.feature
│   ├── evidence_fusion.feature
│   ├── feature_extraction.feature
│   ├── taxonomy_management.feature
│   └── vocabulary_mapping.feature
├── platform/                                   # Tier-1: health + integration
│   ├── steps/
│   │   ├── __init__.py                         # Cross-domain: imports tagging steps
│   │   ├── helpers.py                          # Connections, Atlas API, catalog bridge
│   │   ├── health_steps.py                     # Health check steps (idempotent)
│   │   └── integration_steps.py                # Catalog sync + meta-tagging steps
│   ├── health_atlas.feature
│   ├── health_impala.feature
│   ├── health_kerberos.feature
│   ├── health_kudu.feature
│   ├── health_postgres.feature
│   ├── integration_catalog_sync.feature
│   ├── integration_meta_tagging.feature
│   └── data_lifecycle.feature                  # Planned: Kudu→Iceberg lifecycle
├── tagging/                                    # Tier-1: end-to-end tagging pipeline
│   ├── steps/
│   │   ├── __init__.py                         # Cross-domain: imports integration steps
│   │   └── tagging_steps.py                    # Pipeline tagging steps
│   └── pipeline_tagging.feature
└── steps/
    └── steps.py                                # Re-exports for behave discovery

Cross-Domain Step Discovery

Behave discovers step definitions from the feature file’s parent steps/ directory. When a feature in features/platform/ uses steps defined in features/tagging/steps/, behave won’t find them automatically.

Each domain’s steps/__init__.py imports steps from sibling domains:

• features/platform/steps/__init__.py → imports features.tagging.steps.tagging_steps
• features/tagging/steps/__init__.py → imports features.platform.steps.integration_steps

This ensures steps are discoverable regardless of which domain’s feature file is being run.

Config-Driven BDD

All integration step helpers (features/platform/steps/helpers.py) load connection parameters from the HOCON config (config/base.conf) via sigint.config.load_config(). This ensures BDD tests use the same configuration as the application code — no hardcoded hosts, ports, or credentials that could drift.

from sigint.config import load_config

_CFG = load_config()          # HOCON base.conf + env var overrides

def impala_conn():
    return impala_connect(host=_CFG.impala_host, port=_CFG.impala_port, ...)

def atlas_api(path, method="GET", **kwargs):
    url = f"{_CFG.atlas_url}/api/atlas/v2{path}"
    auth = (_CFG.atlas_user, _CFG.atlas_password)
    ...

Catalog Bridge

The tier-1 integration tests use a Python catalog bridge function (register_impala_table_in_atlas in helpers.py) that registers Impala-managed Kudu tables in Atlas without requiring Kafka or the Atlas hook infrastructure:

1. Runs DESCRIBE {table} on Impala to get column metadata
2. Builds rdbms_* entities with temporary GUIDs (Atlas 2000-RDBMS model, Aegir-aligned)
3. POSTs to Atlas POST /v2/entity/bulk — single atomic call
4. Atlas resolves rdbms_table_columns COMPOSITION and assigns real GUIDs

This validates the Atlas entity contract for governed Kudu tables and enables classification and search tests without Java hook changes.

qualifiedName Convention

Product registration uses rdbms_* only (not hive_*). Physical Kudu names remain separate (kudu_table FDW option / impala::db.table). See Roadmap: Entity Type Evolution.

After-Scenario Cleanup

environment.py runs best-effort cleanup after each scenario:

• Deletes Atlas entities created by bridge registration (columns first, then table)
• Deletes Atlas entities tagged during classification tests
• Drops Impala tables created during the scenario

Health check steps are idempotent — they verify resources exist rather than requiring a clean state. This allows health scenarios to pass on repeated runs without cleanup.

Unit Tests

In addition to BDD scenarios, the sigint package has comprehensive pytest coverage:

uv run pytest tests/sigint/ -v          # 363 tests

These test individual modules (features, belief, mass functions, config, classifier, etc.) in isolation. The BDD scenarios validate end-to-end behavior across the full stack.

S01: Agent-Mediated Visualization

Feature Set A — features/agent/visualization.feature

Intent

A user interacts with the agent through the web terminal. The agent directs HoloViews/Datashader/Dask visualizations so the user can explore data interactively with resolution autoscaling and persona-appropriate views.

Scenarios

User issues command via WASM terminal

Tier 2 @engine-required

The user connects to the web terminal and submits an instruction. The instruction is delivered to the engine via gRPC and the agent acknowledges the request.

This validates the basic instruction path: terminal → gRPC → engine → acknowledgement.

Agent recomputes visualization on interaction

Tier 3 @viz-required

Given an active HoloViews session, user interaction with a visualization component triggers Dask to recompute the view in parallel. Datashader rasterizes the updated result and the visualization streams to the web client.

Resolution autoscaling on zoom

Tier 3 @viz-required

When the user changes the zoom level on a Datashader-rendered view, the view recomputes at the new resolution. Detail increases as the viewport narrows — no raw data is transferred to the client.

Persona-appropriate view (Scenario Outline)

Tier 3 @viz-required

The agent adapts visualization conventions based on the selected persona. Each persona foregrounds relevant metrics:

Capabilities Addressed

From the draft overview:

• Single data product for temporal, spatial, and spectral analysis
• Data resolution autoscaling with active recomputation
• Single data product with visualization for multiple personas
• Base line agent flows
• MLOps for feature extraction, correlation, outlier detection, signals evaluation

S02: Algorithm Extension Lifecycle

Feature Set A — features/agent/extension.feature

Intent

An algorithm developer independently develops analysis code leveraging Dask, packages it as a platform extension, and deploys it so the agent can invoke it in distributed compute contexts.

Scenarios

Package algorithm as extension

Tier 0 (pure, no services required)

The developer has a Dask-based analysis module. They package it as a platform extension with valid metadata and a distributable archive. This validates the packaging format without requiring any running services.

Deploy extension to running platform

Tier 2 @engine-required

A packaged extension is deployed to the platform. The agent registers the new capability and it appears in the capability inventory.

Agent invokes extension in compute context

Tier 3 @engine-required @viz-required

The agent selects a deployed extension for an analysis task. The extension executes within the Dask distributed context and results are surfaced through HoloViews components.

Draft Scenario Mapping

From the draft overview (Scenario 02):

• Algo dev identifies analysis needs → Package algorithm as extension
• Independently develops code leveraging Dask or Ray → Package algorithm as extension
• Packages and deploys as platform extension → Deploy extension to running platform
• Loads, deploys, tests, and manages the extension → Deploy extension to running platform
• Agent includes algorithm as optional capability → Agent invokes extension in compute context

S03: Agent Self-Improvement

Feature Set A — features/agent/evolution.feature

Intent

An evaluation developer defines high-level performance objectives with discrete outcomes. The agent engages in self-improvement cycles to satisfy quantitative improvements, producing enhanced workflow templates available for download and redeployment.

Scenarios

Define performance objective

Tier 2 @engine-required

The developer specifies a quantitative performance target. The agent receives the objective, establishes a baseline measurement, and begins an improvement iteration.

Agent completes improvement cycle

Tier 2 @engine-required

When an improvement iteration completes, quantitative results are recorded and the agent reports whether the target was met. This validates the feedback loop between objective definition and measured outcome.

Export enhanced workflow template

Tier 2 @engine-required

After a successful improvement cycle, the developer requests the enhanced workflow. A workflow template is generated and made available for download and redeployment.

Draft Scenario Mapping

From the draft overview (Scenario 03):

• Eval dev identifies high-level performance objectives with discrete outcomes → Define performance objective
• Agent engages in self-improvement to satisfy quantitative improvements → Agent completes improvement cycle
• Enhanced agent workflow templates available for download and redeployment → Export enhanced workflow template
• Optional direct integration with GitHub/GitLab SDLC → Future extension

S04: OTel Root Cause Analysis

Feature Set A — features/analytics/otel_investigation.feature

Intent

An OTel analyst performing RCA correlates observed degradation with telemetry signals and infrastructure changes, engaging the agent to capture evidence of correlation and compose multi-system views.

Scenarios

Correlate degradation with telemetry signals

Tier 3 @engine-required @viz-required

Given access to OTel telemetry data, the analyst identifies an observed performance degradation. The agent correlates it with infrastructure change events and presents a timeline of contributing factors.

Compose multi-system RCA view

Tier 3 @engine-required @viz-required

Given correlated signals from multiple systems, the analyst requests a root cause analysis view. The agent composes a multi-system visualization with evidence of correlation captured in the report.

Draft Scenario Mapping

From the draft overview (Scenario 04):

• OTel analyst performing RCA report generation → Both scenarios
• Correlate observed degradation with telemetry signals with infrastructure changes → Correlate degradation with telemetry signals
• Engage agent to capture evidence of correlation and likely root cause → Compose multi-system RCA view
• Compose multi-system views → Compose multi-system RCA view

S05: Cybersecurity Investigation

Feature Set A — features/analytics/cybersec_investigation.feature

Intent

A KQL/SPL cybersecurity analyst investigates malicious behavior on their network/systems. They leverage AWS and systems logs alongside existing point solutions to triage threats, capture symptoms and representative signals, and validate novel detection logic.

Scenarios

Triage with AWS and system logs

Tier 3 @engine-required @viz-required

Given access to AWS logs and system telemetry, the analyst describes suspicious network behavior. The agent queries relevant log sources and surfaces symptoms and representative signals.

Secondary correlative investigation

Tier 3 @engine-required @viz-required

After primary indicators of compromise are identified, the analyst requests secondary system correlation. The agent correlates with OTel traces and eBPF data, consults historical binary fingerprints, and illustrates environmental conditions.

Validate novel detection logic

Tier 3 @engine-required @viz-required

Given a proposed detection rule, the agent evaluates it against historical data. It confirms whether the rule produces actionable views and reports false positive rates.

Draft Scenario Mapping

From the draft overview (Scenario 05):

• KQL/SPL analyst investigating malicious behavior → All scenarios
• Leverage AWS and systems logs alongside existing point solutions → Triage with AWS and system logs
• Provide insight into primary behavior through secondary correlative investigation → Secondary correlative investigation
• Illustrate system/environment conditions in support of findings → Secondary correlative investigation
• Confirm novel detection logic provides actionable views → Validate novel detection logic

S06: Streaming Ontology and Feature Discovery

Feature Set B — features/analytics/streaming_ontology.feature

Intent

A data scientist performing click stream analysis intends to inform persona-oriented strategy decisions. High-velocity wide-schema records yield ontology-grounded feature patterns. The analyst observes and directs feature growth via agent-mediated sketch-based algorithms, reviews proposed software changes, and validates persona-grounded visit behavior views.

Scenarios

Ingest wide-schema stream records

Tier 3 @engine-required @viz-required

A high-volume click stream source is configured. As records begin arriving, the agent identifies ontology-grounded feature patterns and pattern counts accumulate over the stream.

Human-in-the-loop feature direction

Tier 3 @engine-required @viz-required

As feature patterns are being discovered, the analyst observes and directs pattern growth via the agent. The agent adjusts sketch-based algorithms accordingly and the feature set reflects the analyst’s guidance.

Agent proposes software changes

Tier 3 @engine-required @viz-required

Once directed feature patterns have stabilized, the agent identifies implementation requirements and proposes source changes via the SDLC integration. The analyst can review the proposed changes before acceptance.

Validate persona-grounded visit behaviors

Tier 3 @engine-required @viz-required

After processing a stream accumulation, the analyst requests a persona-grounded behavior view. The sketch algorithm produces actionable visit behavior views differentiated by persona segment.

Draft Scenario Mapping

From the draft overview (Scenario 06):

• Data Scientist performing click stream analysis → All scenarios
• High volume/velocity wide-schema records yield ontology-grounded feature patterns → Ingest wide-schema stream records
• Observe and direct feature pattern growth via agent-mediated sketch-based algorithms → Human-in-the-loop feature direction
• Review proposed software changes (agent-driven SDLC) → Agent proposes software changes
• Confirm sketch algo provides actionable views of persona-grounded visit behaviors → Validate persona-grounded visit behaviors

Infrastructure Overview

The infrastructure stack is layered: OpenTofu provisions cloud resources, Ansible configures them, Zarf packages for air-gap, Tilt enables dev iteration, and OPA policies provide guardrails.

Layers

Deployment Flow

For air-gap environments, the Ansible flow uses the zarf-deploy role instead of individual service roles.

Ansible Roles

Playbooks

Task Reference

Local Cluster (k8s:*)

AWS Cloud (aws:*)

OpenTofu (AWS)

AWS infrastructure is provisioned with OpenTofu (open-source Terraform fork) in infra/aws/tofu/.

Resources

Network Architecture

Air-Gap Mode

Setting airgap_mode = true in terraform.tfvars:

• Removes the NAT gateway (no outbound internet from private subnet)
• S3 access preserved via VPC gateway endpoint
• All software must be pre-staged via Zarf packages

Variables

Key variables in variables.tf:

Usage

cd infra/aws/tofu

# Initialize
tofu init

# Plan with policy check
tofu plan -out=plan.tfplan
conftest test plan.tfplan -p ../../policy/tofu/

# Apply
tofu apply plan.tfplan

# Outputs for Ansible inventory
tofu output -json

Ansible Roles

Configuration management for RKE2 clusters and application services lives in infra/aws/ansible/.

Directory Structure

infra/aws/ansible/
├── group_vars/all.yml        # Shared variables
├── inventory/
│   └── hosts.example         # Template (tofu-generated in practice)
├── playbooks/
│   ├── site.yml              # Full stack deployment
│   ├── dask-only.yml         # Standalone Dask
│   ├── airgap-deploy.yml     # Zarf air-gap
│   ├── teardown.yml          # Clean workloads
│   └── validate.yml          # Health checks
└── roles/
    ├── common/               # Base node setup
    ├── rke2-server/          # Control plane
    ├── rke2-agent/           # Workers
    ├── dask/                 # Dask operator + cluster
    ├── jupyterhub/           # JupyterHub
    ├── signals-engine/       # gRPC engine
    ├── cloudflare-tunnel/    # Ingress
    └── zarf-deploy/          # Air-gap orchestrator

Roles

common

Prepares Ubuntu 24.04 nodes for RKE2:

• System packages (iptables, open-iscsi, nfs-common)
• Kernel modules (br_netfilter, overlay)
• Sysctl tuning for Dask workloads (vm.max_map_count, somaxconn)
• Swap disabled, data volume formatted and mounted (workers)

rke2-server

Installs RKE2 control plane:

• Downloads and runs official RKE2 installer
• Configures Canal CNI, TLS SANs, disables default ingress
• First server: waits for node token, propagates to all cluster members
• Additional servers: joins existing cluster with token

rke2-agent

Installs RKE2 worker nodes:

• Fetches node token from control plane automatically
• Configures node labels (dask.org/node-type=worker, signals.io/role=compute)
• Joins the cluster

dask

Deploys Dask via Helm:

• Adds Dask Helm repository
• Installs dask-kubernetes-operator
• Deploys DaskCluster CRD with configurable worker replicas, memory limits, and spill-to-disk
• Waits for scheduler pod to be ready

jupyterhub

Deploys JupyterHub via Helm:

• Dummy authenticator for development (password: signals)
• Singleuser image shared with Dask workers
• ClusterIP service (accessed via Cloudflare Tunnel)

signals-engine

Deploys the gRPC engine:

• Namespace, Deployment, and Service
• gRPC health check probes (readiness + liveness)
• Configurable resource limits

cloudflare-tunnel

Deploys cloudflared for Zero Trust ingress:

• Validates tunnel token and ID (from OpenTofu outputs)
• Creates credentials Secret and config ConfigMap
• Routes: Dask dashboard, JupyterHub, Signals Engine
• 2 replicas with health checks

zarf-deploy

4-phase air-gap deployment orchestrator:

1. Stage — copy Zarf binary and packages to control plane
2. Init — create PVs, initialize Zarf registry
3. Deploy — deploy application package with variables
4. Verify — check all components running

Usage

cd infra/aws/ansible

# Full stack deployment
ansible-playbook playbooks/site.yml

# Deploy only Dask
ansible-playbook playbooks/dask-only.yml

# Air-gap deployment
ansible-playbook playbooks/airgap-deploy.yml

# Teardown workloads (keep RKE2)
ansible-playbook playbooks/teardown.yml

# Post-deployment validation
ansible-playbook playbooks/validate.yml

Air-Gap (Zarf)

Zarf enables deployment to disconnected (air-gap) environments by packaging container images, Helm charts, and manifests into a single archive that can be transferred to isolated networks.

Package Definition

The Zarf package (zarf/zarf.yaml) defines the following components:

Variables

Workflow

Build the package

cd zarf

# Download Helm charts to charts/
helm pull dask/dask-kubernetes-operator --version 2024.1.0 -d charts/
helm pull jupyterhub/jupyterhub --version 4.0.0 -d charts/

# Create manifests
# ... (namespace.yaml, dask-cluster.yaml, engine.yaml, etc.)

# Build package
zarf package create --confirm

Deploy to air-gap cluster

Using Ansible:

ansible-playbook playbooks/airgap-deploy.yml

Or manually:

# Stage on control plane
scp zarf signals-360-amd64.tar.zst user@cp:/opt/zarf/

# Initialize (creates internal registry)
zarf init --components=git-server --confirm

# Deploy
zarf package deploy signals-360-amd64.tar.zst \
  --set DASK_WORKER_REPLICAS=4 \
  --set S3_BUCKET=signals-360-data \
  --confirm

Air-Gap Network Requirements

When airgap_mode = true in OpenTofu:

• NAT gateway is removed — no outbound internet from private subnet
• S3 access preserved via VPC gateway endpoint
• Container images served from Zarf’s internal registry (port 31999)
• Git repos served from Zarf’s internal Gitea instance
• ML model artifacts served from local filesystem or S3 via VPC endpoint

ML Model Artifacts

The sigint classification pipeline uses a sentence-transformer model (all-MiniLM-L6-v2, ~80MB) for embedding classification. In air-gap environments, this model must be pre-cached — the pipeline cannot download from HuggingFace at runtime.

Model Cache Bootstrap

# Ensure MiniLM via HF_HOME / SENTENCE_TRANSFORMERS_HOME (RAID preferred)
just cache-models
# or: devenv tasks run sigint:cache-models