Platform Metaflow (YK · Knative · Airflow)

Status: Architecture direction (platform-grade)
Date: 2026-08-11
Scope: Signals-owned ground-up Metaflow capability — not Gaius, not Marquez. Any federated engine that integrates via signals-protocol can consume it as a platform service.
Submodule: components/metaflow → weathership/oss-metaflow branch rch/devenv (custom line, same convention as other ASF/oss components).
References:

• Outerbounds AWS K8s deployment (eks_airflow Terraform path preferred for us)
• Metaflow Airflow scheduling · Outerbounds Airflow ops
• Outerbounds event triggering (stock stack uses Argo Events; we substitute Knative Eventing)

Thesis

Signals provides a platform Metaflow so current and future engines (Gaius, Aegir, Atelier, …) do not each stand up thrashing metadata services on shared RKE2. Engine repos stay free to keep isolated Metaflow for solo devenv; this tree does not modify Gaius or Marquez. Platform work lands here: submodule patches, federation packaging, Airflow + Knative Eventing, YuniKorn admission, signals-ui visibility.

With YuniKorn + Knative Serving already locked as the federation control plane, Metaflow is a first-class platform service:

Stock Outerbounds prefers Argo Workflows when in doubt (full event triggering). We deliberately take the eks_airflow shape and close the event gap with Knative Eventing, not Argo Events / Argo Workflows.

Submodule and customization line

Platform-specific changes to Metaflow itself (Airflow event-bridge hooks, Knative-friendly defaults, YK labels, devenv packaging) go on rch/devenv in that repo — not as one-off patches only in Signals, and not by forking work into Gaius.

Isolation elsewhere (out of scope here)

Other products may still run engine-local Metaflow (Tilt metadata, private PG/S3) for isolated devenv. That pattern is fine for those repos; Signals does not implement or maintain it. Platform Metaflow is the shared SoR when an engine opts into federation via signals-protocol.

Why platform upgrade

Official stack vs Signals adaptation

Outerbounds eks_argo (reference default)
  Metaflow service + S3 + EKS
  Argo Workflows  ←── steps
  Argo Events     ←── event triggering

Outerbounds eks_airflow (our base)
  Metaflow service + S3 + EKS
  Airflow         ←── steps as KubernetesPodOperator DAGs
  @schedule only  ←── stock Metaflow→Airflow (no native event trigger)

Signals platform (target)
  Metaflow service + RustFS + RKE2
  Airflow         ←── production DAGs (schedule + API-triggered runs)
  Knative Eventing←── event fabric (replace Argo Events role)
  YuniKorn        ←── schedule all Metaflow + sentinel pods
  Knative Serving ←── sentinels / scale-to-zero (already federation)

Critical product fact: Metaflow’s first-class event triggering is documented as Argo Workflows only today. Airflow docs state event-based triggering is not supported via Metaflow’s Airflow integration (only @schedule / cron). Gaius already showed that application-level and platform-level triggering is enough for real product capability. We institutionalize that as Knative Eventing → Airflow, rather than forking Metaflow’s Argo-only path.

Target architecture

                    ┌─────────────────────────────────────┐
                    │  Producers                          │
                    │  CLI · MCP · Atlas OL · Sentinels · │
                    │  Engine gRPC · webhooks · cron      │
                    └──────────────┬──────────────────────┘
                                   │ CloudEvents
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  Knative Eventing                   │
                    │  Broker / Channel / Triggers        │
                    │  (types: metaflow.run.v1, …)        │
                    └──────────────┬──────────────────────┘
                                   │ Sink: Airflow REST / API
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  Apache Airflow (on RKE2, YK queue) │
                    │  DAGs from: python flow.py          │
                    │             airflow create dag.py   │
                    │  @schedule + event-triggered runs    │
                    └──────────────┬──────────────────────┘
                                   │ KubernetesPodOperator
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  RKE2 + YuniKorn                    │
                    │  Metaflow @kubernetes step pods     │
                    │  Queues root.{engine,signals,…}     │
                    └──────────────┬──────────────────────┘
                          ┌────────┴────────┐
                          ▼                 ▼
              ┌───────────────────┐  ┌──────────────────┐
              │ Metaflow metadata │  │ RustFS (S3)      │
              │ service + PG      │  │ artifacts/cards  │
              └───────────────────┘  └──────────────────┘
                          │
                          ▼
              ┌───────────────────┐
              │ signals-ui        │
              │ Applications ·    │
              │ Queues · Status   │
              └───────────────────┘

Component notes

Artifact store — RustFS only

Metaflow’s product property is that every run retains an immutable snapshot of code, data, and dependencies. That snapshot is the first Signals data product (signals.metaflow.snapshots). Mapping and ACP brief: Data Products History.

signals.ops.metaflow_store.require_rustfs fails closed if any of those drift. Do not point SYSROOT at /tmp, ~/.metaflow, or a foreign S3.

Metaflow metadata service

• Deploy as platform Deployment/Service (not engine Tilt thrash).
• Postgres: platform instance (signals or dedicated metaflow DB on lab PG 5455 / production SoR) — not a second unmanaged store per engine.
• Expose: ClusterIP + NodePort or signals-ui/proxy for lab (METAFLOW_SERVICE_URL).
• Config profile (shared ~/.metaflowconfig shape):

{
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_SERVICE_URL": "http://metaflow-service.platform.svc:8080",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DATASTORE_SYSROOT_S3": "s3://metaflow/metaflow",
  "METAFLOW_S3_ENDPOINT_URL": "http://rustfs.platform.svc:9010",
  "METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
  "METAFLOW_AIRFLOW_KUBERNETES_CONN_ID": "kubernetes_default"
}

Airflow

• Run on RKE2 under YK (queue root.platform for control-plane services; inference steps use root.internal.inference.*, not a project parent).
• Lab/platform pin: Airflow 3.1.7 (components/airflow chart); cncf.kubernetes provider for KPO.
• Metaflow steps → KubernetesPodOperator only (@kubernetes, not @batch).
• Deploy path: python flow.py --with retry airflow create flow_dag.py then ship DAG into Airflow dags/ (Git-sync or ConfigMap pipeline).
• Time schedules: Metaflow @schedule as today.
• Event runs: Airflow REST POST /api/v1/dags/{dag_id}/dagRuns (or stable equivalent) invoked by Knative SinkBinding / Service — this is the platform “bring Airflow up to event-triggering spec.”

Knative Eventing (not Argo Events)

Example event types (illustrative):

• dev.metaflow.flow.finished — inter-flow chaining (@trigger_on_finish analogue)
• dev.signals.sentinel.activated — sentinel woke; start inspection flow
• dev.gaius.article.curate.requested — product CLI/MCP publish
• dev.atlas.openlineage.runEvent — optional OL-driven reactions

Emitter libraries: small platform helper (replace ArgoEvent.publish) that POSTs CloudEvents to the Broker ingress. Gaius/CLI keep product APIs; they publish platform events instead of talking Argo.

YuniKorn

• All Metaflow task pods and Airflow workers labeled into federation queues.
• signals-ui Queues flow already shows root.* capacity; Applications shows admitted apps (including Metaflow steps once scheduled).
• Avoid default-scheduler side channels for production steps.

Sentinels

• Remain Knative Serving (scale-to-zero).
• Eventing can wake work and start Metaflow DAGs when a sentinel signals readiness or C2 heartbeat thresholds — without conflating Serving and Workflow engines.

Isolation vs platform (policy)

Engines opt into platform by setting Metaflow config profile to platform URLs and deploying DAGs to platform Airflow. Isolation remains for unit hermeticity.

Delivery phases

M0 — Document + guardrails (this page)

• Architecture + non-goals; signals-ui Applications already frames YK workloads.

M1 — Platform Metaflow service (landed)

Host PG (:5455) and RustFS (:9010) are bridged into ns/metaflow via Service + Endpoints (signals-postgres, signals-rustfs).

M2 — Airflow 3 on RKE2 (YK) (landed)

Shape: LocalExecutor, external PG (airflow DB on host :5455 via hostNetwork socat proxy — devenv PG is loopback-only), no in-cluster Bitnami Postgres/Redis. Fernet/JWT/API secrets persist under build/config/airflow-secrets.env. CI DAG signals_ci is live; Metaflow airflow create DAGs ship next (same ConfigMap / dags path). REST: AF3 /api/v2 with JWT from POST /auth/token.

M3 — Knative Eventing bridge (landed)

No Argo. Engines (Gaius/Aegir/…) and Metaflow finish hooks publish CloudEvents to the Broker; Triggers filter by type and invoke Airflow. Zarf agent ignore until Eventing images are packaged into signals-federation.

M4 — Multi-engine consumers

• Gaius/Aegir/Atelier profiles point at platform Metaflow
• signals-ui Applications: show Metaflow run correlation (run id ↔ YK app id)
• OL: keep Atlas SoR; Metaflow cards remain UX, not second lineage SoR

Critical plane membership

Metaflow (with YK, Knative, Airflow, RustFS, and host data services) is part of the uniform critical plane for Signals — enforced by just stack-ready / just signals-ready / signals:stack-ready. See Critical plane.

Non-goals

• Touching Gaius, Marquez, or engine repos from this workstream — engines adopt platform Metaflow on their schedule via config + protocol
• Argo Workflows / Argo Events as SoR — rejected for this platform path
• Nested Metaflow foreaches on Airflow (Airflow limitation) — avoid in platform DAGs or gate on Airflow version + design
• Conditional/recursive steps (Metaflow 2.18+) until Airflow support exists
• Marquez DB as Metaflow metadata (Atlas OL remains lineage SoR)

Open design questions

1. Airflow executor: start LocalExecutor / CeleryExecutor vs full KubernetesExecutor (DAG volume sync cost on single-node lab).
2. Event exactly-once: Broker + Airflow dagRun dag_run_id idempotency keys from CloudEvent id.
3. Namespace layout: one metaflow ns vs per-engine namespaces for pods with YK queue labels only.
4. UI: Metaflow UI (Outerbounds) optional lab validation vs signals-ui only.

Related

• signals-federation Zarf — YK + Knative Serving
• Airflow component stub
• signals-ui Applications — workloads surface
• Gaius Metaflow service (isolation reference)