Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Metaflow

Platform Metaflow for Signals — ground-up capability for any federated engine via signals-protocol. Not Gaius-owned; not Marquez.
Submodulecomponents/metaflow
Remotegit@github.com:weathership/oss-metaflow.git
Branchrch/devenv
ArchitecturePlatform Metaflow

Role

ConcernAuthority
Flow authoring / run metadataMetaflow (this submodule + metadata service)
Production DAG scheduleAirflow (components/airflow)
Event / reactive triggerKnative Eventing → Airflow DAG runs
Task podsRKE2 + YuniKorn (@kubernetes)
ArtifactsRustFS (S3)
Lineage SoRAtlas OL (not Marquez DB, not Metaflow cards alone)
Operator UIsignals-ui Applications / Queues

Customization

Platform patches to Metaflow land on rch/devenv in weathership/oss-metaflow (same rch/devenv convention as other ASF/oss submodules). Signals pins that branch via .gitmodules.

git submodule update --init components/metaflow
cd components/metaflow && git checkout rch/devenv && git pull

Status

PhaseState
Submodule pinAdded (rch/devenv)
Platform metadata service on RKE2M1 — just metaflow-platform
Airflow + YKPlanned (M2)
Knative Eventing bridgeM3 — just knative-eventing · CE → Airflow (no Argo)

Lab deploy (M1)

# PG :5455 + RustFS :9010 up (devenv), RKE2 Ready
just metaflow-platform
just metaflow-platform-status
# → curl http://127.0.0.1:30180/ping

mkdir -p ~/.metaflowconfig
cp config/metaflow/platform.json ~/.metaflowconfig/config.json
export METAFLOW_SERVICE_URL=http://127.0.0.1:30180