Data Flow
Data flow through the system connects user interaction, the gRPC engine, compute infrastructure, and data sources.
Instruction Path
Data Sources
Streaming Ingest
High-velocity data (click streams, telemetry) arrives via NiFi and is routed to appropriate storage layers. The engine can direct the agent to observe stream accumulation and discover ontology-grounded feature patterns.
Analytical Storage
| Layer | Engine | Access Pattern |
|---|---|---|
| Graph | PostgreSQL + AGE | Relationship queries, ontology traversal |
| Columnar | Kudu | Low-latency random access, time-series (hot tier) |
| SQL | Impala | Distributed analytical queries (HMS-free) |
| Table Format | Iceberg | Large-scale batch analytics, schema evolution (warm tier) |
| Metadata | Atlas | Data catalog, AI-driven classification |
Object Storage
S3 buckets store raw data, intermediate results, and model artifacts. In air-gap deployments, an S3-compatible endpoint (RustFS or similar) replaces AWS S3.
Response Path
Computed visualizations flow back through the engine to the web client:
- Dask workers complete computation
- Datashader rasterizes the result
- HoloViews composes the view specification
- Engine streams the update over gRPC
- Web client renders the visualization in the top panel