Bootstrap Agent

The bootstrap agent is an LLM-driven pipeline for classifying novel databases when no ground truth exists. It uses DST conflict K as a convergence signal between LLM classifications and ML pipeline predictions, producing ground truth that feeds directly into the self-training pipeline.

Problem

When a novel database arrives — real tables with no codes in column names, no label columns, no pre-existing ground truth — the ML pipeline has no training signal. Manual classification of thousands of columns is infeasible. The bootstrap agent solves this cold-start problem by using an open-weights LLM as a zero-shot classifier, validating outputs through 5-source DST fusion, and iteratively refining disagreements until convergence.

Architecture

Three-Phase Architecture

Phase 1: LLM Sweep

The LLM classifies all columns in a single pass using table-aware batching. Each batch prompt includes:

• Table name and discovered data elements for context
• Column metadata: name, type, sample values, sibling columns, DE membership
• Full taxonomy as a markdown table (code, label, description)
• Instructions to return a JSON array with primary classification, confidence, evidence, and up to 3 alternatives

The alternatives field is critical — it feeds directly into llm_to_mass() where the primary prediction and alternatives are converted into a Dempster-Shafer mass function with calibrated uncertainty.

Batch mechanics:

Phase 2: ML Validation

After LLM labels are assigned, the ML pipeline runs as a validator. It produces independent evidence from 5 orthogonal sources and combines them via Dempster’s rule. The key output is conflict K — a rigorous measure of how much the sources disagree.

A column is flagged for revisit when both conditions hold:

1. The LLM label differs from the ML prediction: llm_code != ml_code
2. The DST conflict exceeds the threshold: K > k_threshold (default 0.2)

Phase 3: Targeted Revisit

When disagreements exist, the agent re-sends only contentious columns to the LLM with enriched context from the ML validation. The convergence loop iterates up to max_iterations (default 5) times with three exit conditions: all resolved, budget exhausted, or mean K below threshold.

Schema Discovery & Data Elements

Before the LLM sees any columns, the system performs structural analysis to discover Data Elements — composite semantic concepts that span columns across one or more tables.

Data Element Pattern Library

11 well-known patterns across 7 governance domains:

Table-Aware Batching

When table_aware_batching = true (the default), columns are grouped by source table instead of flat slices. This ensures the LLM prompt includes a ## Table: payments header and lists discovered data elements, giving the model coherent schema context.

FK Hint Detection

Columns named <entity>_id (e.g., instrument_id) are matched against table names (instruments) with singular/plural handling, identifying cross-table relationships that inform data element membership.

Post-Classification DE Refinement

After bootstrap classification, a second pass discovers data elements that naming patterns missed — using category co-occurrence within tables. Known category combinations (e.g., PAN + Name + Address → PaymentCard) trigger new data element emissions with source="co-occurrence".

Label Propagation

After LLM labels, similar unlabeled columns receive the same label via embedding cosine similarity. This exploits the SAGE finding that column_name accounts for 53% of classification signal — columns with similar names and values likely share the same category.

Propagation rules:

• Only from LLM-direct labels (no cascading)
• Cosine similarity > 0.85 (configurable)
• Skip if ML strongly disagrees (confidence > 0.5 and different code)

After all LLM phases complete, any remaining unlabeled columns with high ML confidence (>= confidence_floor) receive the ML prediction as their label. Every label carries a source_map entry tracking provenance: "llm", "llm_revisit", "propagated", or "ml".

LLM Revisit

When the LLM and ML disagree on a column AND K is high, the column is re-sent to the LLM with enriched context:

### Column 3: account_number (REVISIT)
Type: STRING | Values: ["1234567890", "9876543210"]
Siblings: [customer_name, routing_number, balance]
Data elements: [BankAccount]
ML prediction: BankAccountNumber [Bel=0.35, Pl=0.68, K=0.31]
Confusable: BankAccountNumber / PaymentAccountNumber
Your previous: PaymentAccountNumber (conf=0.65)

This gives the LLM three pieces of information it didn’t have before:

1. ML prediction with belief interval — what the evidence sources think, with calibrated uncertainty
2. Confusable pair — the specific known ambiguity between categories
3. Its own previous answer — a reminder of what it said and at what confidence

LLM Backends

The agent supports three LLM backends via the OpenAI chat completions protocol:

All backends use the same batch prompt format and JSON response parsing. The OpenAI-compatible backend includes lenient parsing (regex fallback) for smaller models.

GLM-4.7 is the default because it is an open-weights model — critical for reproducibility and on-premises deployment in air-gapped environments. The same weights can run via cloud inference APIs or locally via vLLM/TGI, ensuring identical classification behavior regardless of deployment mode. Its reasoning capability (chain-of-thought) improves classification accuracy on ambiguous columns, though it consumes ~80-85% of output tokens on reasoning, requiring llm_max_tokens >= 65536.

LLM as DST Evidence Source

The LLM’s predictions become proper mass functions via llm_to_mass(), not overrides:

The low discount (0.10) reflects the LLM’s high reliability. DST conflict K between LLM and ML sources is a genuine disagreement signal.

Configuration

All bootstrap parameters flow through HOCON (config/base.conf):

Integration with Self-Training

The bootstrap output feeds directly into the self-training pipeline:

# Step 1: Bootstrap — produce initial ground truth
uv run python scripts/bootstrap_classify.py \
    --data-dir ~/data/novel_tables/ \
    --data-elements \
    --output build/bootstrap_gt.json

# Step 2: Self-train — produce production model
uv run python scripts/build_sigint_embeddings.py \
    --data-dir ~/data/novel_tables/ \
    --ground-truth build/bootstrap_gt.json \
    --self-train --auto-generate \
    --output build/sigint_final.parquet

The --data-elements flag enables schema discovery and data element output alongside the ground truth JSON. The self-trained CatBoost model learns to reproduce LLM classifications with 99.4% fidelity — providing deterministic inference, per-item SHAP explanations, calibrated belief intervals, and zero marginal cost at scale.

Key Files