Context Engineering

Context engineering is the discipline of deliberately constructing, measuring, and optimizing the input features fed to an embedding classifier. Rather than treating the text assembled from a column’s metadata as an ad-hoc string, we decompose it into 12 discrete, ablatable features and use SAGE (Shapley Additive Global importancE) to quantify each feature’s contribution to classification accuracy.

The goal: replace intuition about “what information helps the classifier” with measured Shapley values that show exactly which features drive predictions and which are noise.

The Problem

An embedding classifier that encodes column metadata as free-form text achieves high accuracy on well-named columns — but most correct predictions are assisted by string-matching heuristics (name boost). Columns with opaque or generic names (e.g., col0, field_42) expose the model’s dependence on column names: cosine similarity drops dramatically on these columns while maintaining high accuracy on semantically named columns.

To build a classifier that generalizes beyond known column names, we need to:

1. Know which features actually contribute to correct predictions
2. Measure how much accuracy degrades when each feature is removed
3. Systematically reduce reliance on name matching
4. Compare multiple classification methods on the same feature set

Feature Decomposition

Each column is represented by 12 named features extracted from its metadata:

Features are composed into pipe-separated embedding text via ColumnFeatures.to_embedding_text(mask), where the mask controls which features are included. When a column has a generic name (col0, field_1, Unnamed), the value_description feature substitutes for the uninformative name in the embedding text; otherwise it is appended as complementary information. See Heuristic Elucidation for the methodology behind this feature.

With all features enabled, a column might produce:

payment card number | int | 4111111111111111, 5500000000000004 | cardinality=42 |
  patterns: credit_card_pattern | avg_len=16.0 | numeric=1.00 |
  siblings: first name, last name, expiry date | table=personal_data

Pattern Detectors

Eight regex-based pattern detectors scan sample values for structural signals:

A pattern is flagged when at least 1/3 of sample values match. This threshold prevents spurious matches from a single outlier value.

SAGE Feature Importance

SAGE (Shapley Additive Global importancE) measures the global importance of each feature by estimating how much each feature contributes to reducing classification loss. Unlike local explanation methods (LIME, SHAP per-instance), SAGE provides a single importance score per feature across the entire dataset.

How It Works

SAGE marginalizes each feature by substituting values from other samples in the dataset. For example, when marginalizing column_name for column email, SAGE asks: “what if this column had the name trade_secrets but the same sample values, cardinality, and patterns?”

The implementation:

1. Feature index matrix: X has shape (N, 12) where X[i, j] = i — each sample initially uses its own value for every feature.

2. Value lookup tables: For each feature j, a table maps sample index → feature text. When SAGE permutes X[i, j] to index k, the model uses sample k’s value for feature j instead.

3. Model wrapper (FeatureMaskModel): Looks up text from the per-feature tables according to X, assembles embedding text, encodes via sentence-transformers, and returns softmax’d cosine similarities as class probabilities.

4. SAGE estimation: MarginalImputer + PermutationEstimator with cross-entropy loss estimates each feature’s Shapley value across n_permutations (default 512) random orderings.

The result is a vector of 12 importance values with standard deviations — a quantitative answer to “which features matter?”

Pipeline

The classification pipeline runs in four stages, producing a parquet with three independent signals per column:

Three-Signal Comparison

When ground truth is provided (--ground-truth), the pipeline produces three independent classification signals per column:

The CatBoost CV baseline uses category reference embedding augmentation to overcome extreme low-data regimes. Each fold’s training set includes all category reference embeddings (the same texts cosine uses as targets), giving at least 2 training points per class even in held-out folds.

The train→eval pipeline replaces k-fold CV with synthetic training data and several additional techniques. See Classification Training for the full methodology and accuracy progression.

The pipeline always produces Dempster-Shafer belief intervals at every hierarchy level. See Evidence Fusion for the full DST architecture.

Column Kinds

The pipeline classifies each column into one of three kinds based on naming patterns:

• data: Semantically named columns (email, first_name, amount) — cosine excels here
• opaque: Generic or positional names (col0, field_42) — CatBoost train→eval is needed
• row_id: Row identifiers — excluded from GT evaluation

Ablation Runs

The pipeline supports controlled ablation experiments:

# Three-signal comparison with ground truth
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --ground-truth <ground-truth.json> \
    --output build/sigint_embeddings.parquet

# Feature ablation (disable specific features)
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --disable-features sample_values sibling_context \
    --output build/sigint_ablation.parquet

# With SAGE feature importance
uv run python scripts/build_sigint_embeddings.py \
    --data-dir <data-dir> \
    --taxonomy <taxonomy> --threshold 0.25 \
    --sage-permutations 512 \
    --output build/sigint_embeddings.parquet

# Visualize with embedding-atlas
embedding-atlas build/sigint_embeddings.parquet --text embedding_text

Output Schema

Each run produces a parquet (38 columns) and companion report JSON:

Current Status

The train→eval pipeline closes the opaque-name accuracy gap. Remaining errors are inherently confusable category pairs — types whose values are structurally identical (e.g., two address subtypes, two hex identifier formats). See Classification Training for the full methodology.

What remains:

1. Confusable pair resolution — remaining errors cluster in category pairs that share identical value patterns. Resolving these requires either richer context (e.g., table-level schema hints) or category consolidation
2. Feature refinement — richer pattern detectors (date formats, currency symbols, statistical distributions), deeper sample analysis (value distributions, min/max/mode)
3. Cross-dataset validation — accuracy on datasets with different naming conventions and taxonomies (see GitTables CTA benchmark)
4. Feedback integration — analyst corrections feed back as training signal, with SAGE tracking whether corrections improve non-name features