Building a Knowledge Engineering System: An Engineering Guide
Architecture decisions, data pipelines, and production patterns for teams building their first knowledge layer.
The Gap Between Theory and Running Code
Most writing about knowledge engineering describes what the system should do: classify content, extract entities, understand queries, connect facts in a graph. That’s necessary, but it skips the hard part. The hard part is the engineering: what runs where, what talks to what, what breaks first, and what you wish you’d done differently six months in.
This article is the engineering companion. It assumes you already know why knowledge engineering matters — taxonomies, ontologies, entity linking, query understanding. If you need that foundation, start with the conceptual overview. This piece is about how: the architecture, the data pipelines, the storage decisions, the editorial tooling, and the deployment patterns that make a knowledge engineering system actually work in production.
I’ve built these systems for companies ranging from $4B to $125B in revenue, and I’ve open-sourced the Python frameworks — MeaningFlow, Papilon, and PyCausalSim — that I use as building blocks. This article walks through how those pieces fit together into a real system.
System Architecture: The Three Layers
Every knowledge engineering system I’ve built decomposes into three layers. The boundaries between them are non-negotiable — cross them at your peril.
Layer 1: The Knowledge Store
This is the source of truth. It holds the taxonomy, the ontology schema, the entity records, the synonym lexicon, and the classification labels for every piece of content. Everything downstream reads from here. Nothing writes to it except the editorial tooling and the batch enrichment pipelines.
The first architectural decision is storage.
Graph database (Neo4j, Amazon Neptune, TigerGraph) — Use this when entity relationships are the core of your product. If your users search by “show me everything connected to X,” traversal speed matters and a graph database earns its operational overhead. Neo4j is the pragmatic default: Cypher is readable, the community edition is free, and the tooling ecosystem is mature.
Relational database (PostgreSQL) — Use this when the taxonomy and entity catalog are primarily lookup tables consumed by classifiers and search pipelines. Postgres with JSONB columns handles semi-structured entity attributes cleanly, scales to tens of millions of entities without exotic infrastructure, and your team already knows SQL. Most knowledge engineering systems start here and never need to leave.
Hybrid — In practice, many production systems use both. Postgres as the canonical store for entities and taxonomy nodes, with a graph database materialized from it for relationship traversal queries. The graph is a read replica of the relational store, rebuilt nightly or on change events.
My recommendation: start with Postgres. Add Neo4j when (and only when) you have a production query that requires multi-hop traversal and your users are waiting.
A minimal Postgres schema for the knowledge store:
-- Taxonomy: the category hierarchy
CREATE TABLE taxonomy_nodes (
id BIGSERIAL PRIMARY KEY,
parent_id BIGINT REFERENCES taxonomy_nodes(id),
slug TEXT NOT NULL UNIQUE,
label TEXT NOT NULL,
description TEXT,
depth INT NOT NULL DEFAULT 0,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON taxonomy_nodes (parent_id) WHERE is_active;
CREATE INDEX ON taxonomy_nodes (depth) WHERE is_active;
-- Ontology: entity types and their expected attributes
CREATE TABLE entity_types (
id BIGSERIAL PRIMARY KEY,
slug TEXT NOT NULL UNIQUE,
label TEXT NOT NULL,
schema JSONB NOT NULL DEFAULT '{}', -- attribute definitions
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Entities: the actual things in the world
CREATE TABLE entities (
id BIGSERIAL PRIMARY KEY,
entity_type TEXT NOT NULL REFERENCES entity_types(slug),
canonical_name TEXT NOT NULL,
aliases TEXT[] NOT NULL DEFAULT '{}',
attributes JSONB NOT NULL DEFAULT '{}',
embedding VECTOR(384), -- pgvector for similarity search
is_active BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON entities USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX ON entities (entity_type) WHERE is_active;
CREATE INDEX ON entities USING gin (aliases);
-- Entity relationships
CREATE TABLE entity_relations (
id BIGSERIAL PRIMARY KEY,
source_id BIGINT NOT NULL REFERENCES entities(id),
target_id BIGINT NOT NULL REFERENCES entities(id),
relation TEXT NOT NULL,
confidence REAL NOT NULL DEFAULT 1.0,
source_doc TEXT, -- provenance
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON entity_relations (source_id, relation);
CREATE INDEX ON entity_relations (target_id, relation);
-- Synonyms: the living lexicon
CREATE TABLE synonyms (
id BIGSERIAL PRIMARY KEY,
source_term TEXT NOT NULL,
target_term TEXT NOT NULL,
relation TEXT NOT NULL CHECK (relation IN ('exact','directional','misspelling','acronym')),
locale TEXT NOT NULL DEFAULT 'en-US',
confidence REAL NOT NULL DEFAULT 1.0,
created_by TEXT NOT NULL,
rationale TEXT,
review_by DATE,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON synonyms (source_term) WHERE is_active;
-- Content classifications
CREATE TABLE content_classifications (
id BIGSERIAL PRIMARY KEY,
content_id TEXT NOT NULL, -- external content identifier
taxonomy_node BIGINT NOT NULL REFERENCES taxonomy_nodes(id),
classifier TEXT NOT NULL, -- which system assigned this
confidence REAL NOT NULL,
is_override BOOLEAN NOT NULL DEFAULT FALSE, -- editorial override
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON content_classifications (content_id);
CREATE INDEX ON content_classifications (taxonomy_node);
-- Content-entity links
CREATE TABLE content_entities (
id BIGSERIAL PRIMARY KEY,
content_id TEXT NOT NULL,
entity_id BIGINT NOT NULL REFERENCES entities(id),
mention TEXT NOT NULL,
start_char INT,
end_char INT,
confidence REAL NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON content_entities (content_id);
CREATE INDEX ON content_entities (entity_id);Notice: pgvector for entity embeddings is built into the same database that holds the structured data. No separate vector store needed for most knowledge engineering workloads. One fewer system to maintain, one fewer synchronization problem to debug.
Layer 2: The Processing Pipelines
These are the batch and streaming jobs that enrich content. They read raw content, run classifiers and NER, link entities, and write results back to the knowledge store. They never serve user requests directly.
The pipeline has four stages, always in this order:
Stage A: Ingest. Pull raw content from the CMS, the search index, or the content API. Normalize it: strip HTML, extract plain text, parse metadata. Write normalized documents to a staging table or object store.
Stage B: Classify. Run the content through the classification cascade — rules, fine-tuned classifier, LLM fallback. Write classifications to content_classifications with the classifier name and confidence.
Stage C: Extract. Run NER over the content. Disambiguate and link mentions to entities in the knowledge store. Write to content_entities.
Stage D: Reconcile. Cross-check classifications against entity extractions. Flag inconsistencies (a document classified as “Sports” but with no sports entities detected). Route flagged items to the editorial review queue.
The implementation can be as simple as a set of Python scripts orchestrated by cron, or as sophisticated as an Airflow DAG with retry logic and SLAs. Start with the simplest thing. I’ve seen teams spend months building an elaborate orchestration layer for a pipeline that processes a few thousand documents per day — a cron job with error emails would have shipped in a week.
A stripped-down pipeline orchestrator:
"""
knowledge_pipeline.py
Run daily: classify, extract, reconcile.
"""
import logging
from datetime import datetime, timedelta
from dataclasses import dataclass
from db import get_session # your SQLAlchemy or psycopg connection
from classifiers import RuleClassifier, EncoderClassifier, LLMClassifier
from extractors import NERExtractor, EntityLinker
from reconciler import cross_check_and_flag
log = logging.getLogger(__name__)
@dataclass
class PipelineConfig:
lookback_hours: int = 24
confidence_threshold: float = 0.6
max_llm_calls: int = 5000 # budget guard
def run_pipeline(config: PipelineConfig):
session = get_session()
cutoff = datetime.utcnow() - timedelta(hours=config.lookback_hours)
# Stage A: fetch new/updated content
docs = session.execute(
"SELECT id, body, metadata FROM content WHERE updated_at > :cutoff",
{"cutoff": cutoff}
).fetchall()
log.info(f"Pipeline processing {len(docs)} documents")
# Stage B: classify
rule_clf = RuleClassifier.load("rules_v3.yaml")
encoder_clf = EncoderClassifier.load("deberta_taxonomy_v2.pt")
llm_clf = LLMClassifier(budget=config.max_llm_calls)
classifications = []
for doc in docs:
# Cascade: rules → encoder → LLM
result = rule_clf.classify(doc.body)
if result and result.confidence >= config.confidence_threshold:
result.classifier = "rules_v3"
else:
result = encoder_clf.classify(doc.body)
if result.confidence >= config.confidence_threshold:
result.classifier = "deberta_v2"
else:
result = llm_clf.classify(doc.body)
result.classifier = "llm_fallback"
classifications.append((doc.id, result))
bulk_insert_classifications(session, classifications)
# Stage C: extract entities
ner = NERExtractor(model="en_core_web_trf")
linker = EntityLinker(session=session)
for doc in docs:
mentions = ner.extract(doc.body)
linked = linker.link(mentions, context=doc.body)
bulk_insert_entities(session, doc.id, linked)
# Stage D: reconcile
flagged = cross_check_and_flag(session, [d.id for d in docs])
log.info(f"Flagged {len(flagged)} documents for editorial review")
session.commit()
log.info("Pipeline complete")
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
run_pipeline(PipelineConfig())Cost control matters. The LLM classifier in the cascade is the expensive step. The max_llm_calls budget guard prevents a spike in new content from blowing your API bill. In practice, a well-tuned encoder classifier should handle 85-95% of documents with high confidence, leaving only the genuinely ambiguous cases for the LLM fallback.
Layer 3: The Serving Layer
This is what user-facing systems actually call. It exposes the knowledge store through fast, cacheable APIs. It never runs classifiers or NER — that’s the pipeline layer’s job. It just looks things up.
Three APIs cover most use cases:
Classify API — Given a content ID, return its taxonomy classification. This is a database lookup, not a model inference. Response time: single-digit milliseconds.
Entity API — Given a content ID, return the linked entities. Or given an entity ID, return all content linked to it. Again, database lookups with proper indexes.
Query Understanding API — Given a raw query string, return structured intent, entities, and filters. This is the only serving-layer endpoint that runs inference (intent classifier + entity linker on queries), but the models are small and fast — a distilled BERT intent classifier runs in under 10ms on a CPU.
"""
Minimal FastAPI serving layer.
"""
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from db import get_session
from query_understanding import QueryParser
app = FastAPI(title="Knowledge API")
query_parser = QueryParser.load("query_model_v4")
class QueryResult(BaseModel):
intent: str
intent_confidence: float
entities: list[dict]
filters: list[dict]
expanded_terms: list[str]
@app.get("/classify/{content_id}")
async def get_classification(content_id: str):
session = get_session()
result = session.execute(
"""SELECT tn.slug, tn.label, cc.confidence, cc.classifier
FROM content_classifications cc
JOIN taxonomy_nodes tn ON cc.taxonomy_node = tn.id
WHERE cc.content_id = :cid
ORDER BY cc.confidence DESC LIMIT 1""",
{"cid": content_id}
).fetchone()
if not result:
raise HTTPException(404, "Content not classified")
return {
"content_id": content_id,
"category": result.slug,
"label": result.label,
"confidence": result.confidence,
"classifier": result.classifier,
}
@app.get("/entities/{content_id}")
async def get_entities(content_id: str):
session = get_session()
rows = session.execute(
"""SELECT e.id, e.canonical_name, e.entity_type, ce.mention, ce.confidence
FROM content_entities ce
JOIN entities e ON ce.entity_id = e.id
WHERE ce.content_id = :cid
ORDER BY ce.confidence DESC""",
{"cid": content_id}
).fetchall()
return {"content_id": content_id, "entities": [dict(r._mapping) for r in rows]}
@app.post("/query/parse")
async def parse_query(query: str) -> QueryResult:
result = query_parser.parse(query)
return QueryResult(**result)Caching. Content classifications and entity links change only when the pipeline runs. Put a Redis or Memcached layer in front of the classify and entity endpoints. Cache TTL = pipeline run interval. This handles traffic spikes without touching the database.
The Semantic Analysis Layer: MeaningFlow Integration
Before any of the classification or entity work begins, you need to understand the shape of your content and your users’ demand. This is where MeaningFlow fits in the architecture — not as a runtime component, but as an analytical layer that feeds the design of everything else.
Run MeaningFlow on two corpora: your content inventory and your query logs. The output is a pair of semantic graphs — one representing what you have, one representing what users want. The gap between them drives three critical design decisions:
Taxonomy structure. The clusters in the demand graph that have no corresponding supply clusters are the categories your taxonomy is missing. Don’t design taxonomy branches in a conference room. Design them from the data.
Entity type priorities. The entity types that appear most frequently in high-volume query clusters are the types you should extract first. If “Person” entities dominate your top clusters but “Organization” entities are sparse, build the Person NER pipeline first.
Synonym seeds. Terms that co-occur in the same query cluster but use different surface forms are synonym candidates. MeaningFlow’s graph structure makes these pairs visible without manual analysis.
The MeaningFlow analysis runs once during system design, then monthly as a monitoring check. It’s not in the hot path. It’s in the strategy path.
"""
Quarterly semantic health check.
Compare current content coverage against current demand.
"""
from meaningflow import SemanticGraph
import pandas as pd
import json
def semantic_health_check(
content_path: str,
queries_path: str,
previous_report_path: str = None,
) -> dict:
content = pd.read_parquet(content_path)["title"].tolist()
queries = pd.read_parquet(queries_path)["query"].tolist()
supply = SemanticGraph(texts=content, embedder="all-MiniLM-L6-v2", min_cluster_size=30)
supply.fit()
demand = SemanticGraph(texts=queries, embedder="all-MiniLM-L6-v2", min_cluster_size=30)
demand.fit()
gaps = demand.coverage_gaps(reference=supply, similarity_threshold=0.55)
report = {
"timestamp": pd.Timestamp.now().isoformat(),
"supply_clusters": supply.n_clusters,
"demand_clusters": demand.n_clusters,
"gap_clusters": len(gaps),
"top_gaps": [
{"terms": g.top_terms[:5], "volume": g.volume, "size": g.size}
for g in gaps[:20]
],
}
# Drift detection against previous report
if previous_report_path:
with open(previous_report_path) as f:
prev = json.load(f)
report["gap_count_change"] = len(gaps) - prev.get("gap_clusters", 0)
report["demand_cluster_change"] = demand.n_clusters - prev.get("demand_clusters", 0)
return reportBuilding the Classification Pipeline in Detail
The classification cascade — rules, encoder, LLM — deserves a deeper walkthrough because it’s the component that breaks most often and costs the most to debug.
The Rule Layer
Rules are YAML files that editors can update without a deploy:
# classification_rules_v3.yaml
rules:
- name: "breaking_news_override"
priority: 1
conditions:
- field: "metadata.content_type"
operator: "equals"
value: "breaking"
action:
category: "news/breaking"
confidence: 1.0
- name: "publisher_vertical"
priority: 2
conditions:
- field: "metadata.publisher"
operator: "in"
value: ["espn.com", "bleacherreport.com", "theathletic.com"]
action:
category: "sports"
confidence: 0.95
- name: "keyword_finance"
priority: 10
conditions:
- field: "body"
operator: "contains_any"
value: ["stock market", "S&P 500", "Federal Reserve", "interest rate"]
- field: "body"
operator: "not_contains_any"
value: ["fantasy", "betting", "odds"]
action:
category: "finance/markets"
confidence: 0.85The rule engine evaluates in priority order and returns the first match. Lower priority number = higher precedence. The not_contains_any guard in the finance rule prevents sports betting content from being misclassified — a real production bug I've seen at three different companies.
The Encoder Layer
A fine-tuned DeBERTa-v3-small classifier trained on your labeled data. The training loop is standard, but two details matter for production:
Multi-label support. Content can belong to multiple categories. A movie review is both “entertainment/film” and “culture/criticism.” Train with binary cross-entropy per class, not softmax. At inference time, return all classes above a confidence threshold rather than just the argmax.
Calibration. Neural classifiers are notoriously overconfident. The raw softmax probabilities don’t mean what you think they mean. Calibrate with temperature scaling on a held-out set so that a confidence of 0.8 actually means the classifier is right 80% of the time. Without calibration, your confidence thresholds are meaningless.
"""
Temperature scaling for classifier calibration.
Fit on a held-out validation set after training.
"""
import numpy as np
from scipy.optimize import minimize_scalar
from sklearn.metrics import log_loss
def find_temperature(logits: np.ndarray, labels: np.ndarray) -> float:
"""Find the temperature that minimizes NLL on the validation set."""
def nll_at_temp(T):
scaled = logits / T
probs = np.exp(scaled) / np.exp(scaled).sum(axis=1, keepdims=True)
return log_loss(labels, probs)
result = minimize_scalar(nll_at_temp, bounds=(0.1, 10.0), method="bounded")
return result.x
def calibrated_predict(logits: np.ndarray, temperature: float) -> np.ndarray:
"""Apply temperature scaling to produce calibrated probabilities."""
scaled = logits / temperature
return np.exp(scaled) / np.exp(scaled).sum(axis=1, keepdims=True)The LLM Fallback
For the remaining 5–15% of documents where the encoder is uncertain, send the text to an LLM with the taxonomy embedded in the prompt. This is the same pattern I described in the previous article, but with a production-critical addition: caching.
Many documents that confuse the encoder are near-duplicates of each other — syndicated content, press releases picked up by multiple outlets, templated product pages. Hash the input text, check the cache before calling the API. A simple content-hash → classification cache cuts LLM costs by 30–50% in most content pipelines.
import hashlib
import json
from functools import lru_cache
class LLMClassifier:
def __init__(self, budget: int = 5000):
self.budget = budget
self.calls_made = 0
self._cache = {} # In production, use Redis
def _content_hash(self, text: str) -> str:
# Hash first 2000 chars — enough for classification, avoids
# hashing entire long-form articles
return hashlib.sha256(text[:2000].encode()).hexdigest()
def classify(self, text: str) -> dict:
h = self._content_hash(text)
if h in self._cache:
return self._cache[h]
if self.calls_made >= self.budget:
return {"category": "uncategorized", "confidence": 0.0,
"classifier": "llm_budget_exhausted"}
result = self._call_llm(text)
self.calls_made += 1
self._cache[h] = result
return result
def _call_llm(self, text: str) -> dict:
# Your LLM API call here — OpenAI, local model, etc.
# Returns {"category": "...", "confidence": 0.0-1.0}
...Building the Entity Pipeline
Entity extraction and linking is the second major pipeline, and it has its own architectural concerns distinct from classification.
NER Model Selection
For most content domains, spaCy’s transformer-based models (en_core_web_trf) handle the standard entity types — Person, Organization, Location, Date — well enough to start. Where you'll need custom NER is domain-specific entity types: product names, movie titles, song titles, medical terms, financial instruments.
Two approaches to custom NER:
Few-shot with an LLM. Same cost-control patterns as the classification fallback. Send a chunk of text with a prompt asking for entities of your custom types. Works well for low-volume, high-value content. Not viable for millions of documents per day.
Fine-tuned span classifier. Train a small model (SpanBERT, DeBERTa) on a few thousand annotated examples. This is the long-term answer. The annotation burden is the bottleneck — use the LLM approach to bootstrap the training set, then have editors correct the results.
Entity Disambiguation at Scale
The toy version of disambiguation — embed the mention context, embed each candidate, take the nearest — works for a few thousand entities. At scale (millions of entities in the knowledge store), you need approximate nearest neighbor search.
The architecture: store entity embeddings in pgvector (already in your schema above). At disambiguation time, embed the mention-in-context, query pgvector for the top-k nearest entity embeddings, then re-rank with a lightweight cross-encoder or heuristic features (entity type match, popularity prior, session context).
"""
Scalable entity disambiguation with pgvector + re-ranking.
"""
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np
class EntityDisambiguator:
def __init__(self, session, embedder_name="all-MiniLM-L6-v2"):
self.session = session
self.embedder = SentenceTransformer(embedder_name)
self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def disambiguate(self, mention: str, context: str,
entity_type: str, top_k: int = 10) -> dict:
# Embed the mention in context
query_text = f"{mention}: {context[:300]}"
query_emb = self.embedder.encode(query_text).tolist()
# ANN search in pgvector
candidates = self.session.execute(
"""SELECT id, canonical_name, attributes
FROM entities
WHERE entity_type = :etype AND is_active
ORDER BY embedding <=> :emb::vector
LIMIT :k""",
{"etype": entity_type, "emb": str(query_emb), "k": top_k}
).fetchall()
if not candidates:
return {"entity_id": None, "confidence": 0.0}
# Re-rank with cross-encoder
pairs = [(query_text, c.canonical_name) for c in candidates]
scores = self.reranker.predict(pairs)
best_idx = int(np.argmax(scores))
best = candidates[best_idx]
return {
"entity_id": best.id,
"canonical_name": best.canonical_name,
"confidence": float(scores[best_idx]),
}Session-Aware Disambiguation
In the previous article, I described a session memory pattern based on my Memory-Node Encapsulation (MNE) research. In the system architecture, the session memory lives in Redis with a TTL equal to the session timeout. The query understanding API reads it, and the entity disambiguation step uses it as a prior:
import redis
import json
from time import time
class SessionStore:
def __init__(self, redis_url: str, ttl_seconds: int = 1800):
self.r = redis.from_url(redis_url)
self.ttl = ttl_seconds
def record_entity(self, session_id: str, entity_id: int):
key = f"session:{session_id}:entities"
self.r.zadd(key, {str(entity_id): time()})
self.r.expire(key, self.ttl)
def get_recent_entities(self, session_id: str, n: int = 10) -> list[int]:
key = f"session:{session_id}:entities"
# Most recent first
results = self.r.zrevrange(key, 0, n - 1)
return [int(eid) for eid in results]
def entity_recency_boost(self, session_id: str, entity_id: int,
half_life: float = 300.0) -> float:
key = f"session:{session_id}:entities"
ts = self.r.zscore(key, str(entity_id))
if ts is None:
return 0.0
age = time() - ts
return 0.5 ** (age / half_life)The disambiguation step adds entity_recency_boost to the cross-encoder score before picking the best candidate. Users who just interacted with "Apple Inc." get a disambiguated toward the company on their next query — without any explicit signal from the user.
Editorial Tooling: The Most Underinvested Layer
In every knowledge engineering system I’ve seen fail, the failure mode is the same: the ML pipelines work, the knowledge store is correctly designed, but editors can’t actually use the system because the tooling is terrible. They’re editing YAML in a text editor, reviewing classifications in a spreadsheet, and managing the synonym lexicon through Jira tickets.
Build the editorial UI before you scale the pipelines. Three screens cover the core workflows:
Screen 1: Classification Review Queue
A paginated list of documents flagged by the reconciliation step — low-confidence classifications, classifier-graph disagreements, documents in the “uncategorized” bucket. For each document:
- The title and first 200 characters of body text.
- The top three candidate classifications with confidence scores.
- One-click buttons to accept a candidate, override with a different category, or skip.
- A “needs new category” flag that routes to the taxonomy editor.
The key metric: time per review decision. Measure it. If editors are averaging more than 15 seconds per document, the UI is making them work too hard. Common causes: too many clicks, category picker is hard to navigate, no keyboard shortcuts.
Screen 2: Taxonomy Editor
A tree view of the taxonomy with drag-and-drop reordering, inline rename, add child, merge branches, and deprecate (soft delete with a redirect target). Every change writes an audit log entry. Every change triggers a downstream re-classification of affected content — but queued as a batch job, not inline.
Screen 3: Synonym Manager
A searchable table of the synonym lexicon. Editors can add entries, deactivate entries, and see usage stats (how many queries matched this synonym in the last 30 days). An “auto-suggest” tab shows synonym candidates proposed by the MeaningFlow clustering — editors approve or reject with one click.
Build these as a simple internal web app. React + your existing API layer. Do not build them as a Slack bot, a Notion integration, or a Google Sheet. Those approaches always start “simple” and end in unmaintainable chaos.
Monitoring and Observability
A knowledge engineering system in production needs three monitoring dimensions.
Pipeline Health
Standard data pipeline monitoring: did the pipeline run? How long did each stage take? How many documents were processed? How many failed? Alert on failures and on throughput drops (which often indicate an upstream data issue rather than a pipeline bug).
Classification Quality
Track the distribution of classifier confidence over time. A sudden shift — average confidence dropping from 0.82 to 0.71 — means either the content mix has changed or the model is stale. Either way, investigate.
Track the rate of “uncategorized” documents. This is your exhaust valve. A rising uncategorized rate means your taxonomy or your classifier isn’t keeping up with the content.
Track editorial override rate. If editors are overriding more than 10% of machine classifications, the classifier needs retraining. If they’re overriding less than 2%, you might be able to reduce the review queue and free up editorial time.
Drift Detection with Papilon
For deeper attribution — understanding why quality shifted and which changes drove which outcomes — I use Papilon to build causal models over the pipeline’s operational data.
"""
Monthly causal attribution: what drove classification quality changes?
"""
import papilon as pp
def monthly_attribution(metrics_path: str):
df = pp.load_panel(metrics_path)
dag = pp.discover_causal_structure(
df,
treatments=[
"taxonomy_version",
"encoder_version",
"rule_update_count",
"content_volume_change",
],
outcome="editorial_override_rate",
confounders=["month", "content_mix_entropy"],
)
for treatment in ["taxonomy_version", "encoder_version", "rule_update_count"]:
effect = pp.estimate_effect(
df,
treatment=treatment,
outcome="editorial_override_rate",
dag=dag,
)
print(f"{treatment}: {effect.point:.4f} "
f"(95% CI {effect.ci_low:.4f}–{effect.ci_high:.4f})")This tells you things like: “The taxonomy expansion in March reduced the editorial override rate by 3.2 percentage points, independent of the classifier retrain that shipped the same week.” That’s the evidence you need to justify continued investment in the knowledge engineering program.
Deployment Topology
For a team of 2–5 engineers supporting a medium-scale content platform (millions of documents, tens of millions of queries per month), the deployment looks like this:
Knowledge Store: Single Postgres instance with pgvector extension. If you outgrow this, shard by entity type before reaching for a distributed database.
Pipeline: Python scripts running on a schedule (cron or Airflow). One worker for classification, one for entity extraction. Run on spot instances — pipeline jobs are idempotent and restartable.
Serving Layer: FastAPI behind a load balancer, with Redis caching. Two small instances handle most traffic. Scale horizontally if needed — the API is stateless.
Editorial UI: React SPA served from a CDN, talking to the same FastAPI backend. Internal-only, behind your VPN or auth provider.
MeaningFlow Analysis: Jupyter notebooks or scheduled Python scripts running monthly. Not a production service — an analytical tool. Results feed into the editorial process, not into the serving layer.
Papilon Attribution: Monthly batch job. Reads operational metrics from the pipeline’s log tables, outputs a report. Same deployment pattern as MeaningFlow — analytical, not serving.
Total infrastructure cost for this setup: $500–2,000/month depending on content volume and LLM API usage. The LLM fallback classifier is the variable cost — everything else is fixed and modest.
What Breaks First
In the interest of saving you six months of debugging, here are the failure modes I’ve seen repeatedly:
Taxonomy drift. The world changes faster than the taxonomy. New categories emerge, old categories become irrelevant, and the classifier keeps assigning content to stale branches. Fix: quarterly MeaningFlow health checks with mandatory taxonomy review.
Entity proliferation. Without deduplication, the entity store fills with near-duplicate records — “Kendrick Lamar,” “kendrick lamar,” “K. Lamar,” “Kendrick Lamar Duckworth.” Your entity linker starts splitting mentions across duplicates, degrading every downstream system. Fix: a deduplication pipeline that runs weekly, using embedding similarity plus edit distance to propose merge candidates for editorial review.
Synonym collisions. An editor adds a synonym that creates an unintended mapping — “apple” → “fruit” breaks searches for Apple Inc. Fix: every synonym addition triggers an impact assessment showing which existing queries and classifications would be affected. Display this in the editorial UI before the editor clicks “save.”
LLM cost spikes. A content migration or a new publisher integration suddenly doubles the volume of documents hitting the LLM fallback. The budget guard catches this eventually, but not before you’ve burned through a week’s API budget in a day. Fix: rate limiting per-source, with alerts when any single source exceeds its expected volume by more than 2x.
The cold start problem. When you launch a new taxonomy branch, you have no training data for it. The encoder classifier can’t classify content into a category it’s never seen. The LLM fallback works but is expensive at the volume of a new branch. Fix: use the LLM to generate the initial labeled dataset (500–1,000 examples), fine-tune the encoder, then shift to the standard cascade. This is a one-time cost per new branch.
The Build Sequence
If I were starting a knowledge engineering system from scratch tomorrow, this is the order I’d build in:
Week 1–2: Postgres schema. Seed taxonomy (3–5 top-level categories from editorial judgment). MeaningFlow analysis to validate and expand the seed.
Week 3–4: Rule classifier (YAML-based). Run it over the full content corpus. Measure coverage — how much content do rules handle alone?
Week 5–6: LLM fallback classifier for everything rules miss. Use the results as training data. Editorial review of a random sample to establish baseline quality.
Week 7–8: Fine-tune encoder classifier on the LLM-generated + editor-corrected labels. Implement the full cascade. Measure the encoder’s coverage at your confidence threshold.
Week 9–10: NER pipeline. Entity store. Basic disambiguation using pgvector. Start with the top two entity types by query volume.
Week 11–12: Editorial UI — classification review queue and synonym manager. Get editors using the system daily.
Week 13–14: Serving API. Redis cache. Integration with the search or content platform.
Week 15–16: Monitoring dashboard. Drift detection. First causal attribution run with Papilon.
Sixteen weeks to a production knowledge engineering system. Not a prototype — a system with editorial tooling, monitoring, and causal attribution. Every subsequent quarter is iterative improvement: more entity types, deeper taxonomy, better classifiers, more synonyms, more coverage.
Closing Thought
The difference between a knowledge engineering project and a knowledge engineering system is infrastructure. The project produces a taxonomy and a classifier and declares victory. The system produces a continuously-improving representation of meaning that gets better every week because editors can use it, engineers can monitor it, and leadership can see the causal impact of investing in it.
Build the system. It’s more work upfront, but it’s the only version that compounds.
Brian Curry is a Kansas City–based data scientist, AI researcher, and founder of Vector1 Research, an independent lab advancing marketing economics, causal inference, and cognitive AI systems. He has led data and analytics initiatives at enterprises from $4B to $125B, including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T. He is the creator of MeaningFlow (semantic content modeling), Papilon (causal inference and complex systems), PyCausalSim (causal discovery through simulation), and the Memory-Node Encapsulation (MNE) architecture for cognitive AI.
Contact: brian at vector1.ai | Vector1 Research | GitHub
