The 3D Semantic Footprint: Measuring and Visualizing a Company’s Entity Landscape for LLM Optimization

The 3D Semantic Footprint: Advanced Measurement and Visualization of a Company’s Entity Landscape for LLM Optimization

Executive Summary

Modern large language models and AI-powered search systems reason over high-dimensional vector embeddings, structured entities, and semantic relationships. Your organization’s semantic footprint — the mathematical structure that emerges from your public content — directly influences how these systems understand, retrieve, and associate your brand with relevant concepts. This comprehensive guide introduces a production-ready methodology for extracting, analyzing, and visualizing this footprint as an interactive 3D representation, enabling data-driven optimization of content strategy, information architecture, and LLM performance.

Key outcomes:

• Identify semantic clusters representing thematic content hubs
• Discover bridging topics that connect disparate concept areas
• Expose coverage gaps in your entity landscape
• Optimize for improved LLM recall, grounding accuracy, and brand association
• Enable quantitative measurement of semantic coherence over time

1. Strategic Context: Why Semantic Footprints Matter

1.1 The Shift from Keyword Matching to Semantic Understanding

Traditional search and information retrieval systems operated on lexical matching — literal string comparisons with minor sophistication through TF-IDF weighting and Boolean logic. Modern AI systems fundamentally differ:

Vector-Space Semantics: Contemporary LLMs encode text into continuous vector representations where semantic similarity corresponds to geometric proximity in high-dimensional space (typically 384 to 4096 dimensions). This enables:

• Synonym and paraphrase recognition without explicit mappings
• Cross-lingual semantic matching through shared embedding spaces
• Contextual disambiguation based on surrounding content
• Analogical reasoning through vector arithmetic

Entity-Centric Knowledge Graphs: Rather than treating all tokens equally, modern systems extract and canonicalize entities (people, organizations, products, concepts) and model explicit relationships between them. This structured representation enables:

• Fact verification against knowledge bases
• Multi-hop reasoning across connected entities
• Attribution and provenance tracking
• Disambiguation of polysemous terms

1.2 How LLMs “See” Your Content

When an LLM encounters your website, documentation, or corpus, it constructs an internal representation through multiple mechanisms:

Embedding-Based Retrieval: Systems like RAG (Retrieval-Augmented Generation) chunk your content, compute embeddings, and store them in vector databases. At inference time, query embeddings retrieve the top-k most similar chunks using approximate nearest neighbor search (ANN) algorithms like HNSW or IVF.

Entity Extraction and Linking: Named Entity Recognition (NER) identifies mentions of entities, which are then normalized and potentially linked to canonical identifiers in knowledge bases (Wikidata, DBpedia, proprietary ontologies). This enables:

• Aggregation of information across mentions
• Resolution of co-reference chains
• Connection to broader world knowledge

Attention Pattern Analysis: Transformer architectures learn which tokens and entities attend to each other most strongly, forming implicit relationship graphs. Content with clear semantic structure generates more coherent attention patterns.

1.3 Business Impact of Semantic Optimization

Organizations with well-structured semantic footprints experience:

Higher Retrieval Precision: When users query AI systems about your domain, optimized content appears more frequently and accurately in responses (30–60% improvement in controlled studies).

Enhanced Brand Association: Semantic clustering around your brand terms strengthens the model’s learned associations, making your organization the default answer for relevant queries.

Improved Grounding: LLMs are less likely to hallucinate or misattribute information when your content provides clear, well-linked entity relationships with supporting evidence.

Competitive Intelligence: Comparing your semantic footprint to competitors reveals strategic gaps and opportunities for differentiation.

2. Architecture and Pipeline Design

2.1 System Overview

The semantic footprint pipeline consists of seven primary stages, each with specific technical requirements and optimization considerations:

[Web Crawling] → [Text Extraction] → [Passage Chunking] → [Entity Extraction] → 
[Embedding Generation] → [Dimensionality Reduction] → [Visualization & Analysis]

Design Principles:

• Idempotency: Pipeline stages should be rerunnable without side effects
• Caching: Expensive operations (embedding generation, NER) should cache results
• Scalability: Support for distributed processing of large corpora (10K+ pages)
• Versioning: Track model versions, parameters, and timestamps for reproducibility

2.2 Detailed Component Architecture

Stage 1: Intelligent Web Crawling

Objectives:

• Discover all publicly accessible content systematically
• Respect robots.txt and rate limits
• Handle authentication and dynamic content where appropriate
• Deduplication and canonicalization

Implementation Considerations:

class SemanticCrawler:
    """Production-grade crawler with rate limiting and error handling."""
    
    def __init__(self, 
                 base_urls: List[str],
                 max_depth: int = 3,
                 rate_limit: float = 1.0,  # requests per second
                 user_agent: str = "SemanticFootprintBot/2.0",
                 respect_robots: bool = True):
        self.base_urls = base_urls
        self.max_depth = max_depth
        self.rate_limit = rate_limit
        self.user_agent = user_agent
        self.respect_robots = respect_robots
        self.session = requests.Session()
        self.visited = set()
        self.url_queue = queue.PriorityQueue()
        
    def crawl_from_sitemap(self, sitemap_url: str) -> List[str]:
        """Extract URLs from XML sitemap (preferred method)."""
        response = self.session.get(sitemap_url)
        soup = BeautifulSoup(response.content, 'xml')
        urls = [loc.text for loc in soup.find_all('loc')]
        return urls
    
    def extract_links(self, html: str, base_url: str) -> List[str]:
        """Extract and normalize all links from HTML."""
        soup = BeautifulSoup(html, 'html.parser')
        links = []
        for a in soup.find_all('a', href=True):
            url = urljoin(base_url, a['href'])
            # Normalize: remove fragments, sort params, lowercase domain
            url = urlparse(url)._replace(fragment='').geturl()
            if self._is_valid_url(url):
                links.append(url)
        return list(set(links))
    
    def _is_valid_url(self, url: str) -> bool:
        """Filter out non-HTML resources and external domains."""
        parsed = urlparse(url)
        
        # Check domain whitelist
        if not any(base in parsed.netloc for base in self.base_urls):
            return False
            
        # Exclude binary files
        excluded_extensions = {'.pdf', '.jpg', '.png', '.gif', '.zip', 
                               '.mp4', '.mp3', '.css', '.js'}
        if any(parsed.path.lower().endswith(ext) for ext in excluded_extensions):
            return False
            
        return True
    
    def crawl(self) -> pd.DataFrame:
        """Execute breadth-first crawl with rate limiting."""
        # Implementation with priority queue, rate limiting, retry logic
        pass

Sitemap vs Crawling Trade-offs:

Approach Pros Cons Best For XML Sitemap Fast, complete, respects publisher intent May miss unlisted pages Most websites Recursive Crawl Discovers all linked content Slow, may hit rate limits Documentation sites Hybrid Comprehensive Complex implementation Large-scale enterprise

Stage 2: Content Extraction and Cleaning

Challenges:

• Removing boilerplate (headers, footers, navigation)
• Handling dynamic content (JavaScript-rendered)
• Extracting from various formats (HTML, PDF, Markdown)
• Preserving semantic structure (headings, lists, tables)

Advanced Extraction with Trafilatura:

import trafilatura
from trafilatura.settings import use_config

def extract_main_content(html: str, url: str) -> Dict[str, Any]:
    """Extract main content while preserving structure."""
    
    # Configure extraction settings
    config = use_config()
    config.set("DEFAULT", "EXTRACTION_TIMEOUT", "30")
    
    # Extract with metadata
    result = trafilatura.extract(
        html,
        include_comments=False,
        include_tables=True,
        include_images=False,
        include_links=True,
        output_format='json',
        url=url,
        config=config
    )
    
    if result:
        data = json.loads(result)
        return {
            'text': data.get('text', ''),
            'title': data.get('title', ''),
            'author': data.get('author', ''),
            'date': data.get('date', ''),
            'description': data.get('description', ''),
            'url': url
        }
    return None

def clean_text(text: str) -> str:
    """Normalize whitespace and remove artifacts."""
    # Remove excessive whitespace
    text = re.sub(r'\s+', ' ', text)
    
    # Remove common artifacts
    text = re.sub(r'[\x00-\x08\x0B-\x0C\x0E-\x1F\x7F]', '', text)
    
    # Normalize unicode
    text = unicodedata.normalize('NFKC', text)
    
    return text.strip()

Handling PDFs:

import pdfplumber
from pdfminer.high_level import extract_text_to_fp
from io import StringIO

def extract_pdf_content(pdf_path: str) -> str:
    """Extract text from PDF with table detection."""
    text_parts = []
    
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            # Extract tables separately for better structure
            tables = page.extract_tables()
            for table in tables:
                # Convert table to markdown-style text
                table_text = '\n'.join([' | '.join(map(str, row)) for row in table])
                text_parts.append(table_text)
            
            # Extract remaining text
            text = page.extract_text()
            if text:
                text_parts.append(text)
    
    return '\n\n'.join(text_parts)

Stage 3: Semantic Chunking Strategies

The Chunking Dilemma: Passage length directly impacts embedding quality and retrieval performance. Trade-offs include:

• Short chunks (100–300 chars): High precision but loss of context
• Medium chunks (300–1100 chars): Balanced approach, recommended
• Long chunks (1100+ chars): Rich context but diluted similarity scores

Advanced Chunking Algorithms:

from typing import List, Tuple
import spacy

class SemanticChunker:
    """Chunk text based on semantic boundaries rather than character counts."""
    
    def __init__(self, nlp_model: str = "en_core_web_lg"):
        self.nlp = spacy.load(nlp_model)
        self.min_chars = 300
        self.max_chars = 1100
        self.target_chars = 700
    
    def chunk_by_sentences(self, text: str) -> List[str]:
        """Create chunks at sentence boundaries."""
        doc = self.nlp(text)
        chunks = []
        current_chunk = []
        current_length = 0
        
        for sent in doc.sents:
            sent_text = sent.text.strip()
            sent_length = len(sent_text)
            
            # If adding this sentence exceeds max, finalize current chunk
            if current_length + sent_length > self.max_chars and current_chunk:
                chunks.append(' '.join(current_chunk))
                current_chunk = [sent_text]
                current_length = sent_length
            else:
                current_chunk.append(sent_text)
                current_length += sent_length
                
                # If we're past target and at a good stopping point, finalize
                if current_length >= self.target_chars:
                    chunks.append(' '.join(current_chunk))
                    current_chunk = []
                    current_length = 0
        
        # Add remaining
        if current_chunk:
            final_text = ' '.join(current_chunk)
            if len(final_text) >= self.min_chars:
                chunks.append(final_text)
            elif chunks:
                # Merge with last chunk if too short
                chunks[-1] += ' ' + final_text
            else:
                chunks.append(final_text)
        
        return chunks
    
    def chunk_by_topics(self, text: str, similarity_threshold: float = 0.7) -> List[str]:
        """Create chunks based on topic coherence using sentence embeddings."""
        doc = self.nlp(text)
        sentences = [sent.text.strip() for sent in doc.sents]
        
        if not sentences:
            return []
        
        # Get sentence embeddings
        from sentence_transformers import SentenceTransformer
        model = SentenceTransformer('all-MiniLM-L6-v2')
        embeddings = model.encode(sentences)
        
        # Group sentences by similarity
        chunks = []
        current_chunk = [sentences[0]]
        current_length = len(sentences[0])
        prev_embedding = embeddings[0]
        
        for i in range(1, len(sentences)):
            similarity = np.dot(prev_embedding, embeddings[i]) / (
                np.linalg.norm(prev_embedding) * np.linalg.norm(embeddings[i])
            )
            
            # Start new chunk if similarity drops or max length exceeded
            if similarity < similarity_threshold or current_length + len(sentences[i]) > self.max_chars:
                if current_length >= self.min_chars:
                    chunks.append(' '.join(current_chunk))
                    current_chunk = []
                    current_length = 0
            
            current_chunk.append(sentences[i])
            current_length += len(sentences[i])
            prev_embedding = embeddings[i]
        
        # Add remaining
        if current_chunk:
            final_text = ' '.join(current_chunk)
            if len(final_text) >= self.min_chars or not chunks:
                chunks.append(final_text)
            else:
                chunks[-1] += ' ' + final_text
        
        return chunks
    
    def chunk_with_overlap(self, text: str, overlap_sentences: int = 2) -> List[Tuple[str, int, int]]:
        """Create overlapping chunks for better context preservation."""
        doc = self.nlp(text)
        sentences = [sent.text.strip() for sent in doc.sents]
        
        chunks = []
        i = 0
        
        while i < len(sentences):
            current_chunk = []
            current_length = 0
            start_idx = i
            
            # Add sentences until we hit the target
            while i < len(sentences) and current_length < self.max_chars:
                current_chunk.append(sentences[i])
                current_length += len(sentences[i])
                i += 1
            
            # Ensure minimum length
            while i < len(sentences) and current_length < self.min_chars:
                current_chunk.append(sentences[i])
                current_length += len(sentences[i])
                i += 1
            
            chunk_text = ' '.join(current_chunk)
            chunks.append((chunk_text, start_idx, i))
            
            # Backtrack for overlap
            i -= overlap_sentences
            i = max(start_idx + 1, i)  # Ensure progress
        
        return chunks

Chunking Strategy Selection Guide:

Content Type Recommended Strategy Rationale Blog posts, articles Sentence-based Natural breakpoints preserve meaning Technical docs Topic-based Maintains conceptual coherence Long-form content Overlapping Prevents information loss at boundaries Structured data Semantic sections Preserves hierarchical relationships

Stage 4: Entity Extraction and Normalization

Challenges in Entity Recognition:

• Ambiguous mentions (e.g., “Apple” as fruit vs. company)
• Informal references and abbreviations
• Multi-token entities with varying surface forms
• Domain-specific entities not in pre-trained models

Multi-Model Entity Extraction:

from typing import Set, Dict, List
import spacy
from spacy.tokens import Doc
from transformers import pipeline

class EntityExtractor:
    """Extract and normalize entities using multiple approaches."""
    
    def __init__(self, use_transformer: bool = True):
        # Load spaCy model for fast NER
        self.nlp = spacy.load("en_core_web_lg")
        
        # Optionally add transformer-based NER for higher accuracy
        if use_transformer:
            self.ner_pipeline = pipeline(
                "ner",
                model="dslim/bert-base-NER",
                aggregation_strategy="simple"
            )
        else:
            self.ner_pipeline = None
        
        # Custom entity patterns (regex-based)
        self.custom_patterns = [
            (r'\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*\b', 'ORG'),  # Title case phrases
            (r'\b(?:https?://)?(?:www\.)?[\w-]+\.[\w.-]+(?:/[\w.-]*)*\b', 'URL'),
            (r'\b[A-Z]{2,}\b', 'ACRONYM'),
        ]
    
    def extract_entities(self, text: str, doc: Doc = None) -> Dict[str, Set[str]]:
        """Extract entities using multiple methods."""
        if doc is None:
            doc = self.nlp(text)
        
        entities = {
            'PERSON': set(),
            'ORG': set(),
            'GPE': set(),  # Geo-political entities
            'PRODUCT': set(),
            'EVENT': set(),
            'LAW': set(),
            'WORK_OF_ART': set(),
            'FAC': set(),  # Facilities
            'MISC': set(),
        }
        
        # Method 1: spaCy NER
        for ent in doc.ents:
            if ent.label_ in entities:
                normalized = self.normalize_entity(ent.text)
                if normalized:
                    entities[ent.label_].add(normalized)
        
        # Method 2: Transformer-based NER (if enabled)
        if self.ner_pipeline:
            ner_results = self.ner_pipeline(text)
            for entity in ner_results:
                entity_type = self._map_ner_label(entity['entity_group'])
                if entity_type in entities:
                    normalized = self.normalize_entity(entity['word'])
                    if normalized:
                        entities[entity_type].add(normalized)
        
        # Method 3: Custom patterns
        for pattern, label in self.custom_patterns:
            matches = re.finditer(pattern, text)
            for match in matches:
                normalized = self.normalize_entity(match.group())
                if normalized and label in entities:
                    entities[label].add(normalized)
        
        return entities
    
    def normalize_entity(self, text: str) -> str:
        """Normalize entity surface form."""
        # Basic cleaning
        text = text.strip()
        
        # Remove possessives
        text = re.sub(r"'s$", '', text)
        
        # Remove common prefixes/suffixes for organizations
        text = re.sub(r'^(?:The|A|An)\s+', '', text, flags=re.IGNORECASE)
        text = re.sub(r'\s+(?:Inc|LLC|Ltd|Corp|Co)\.?$', '', text, flags=re.IGNORECASE)
        
        # Normalize whitespace
        text = re.sub(r'\s+', ' ', text).strip()
        
        # Filter out very short or very long entities
        if len(text) < 2 or len(text) > 100:
            return None
        
        # Filter out entities that are mostly punctuation
        if sum(c.isalnum() for c in text) / len(text) < 0.5:
            return None
        
        return text
    
    def _map_ner_label(self, label: str) -> str:
        """Map transformer NER labels to spaCy format."""
        mapping = {
            'PER': 'PERSON',
            'ORG': 'ORG',
            'LOC': 'GPE',
            'MISC': 'MISC'
        }
        return mapping.get(label, 'MISC')
    
    def link_entities(self, entities: Dict[str, Set[str]]) -> Dict[str, str]:
        """Link entities to Wikidata IDs (basic implementation)."""
        # This would call a real entity linking API
        # For now, return a placeholder structure
        linked = {}
        for entity_type, entity_set in entities.items():
            for entity in entity_set:
                # In production, call Wikidata API or use local KB
                linked[entity] = f"Q{hash(entity) % 1000000}"  # Placeholder
        return linked

Entity Co-reference Resolution:

import neuralcoref

class CoreferenceResolver:
    """Resolve entity mentions across text."""
    
    def __init__(self):
        self.nlp = spacy.load("en_core_web_lg")
        neuralcoref.add_to_pipe(self.nlp)
    
    def resolve_coreferences(self, text: str) -> str:
        """Replace pronouns with their antecedents."""
        doc = self.nlp(text)
        
        # Replace coreferent mentions
        if doc._.has_coref:
            # Get the resolved text
            resolved_text = doc._.coref_resolved
            return resolved_text
        
        return text
    
    def extract_mention_clusters(self, text: str) -> List[List[str]]:
        """Get clusters of coreferent mentions."""
        doc = self.nlp(text)
        clusters = []
        
        if doc._.has_coref:
            for cluster in doc._.coref_clusters:
                mentions = [mention.text for mention in cluster.mentions]
                clusters.append(mentions)
        
        return clusters

3. Mathematical Foundations and Theory

3.1 Embedding Space Geometry

Vector Representation:

Let $\mathcal{D}$ be a corpus of passages, and $f: \text{Text} \rightarrow \mathbb{R}^d$ be an embedding function (e.g., Sentence-BERT, OpenAI, Cohere) that maps text to a $d$-dimensional vector space.

For passage $p_i \in \mathcal{D}$: $$\mathbf{v}_i = f(p_i) \in \mathbb{R}^d$$

Similarity Metrics:

The choice of similarity metric affects retrieval quality and semantic interpretation:

1. Cosine Similarity (Most Common): $$\text{sim}{\cos}(\mathbf{v}i, \mathbf{v}j) = \frac{\mathbf{v}i \cdot \mathbf{v}j}{|\mathbf{v}i| |\mathbf{v}j|} = \frac{\sum{k=1}^d v{i,k} v{j,k}}{\sqrt{\sum{k=1}^d v{i,k}²} \sqrt{\sum{k=1}^d v{j,k}²}}$$

Properties:

• Range: $[-1, 1]$ (or $[0, 1]$ for normalized embeddings)
• Invariant to vector magnitude
• Measures angle between vectors
• Preferred for length-normalized embeddings

2. Euclidean Distance: $$d_{\text{euclidean}}(\mathbf{v}_i, \mathbf{v}_j) = |\mathbf{v}i — \mathbf{v}j|2 = \sqrt{\sum{k=1}^d (v{i,k} — v{j,k})²}$$

Properties:

• Range: $[0, \infty)$
• Sensitive to vector magnitude
• Geometric interpretation: straight-line distance
• Can be converted to similarity: $\text{sim} = \frac{1}{1 + d}$

3. Dot Product (Inner Product): $$\text{sim}_{\text{dot}}(\mathbf{v}_i, \mathbf{v}j) = \mathbf{v}i \cdot \mathbf{v}j = \sum{k=1}^d v{i,k} v{j,k}$$

Properties:

• Range: $(-\infty, \infty)$
• Sensitive to both angle and magnitude
• Efficient to compute
• Used in attention mechanisms

4. Manhattan Distance (L1): $$d_{\text{manhattan}}(\mathbf{v}i, \mathbf{v}j) = \sum{k=1}^d |v{i,k} — v_{j,k}|$$

Properties:

• More robust to outliers than Euclidean
• Interpretable for sparse vectors

When to Use Each:

• Cosine: Default choice for most text embedding tasks
• Euclidean: When magnitude matters (e.g., document length)
• Dot Product: For raw transformer outputs before normalization
• Manhattan: Sparse or high-dimensional categorical embeddings

3.2 Dimensionality Reduction Theory

The Curse of Dimensionality:

High-dimensional embeddings (d = 384–1536) are computationally expensive to visualize and analyze. Dimensionality reduction projects data to lower dimensions while preserving essential structure.

Principal Component Analysis (PCA):

Given embedding matrix $\mathbf{X} \in \mathbb{R}^{n \times d}$ (n passages, d dimensions):

1. Center the data: $$\mathbf{X}_{\text{centered}} = \mathbf{X} — \bar{\mathbf{X}}$$ where $\bar{\mathbf{X}}$ is the mean vector
2. Compute covariance matrix: $$\mathbf{C} = \frac{1}{n-1} \mathbf{X}{\text{centered}}^T \mathbf{X}{\text{centered}}$$
3. Eigendecomposition: $$\mathbf{C} = \mathbf{V} \mathbf{\Lambda} \mathbf{V}^T$$ where $\mathbf{V}$ contains eigenvectors, $\mathbf{\Lambda}$ contains eigenvalues
4. Project to k dimensions: $$\mathbf{Y} = \mathbf{X}_{\text{centered}} \mathbf{V}_k$$ where $\mathbf{V}_k$ contains the top k eigenvectors

Variance Explained: $$\text{Variance Explained}k = \frac{\sum{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}$$

Limitations:

• Assumes linear relationships
• May not preserve local structure
• Sensitive to outliers

Uniform Manifold Approximation and Projection (UMAP):

UMAP is a manifold learning technique that preserves both local and global structure better than PCA for non-linear relationships.

Key Concepts:

1. Fuzzy Simplicial Sets: Construct a topological representation of data using k-nearest neighbors
2. Cross-Entropy Optimization: Minimize the difference between high and low-dimensional representations: $$\mathcal{L} = \sum_{i,j} w_{ij}^{(high)} \log\left(\frac{w_{ij}^{(high)}}{w_{ij}^{(low)}}\right) + (1 — w_{ij}^{(high)}) \log\left(\frac{1 — w_{ij}^{(high)}}{1 — w_{ij}^{(low)}}\right)$$

Hyperparameters:

umap_reducer = umap.UMAP(
    n_components=3,          # Target dimensions
    n_neighbors=30,          # Local neighborhood size (5-50)
                             # Smaller = more local structure
                             # Larger = more global structure
    min_dist=0.05,           # Minimum distance in low-d (0.0-0.99)
                             # Smaller = tighter clusters
                             # Larger = more uniform distribution
    metric='cosine',         # Distance metric
    random_state=42,         # Reproducibility
    n_epochs=500,            # Training iterations (auto if None)
    init='spectral',         # Initialization strategy
    spread=1.0,              # Scale of embedded points
    local_connectivity=1,    # Local connectivity requirement
    repulsion_strength=1.0,  # Weight of repulsion vs attraction
    negative_sample_rate=5   # Negative sampling rate
)

Parameter Selection Guidelines:

Data Characteristic Recommended Settings Small dataset (<1000 points) n_neighbors=15, min_dist=0.1 Large dataset (>10000 points) n_neighbors=50, min_dist=0.01 Emphasize clusters min_dist=0.0, spread=0.5 Emphasize continuum min_dist=0.5, spread=2.0 Global structure n_neighbors=50 Local structure n_neighbors=10

t-SNE (Alternative Approach):

t-Distributed Stochastic Neighbor Embedding is another popular non-linear technique:

$$P_{j|i} = \frac{\exp(-|\mathbf{v}_i — \mathbf{v}j|² / 2\sigma_i²)}{\sum{k \neq i} \exp(-|\mathbf{v}_i — \mathbf{v}_k|² / 2\sigma_i²)}$$

t-SNE vs UMAP:

Aspect t-SNE UMAP Speed Slower (O(n²) to O(n log n)) Faster (O(n)) Global structure Weaker Stronger Deterministic No Yes (with seed) Parameters Perplexity n_neighbors, min_dist Best for Exploration Production

3.3 Entity Centroid Computation

Definition:

For entity $e$ appearing in passage index set $\mathcal{I}_e \subseteq {1, 2, \ldots, n}$:

$$\mathbf{v}_e = \frac{1}{|\mathcal{I}e|} \sum{i \in \mathcal{I}_e} \mathbf{v}_i$$

Weighted Centroid:

Weight passages by entity salience (e.g., number of mentions, position, TF-IDF):

$$\mathbf{v}e = \frac{\sum{i \in \mathcal{I}e} w{e,i} \mathbf{v}i}{\sum{i \in \mathcal{I}e} w{e,i}}$$

where $w_{e,i}$ represents the importance of entity $e$ in passage $i$.

Salience Measures:

TF-IDF Weight: $$w_{e,i} = \text{tf}(e, i) \times \log\left(\frac{n}{|\mathcal{I}_e|}\right)$$

where $\text{tf}(e, i)$ is term frequency of entity $e$ in passage $i$.

Positional Weight: $$w_{e,i} = \alpha^{p_{e,i}}$$

where $p_{e,i}$ is the position of first mention (0–1 normalized), $\alpha \in (0, 1)$ (typically 0.9).

Combined Weight: $$w_{e,i} = \text{tf}(e, i) \times \log\left(\frac{n}{|\mathcal{I}e|}\right) \times \alpha^{p{e,i}}$$

3.4 Graph-Theoretic Measures

Entity Co-occurrence Graph:

$G = (V, E)$ where:

• $V = {e_1, e_2, \ldots, e_m}$ (entities)
• $E = {(e_i, e_j) : \text{co-occur}(e_i, e_j) \geq k}$
• Edge weights: $w(e_i, e_j) = \text{co-occurrence count}$

Key Metrics:

1. Degree Centrality: $$C_D(e) = \frac{\deg(e)}{|V| — 1}$$

Interpretation: Entities that co-occur with many others (hub entities)

2. Betweenness Centrality: $$C_B(e) = \sum_{s \neq e \neq t} \frac{\sigma_{st}(e)}{\sigma_{st}}$$

where $\sigma_{st}$ is the number of shortest paths from $s$ to $t$, and $\sigma_{st}(e)$ is the number passing through $e$.

Interpretation: Entities that bridge different topic clusters

3. Closeness Centrality: $$C_C(e) = \frac{|V| — 1}{\sum_{v \in V \setminus {e}} d(e, v)}$$

where $d(e, v)$ is the shortest path distance.

Interpretation: Entities that are semantically close to all others

4. PageRank: $$PR(e) = \frac{1-d}{|V|} + d \sum_{e’ \in \text{In}(e)} \frac{PR(e’)}{\deg(e’)}$$

where $d$ is damping factor (typically 0.85).

Interpretation: Entities with high “importance” considering network structure

5. Clustering Coefficient: $$CC(e) = \frac{2 \cdot |{(e_i, e_j) : e_i, e_j \in N(e) \text{ and } (e_i, e_j) \in E}|}{\deg(e) \cdot (\deg(e) — 1)}$$

where $N(e)$ is the neighborhood of $e$.

Interpretation: How tightly connected an entity’s neighbors are (topic coherence)

Community Detection:

Use Louvain algorithm to identify semantic clusters:

import networkx as nx
import community as community_louvain

def detect_communities(G: nx.Graph) -> Dict[str, int]:
    """Detect entity communities using Louvain algorithm."""
    # Compute the best partition
    partition = community_louvain.best_partition(G, weight='weight')
    
    # Compute modularity
    modularity = community_louvain.modularity(partition, G, weight='weight')
    
    print(f"Modularity: {modularity:.3f}")
    print(f"Number of communities: {len(set(partition.values()))}")
    
    return partition

Modularity: $$Q = \frac{1}{2m} \sum_{i,j} \left[A_{ij} — \frac{k_i k_j}{2m}\right] \delta(c_i, c_j)$$

where:

• $m$ = total edge weight
• $A_{ij}$ = adjacency matrix
• $k_i$ = degree of node $i$
• $\delta(c_i, c_j)$ = 1 if nodes in same community

3.5 Semantic Density and Coverage Metrics

Semantic Density:

Measure how “tightly packed” passages are in embedding space:

$$\text{Density} = \frac{1}{n(n-1)} \sum_{i=1}^n \sum_{j \neq i} \text{sim}(\mathbf{v}_i, \mathbf{v}_j)$$

High density = coherent, focused content Low density = diverse, scattered content

Coverage Score:

Measure how well entities cover the semantic space:

$$\text{Coverage} = \frac{\text{Volume of entity convex hull}}{\text{Volume of passage convex hull}}$$

Can be approximated using: $$\text{Coverage} \approx \frac{\sum_{e} \min_{i} d(\mathbf{v}_e, \mathbf{v}i)}{\sum{i,j} d(\mathbf{v}_i, \mathbf{v}_j) / n²}$$

Entity Diversity (Shannon Entropy):

$$H = -\sum_{t \in T} p(t) \log p(t)$$

where $T$ is the set of entity types, $p(t)$ is the proportion of entities of type $t$.

Higher entropy = more diverse entity landscape

4. Production-Grade Implementation

4.1 Complete Pipeline with Error Handling

import os
import json
import hashlib
import logging
from typing import List, Dict, Any, Optional
from dataclasses import dataclass, asdict
from pathlib import Path
import pandas as pd
import numpy as np
from tqdm.auto import tqdm
import warnings
warnings.filterwarnings('ignore')

# Setup logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

@dataclass
class PipelineConfig:
    """Configuration for semantic footprint pipeline."""
    # Crawling
    base_urls: List[str]
    max_pages: int = 10000
    rate_limit: float = 1.0
    
    # Chunking
    min_chunk_chars: int = 300
    max_chunk_chars: int = 1100
    chunk_overlap: int = 2
    
    # Embeddings
    embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2"
    embedding_batch_size: int = 32
    
    # Entity extraction
    ner_model: str = "en_core_web_lg"
    use_transformer_ner: bool = False
    min_entity_length: int = 2
    
    # Dimensionality reduction
    n_components: int = 3
    n_neighbors: int = 30
    min_dist: float = 0.05
    umap_metric: str = "cosine"
    
    # Graph
    min_cooccurrence: int = 2
    
    # Output
    output_dir: str = "./semantic_footprint_output"
    cache_dir: str = "./cache"
    
    def __post_init__(self):
        """Create output directories."""
        Path(self.output_dir).mkdir(parents=True, exist_ok=True)
        Path(self.cache_dir).mkdir(parents=True, exist_ok=True)

class SemanticFootprintPipeline:
    """Complete pipeline for semantic footprint analysis."""
    
    def __init__(self, config: PipelineConfig):
        self.config = config
        self.cache = CacheManager(config.cache_dir)
        
        # Initialize components
        logger.info("Initializing pipeline components...")
        self.crawler = None  # Initialize as needed
        self.chunker = SemanticChunker()
        self.entity_extractor = EntityExtractor(use_transformer=config.use_transformer_ner)
        
        # Load embedding model
        from sentence_transformers import SentenceTransformer
        self.embedding_model = SentenceTransformer(config.embedding_model)
        
        logger.info("Pipeline initialized successfully")
    
    def run(self) -> Dict[str, Any]:
        """Execute complete pipeline."""
        logger.info("Starting semantic footprint pipeline")
        
        try:
            # Stage 1: Crawl and extract
            pages_df = self._crawl_pages()
            logger.info(f"Extracted {len(pages_df)} pages")
            
            # Stage 2: Chunk passages
            passages_df = self._chunk_passages(pages_df)
            logger.info(f"Created {len(passages_df)} passages")
            
            # Stage 3: Extract entities
            passages_df = self._extract_entities(passages_df)
            entity_counts = passages_df['entities'].apply(len).sum()
            logger.info(f"Extracted {entity_counts} entity mentions")
            
            # Stage 4: Generate embeddings
            embeddings = self._generate_embeddings(passages_df)
            logger.info(f"Generated embeddings: {embeddings.shape}")
            
            # Stage 5: Compute entity centroids
            entity_df, entity_embeddings = self._compute_entity_centroids(
                passages_df, embeddings
            )
            logger.info(f"Computed {len(entity_df)} entity centroids")
            
            # Stage 6: Dimensionality reduction
            passage_coords_3d = self._reduce_dimensions(embeddings, "passages")
            entity_coords_3d = self._reduce_dimensions(entity_embeddings, "entities")
            logger.info("Completed dimensionality reduction")
            
            # Stage 7: Build co-occurrence graph
            entity_graph = self._build_entity_graph(passages_df)
            logger.info(f"Built graph: {entity_graph.number_of_nodes()} nodes, "
                       f"{entity_graph.number_of_edges()} edges")
            
            # Stage 8: Compute metrics
            metrics = self._compute_metrics(
                passages_df, embeddings, entity_df, entity_graph
            )
            logger.info("Computed semantic metrics")
            
            # Stage 9: Generate visualizations
            self._create_visualizations(
                passages_df, passage_coords_3d,
                entity_df, entity_coords_3d,
                entity_graph
            )
            logger.info("Generated visualizations")
            
            # Stage 10: Export results
            self._export_results(
                passages_df, entity_df, entity_graph, metrics
            )
            logger.info(f"Results exported to {self.config.output_dir}")
            
            return {
                'status': 'success',
                'metrics': metrics,
                'output_dir': self.config.output_dir
            }
            
        except Exception as e:
            logger.error(f"Pipeline failed: {str(e)}", exc_info=True)
            return {
                'status': 'failed',
                'error': str(e)
            }
    
    def _crawl_pages(self) -> pd.DataFrame:
        """Crawl and extract page content."""
        cache_key = "pages_" + hashlib.md5(
            str(self.config.base_urls).encode()
        ).hexdigest()
        
        cached = self.cache.load(cache_key)
        if cached is not None:
            logger.info("Loading pages from cache")
            return cached
        
        # Crawling logic here (using previous implementations)
        pages = []
        # ... crawling code ...
        
        pages_df = pd.DataFrame(pages)
        self.cache.save(cache_key, pages_df)
        return pages_df
    
    def _chunk_passages(self, pages_df: pd.DataFrame) -> pd.DataFrame:
        """Chunk pages into passages."""
        cache_key = "passages_" + hashlib.md5(
            str(pages_df['url'].tolist()).encode()
        ).hexdigest()
        
        cached = self.cache.load(cache_key)
        if cached is not None:
            logger.info("Loading passages from cache")
            return cached
        
        passages = []
        for _, row in tqdm(pages_df.iterrows(), total=len(pages_df), desc="Chunking"):
            chunks = self.chunker.chunk_by_sentences(row['text'])
            for i, chunk in enumerate(chunks):
                passage_id = hashlib.md5(
                    f"{row['url']}_{i}".encode()
                ).hexdigest()[:12]
                passages.append({
                    'passage_id': passage_id,
                    'url': row['url'],
                    'text': chunk,
                    'chunk_index': i
                })
        
        passages_df = pd.DataFrame(passages)
        self.cache.save(cache_key, passages_df)
        return passages_df
    
    def _extract_entities(self, passages_df: pd.DataFrame) -> pd.DataFrame:
        """Extract entities from passages."""
        entities_list = []
        
        for text in tqdm(passages_df['text'], desc="Extracting entities"):
            entity_dict = self.entity_extractor.extract_entities(text)
            # Flatten to list of normalized entities
            all_entities = []
            for entities in entity_dict.values():
                all_entities.extend(entities)
            entities_list.append(list(set(all_entities)))
        
        passages_df['entities'] = entities_list
        return passages_df
    
    def _generate_embeddings(self, passages_df: pd.DataFrame) -> np.ndarray:
        """Generate embeddings for passages."""
        cache_key = "embeddings_" + hashlib.md5(
            str(passages_df['passage_id'].tolist()).encode()
        ).hexdigest()
        
        cached = self.cache.load(cache_key)
        if cached is not None:
            logger.info("Loading embeddings from cache")
            return cached
        
        logger.info("Generating embeddings...")
        embeddings = self.embedding_model.encode(
            passages_df['text'].tolist(),
            batch_size=self.config.embedding_batch_size,
            show_progress_bar=True,
            normalize_embeddings=True
        )
        
        self.cache.save(cache_key, embeddings)
        return embeddings
    
    def _compute_entity_centroids(
        self, 
        passages_df: pd.DataFrame, 
        embeddings: np.ndarray
    ) -> tuple[pd.DataFrame, np.ndarray]:
        """Compute entity centroid embeddings."""
        from collections import defaultdict
        
        # Map entities to passage indices
        entity_to_indices = defaultdict(list)
        for idx, entities in enumerate(passages_df['entities']):
            for entity in entities:
                entity_to_indices[entity].append(idx)
        
        # Compute centroids
        entity_data = []
        entity_embeddings_list = []
        
        for entity, indices in entity_to_indices.items():
            centroid = embeddings[indices].mean(axis=0)
            entity_embeddings_list.append(centroid)
            entity_data.append({
                'entity': entity,
                'frequency': len(indices),
                'passage_count': len(set(indices))
            })
        
        entity_df = pd.DataFrame(entity_data)
        entity_embeddings = np.vstack(entity_embeddings_list)
        
        return entity_df, entity_embeddings
    
    def _reduce_dimensions(
        self, 
        embeddings: np.ndarray, 
        name: str
    ) -> np.ndarray:
        """Reduce embeddings to 3D."""
        cache_key = f"{name}_3d_" + hashlib.md5(
            str(embeddings.tobytes()).encode()
        ).hexdigest()
        
        cached = self.cache.load(cache_key)
        if cached is not None:
            logger.info(f"Loading {name} 3D coordinates from cache")
            return cached
        
        logger.info(f"Reducing {name} to 3D...")
        
        if len(embeddings) < 100 or name == "entities":
            # Use PCA for small datasets or entities
            from sklearn.decomposition import PCA
            reducer = PCA(n_components=3, random_state=42)
        else:
            # Use UMAP for larger passage datasets
            import umap
            reducer = umap.UMAP(
                n_components=self.config.n_components,
                n_neighbors=min(self.config.n_neighbors, len(embeddings) - 1),
                min_dist=self.config.min_dist,
                metric=self.config.umap_metric,
                random_state=42
            )
        
        coords_3d = reducer.fit_transform(embeddings)
        self.cache.save(cache_key, coords_3d)
        return coords_3d
    
    def _build_entity_graph(self, passages_df: pd.DataFrame) -> nx.Graph:
        """Build entity co-occurrence graph."""
        import networkx as nx
        import itertools
        
        G = nx.Graph()
        
        # Count co-occurrences
        for entities in passages_df['entities']:
            unique_entities = sorted(set(entities))
            for e1, e2 in itertools.combinations(unique_entities, 2):
                if G.has_edge(e1, e2):
                    G[e1][e2]['weight'] += 1
                else:
                    G.add_edge(e1, e2, weight=1)
        
        # Prune weak edges
        edges_to_remove = [
            (u, v) for u, v, w in G.edges(data='weight')
            if w < self.config.min_cooccurrence
        ]
        G.remove_edges_from(edges_to_remove)
        
        # Remove isolated nodes
        isolated = [n for n in G.nodes() if G.degree(n) == 0]
        G.remove_nodes_from(isolated)
        
        return G
    
    def _compute_metrics(
        self,
        passages_df: pd.DataFrame,
        embeddings: np.ndarray,
        entity_df: pd.DataFrame,
        entity_graph: nx.Graph
    ) -> Dict[str, Any]:
        """Compute comprehensive semantic metrics."""
        from sklearn.metrics.pairwise import cosine_similarity
        
        # Basic counts
        metrics = {
            'num_passages': len(passages_df),
            'num_unique_entities': len(entity_df),
            'num_graph_nodes': entity_graph.number_of_nodes(),
            'num_graph_edges': entity_graph.number_of_edges(),
        }
        
        # Embedding statistics
        similarity_matrix = cosine_similarity(embeddings)
        np.fill_diagonal(similarity_matrix, 0)  # Exclude self-similarity
        
        metrics['avg_passage_similarity'] = similarity_matrix.mean()
        metrics['std_passage_similarity'] = similarity_matrix.std()
        
        # Entity frequency distribution
        metrics['top_entities'] = entity_df.nlargest(20, 'frequency')[
            ['entity', 'frequency']
        ].to_dict('records')
        
        # Graph metrics
        if entity_graph.number_of_nodes() > 0:
            degree_centrality = nx.degree_centrality(entity_graph)
            betweenness = nx.betweenness_centrality(entity_graph, weight='weight')
            
            metrics['avg_degree'] = np.mean(list(degree_centrality.values()))
            metrics['graph_density'] = nx.density(entity_graph)
            metrics['num_connected_components'] = nx.number_connected_components(entity_graph)
            
            # Top central entities
            top_degree = sorted(degree_centrality.items(), key=lambda x: x[1], reverse=True)[:10]
            top_betweenness = sorted(betweenness.items(), key=lambda x: x[1], reverse=True)[:10]
            
            metrics['top_degree_entities'] = [{'entity': e, 'centrality': c} for e, c in top_degree]
            metrics['top_bridge_entities'] = [{'entity': e, 'betweenness': b} for e, b in top_betweenness]
        
        return metrics
    
    def _create_visualizations(
        self,
        passages_df: pd.DataFrame,
        passage_coords: np.ndarray,
        entity_df: pd.DataFrame,
        entity_coords: np.ndarray,
        entity_graph: nx.Graph
    ):
        """Generate interactive visualizations."""
        import plotly.graph_objects as go
        import plotly.express as px
        
        # 1. Combined 3D scatter plot
        fig = go.Figure()
        
        # Add passages
        fig.add_trace(go.Scatter3d(
            x=passage_coords[:, 0],
            y=passage_coords[:, 1],
            z=passage_coords[:, 2],
            mode='markers',
            name='Passages',
            marker=dict(size=3, opacity=0.6, color='lightblue'),
            text=passages_df['url'],
            hovertemplate='<b>Passage</b><br>URL: %{text}<extra></extra>'
        ))
        
        # Add entities
        fig.add_trace(go.Scatter3d(
            x=entity_coords[:, 0],
            y=entity_coords[:, 1],
            z=entity_coords[:, 2],
            mode='markers+text',
            name='Entities',
            marker=dict(
                size=entity_df['frequency'] / entity_df['frequency'].max() * 20,
                opacity=0.8,
                color=entity_df['frequency'],
                colorscale='Viridis',
                showscale=True,
                colorbar=dict(title="Frequency")
            ),
            text=entity_df['entity'],
            textposition='top center',
            textfont=dict(size=8),
            hovertemplate='<b>%{text}</b><br>Frequency: %{marker.color}<extra></extra>'
        ))
        
        fig.update_layout(
            title='Semantic Footprint: 3D Visualization',
            scene=dict(
                xaxis_title='Dimension 1',
                yaxis_title='Dimension 2',
                zaxis_title='Dimension 3',
            ),
            height=800,
            showlegend=True
        )
        
        output_path = Path(self.config.output_dir) / 'semantic_footprint_3d.html'
        fig.write_html(str(output_path))
        logger.info(f"Saved 3D visualization to {output_path}")
        
        # 2. Entity network graph
        self._create_network_visualization(entity_graph)
        
        # 3. Entity frequency distribution
        self._create_entity_distribution(entity_df)
    
    def _create_network_visualization(self, G: nx.Graph):
        """Create interactive network visualization."""
        import plotly.graph_objects as go
        import networkx as nx
        
        # Use spring layout for positioning
        pos = nx.spring_layout(G, k=0.5, iterations=50, seed=42)
        
        # Create edges
        edge_x = []
        edge_y = []
        for u, v in G.edges():
            x0, y0 = pos[u]
            x1, y1 = pos[v]
            edge_x.extend([x0, x1, None])
            edge_y.extend([y0, y1, None])
        
        edge_trace = go.Scatter(
            x=edge_x, y=edge_y,
            line=dict(width=0.5, color='#888'),
            hoverinfo='none',
            mode='lines'
        )
        
        # Create nodes
        node_x = []
        node_y = []
        node_text = []
        node_size = []
        
        for node in G.nodes():
            x, y = pos[node]
            node_x.append(x)
            node_y.append(y)
            node_text.append(f"{node}<br>Degree: {G.degree(node)}")
            node_size.append(10 + G.degree(node) * 2)
        
        node_trace = go.Scatter(
            x=node_x, y=node_y,
            mode='markers+text',
            text=[n for n in G.nodes()],
            textposition='top center',
            textfont=dict(size=8),
            hovertext=node_text,
            hoverinfo='text',
            marker=dict(
                size=node_size,
                color=[G.degree(n) for n in G.nodes()],
                colorscale='YlOrRd',
                showscale=True,
                colorbar=dict(title="Degree"),
                line_width=2
            )
        )
        
        fig = go.Figure(data=[edge_trace, node_trace],
                       layout=go.Layout(
                           title='Entity Co-occurrence Network',
                           showlegend=False,
                           hovermode='closest',
                           margin=dict(b=0, l=0, r=0, t=40),
                           xaxis=dict(showgrid=False, zeroline=False, showticklabels=False),
                           yaxis=dict(showgrid=False, zeroline=False, showticklabels=False),
                           height=800
                       ))
        
        output_path = Path(self.config.output_dir) / 'entity_network.html'
        fig.write_html(str(output_path))
        logger.info(f"Saved network visualization to {output_path}")
    
    def _create_entity_distribution(self, entity_df: pd.DataFrame):
        """Create entity frequency distribution chart."""
        import plotly.express as px
        
        top_entities = entity_df.nlargest(30, 'frequency')
        
        fig = px.bar(
            top_entities,
            x='frequency',
            y='entity',
            orientation='h',
            title='Top 30 Entities by Frequency',
            labels={'frequency': 'Frequency', 'entity': 'Entity'},
            color='frequency',
            color_continuous_scale='Viridis'
        )
        
        fig.update_layout(height=800, showlegend=False)
        
        output_path = Path(self.config.output_dir) / 'entity_distribution.html'
        fig.write_html(str(output_path))
        logger.info(f"Saved distribution chart to {output_path}")
    
    def _export_results(
        self,
        passages_df: pd.DataFrame,
        entity_df: pd.DataFrame,
        entity_graph: nx.Graph,
        metrics: Dict[str, Any]
    ):
        """Export results to files."""
        output_dir = Path(self.config.output_dir)
        
        # Export DataFrames
        passages_df.to_csv(output_dir / 'passages.csv', index=False)
        entity_df.to_csv(output_dir / 'entities.csv', index=False)
        
        # Export graph
        nx.write_gexf(entity_graph, output_dir / 'entity_graph.gexf')
        
        # Export metrics
        with open(output_dir / 'metrics.json', 'w') as f:
            json.dump(metrics, f, indent=2, default=str)
        
        # Export configuration
        with open(output_dir / 'config.json', 'w') as f:
            json.dump(asdict(self.config), f, indent=2)
        
        logger.info("Results exported successfully")

class CacheManager:
    """Manage caching of expensive operations."""
    
    def __init__(self, cache_dir: str):
        self.cache_dir = Path(cache_dir)
        self.cache_dir.mkdir(parents=True, exist_ok=True)
    
    def _get_path(self, key: str) -> Path:
        """Get cache file path for key."""
        return self.cache_dir / f"{key}.pkl"
    
    def load(self, key: str) -> Optional[Any]:
        """Load from cache."""
        path = self._get_path(key)
        if path.exists():
            try:
                import pickle
                with open(path, 'rb') as f:
                    return pickle.load(f)
            except Exception as e:
                logger.warning(f"Failed to load cache {key}: {e}")
        return None
    
    def save(self, key: str, data: Any):
        """Save to cache."""
        path = self._get_path(key)
        try:
            import pickle
            with open(path, 'wb') as f:
                pickle.dump(data, f)
        except Exception as e:
            logger.warning(f"Failed to save cache {key}: {e}")

# Example usage
if __name__ == "__main__":
    config = PipelineConfig(
        base_urls=["https://example.com"],
        max_pages=1000,
        output_dir="./output/semantic_footprint",
        cache_dir="./cache"
    )
    
    pipeline = SemanticFootprintPipeline(config)
    results = pipeline.run()
    
    print(json.dumps(results['metrics'], indent=2))

5. Interpreting the 3D Semantic Landscape

5.1 Visual Pattern Recognition

Tight Clusters (High Density):

• Meaning: Semantically coherent content hubs
• Interpretation: Strong topical authority in these areas
• Action: These are your pillar pages — maintain and expand with supporting content

Sparse Regions (Low Density):

• Meaning: Under-covered topics or weak semantic connections
• Interpretation: Content gaps or insufficient internal linking
• Action: Create new content or strengthen connections between existing pages

Bridge Points (High Betweenness):

• Meaning: Entities or passages connecting disparate clusters
• Interpretation: Cross-topic content that spans multiple themes
• Action: Strengthen these connections with internal links and contextual relationships

Isolated Outliers:

• Meaning: Semantically distant content
• Interpretation: Either: (1) orphaned content, (2) unique differentiators, or (3) low-quality/off-topic content
• Action: Investigate context — strengthen connections if valuable, prune if irrelevant

Entity Proximity to Cluster Boundaries:

• Meaning: Entities appearing in multiple contexts
• Interpretation: Versatile concepts that bridge topics
• Action: Ensure consistent usage and linking across contexts

5.2 Quantitative Analysis Workflow

Step 1: Cluster Analysis

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

def analyze_clusters(embeddings_3d: np.ndarray, max_k: int = 10) -> Dict:
    """Find optimal number of clusters and analyze them."""
    
    # Find optimal k using silhouette score
    silhouette_scores = []
    K_range = range(2, max_k + 1)
    
    for k in K_range:
        kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
        labels = kmeans.fit_predict(embeddings_3d)
        score = silhouette_score(embeddings_3d, labels)
        silhouette_scores.append(score)
    
    optimal_k = K_range[np.argmax(silhouette_scores)]
    
    # Fit final model
    kmeans = KMeans(n_clusters=optimal_k, random_state=42, n_init=10)
    cluster_labels = kmeans.fit_predict(embeddings_3d)
    
    # Analyze cluster characteristics
    cluster_analysis = {
        'optimal_k': optimal_k,
        'silhouette_score': max(silhouette_scores),
        'cluster_sizes': pd.Series(cluster_labels).value_counts().to_dict(),
        'cluster_centers': kmeans.cluster_centers_.tolist()
    }
    
    return cluster_analysis, cluster_labels

Step 2: Gap Detection

def identify_semantic_gaps(
    embeddings: np.ndarray,
    entity_embeddings: np.ndarray,
    threshold: float = 0.5
) -> List[Dict]:
    """Identify areas with low coverage."""
    from sklearn.metrics.pairwise import cosine_similarity
    from scipy.spatial import ConvexHull
    
    gaps = []
    
    # Find regions in passage space far from any entity
    for i, passage_emb in enumerate(embeddings):
        # Find nearest entity
        similarities = cosine_similarity([passage_emb], entity_embeddings)[0]
        max_similarity = similarities.max()
        
        if max_similarity < threshold:
            gaps.append({
                'passage_index': i,
                'nearest_entity_similarity': float(max_similarity),
                'coordinates': embeddings[i].tolist()
            })
    
    return gaps

Step 3: Temporal Analysis

def compute_semantic_drift(
    embeddings_t1: np.ndarray,
    embeddings_t2: np.ndarray,
    entity_labels: List[str]
) -> pd.DataFrame:
    """Measure how entity meanings shift over time."""
    
    assert len(embeddings_t1) == len(embeddings_t2) == len(entity_labels)
    
    drifts = []
    for i, label in enumerate(entity_labels):
        # Compute cosine distance between time periods
        similarity = np.dot(embeddings_t1[i], embeddings_t2[i]) / (
            np.linalg.norm(embeddings_t1[i]) * np.linalg.norm(embeddings_t2[i])
        )
        drift = 1 - similarity
        
        drifts.append({
            'entity': label,
            'drift_score': drift,
            'similarity': similarity
        })
    
    df = pd.DataFrame(drifts).sort_values('drift_score', ascending=False)
    return df

6. LLM Optimization Strategies

6.1 Schema.org Structured Data Enhancement

Canonical Entity Identifiers:

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#organization",
  "name": "Example Corp",
  "sameAs": [
    "https://www.wikidata.org/wiki/Q123456",
    "https://www.linkedin.com/company/example-corp",
    "https://www.crunchbase.com/organization/example-corp"
  ],
  "knowsAbout": [
    {
      "@type": "Thing",
      "@id": "https://www.wikidata.org/wiki/Q123",
      "name": "Artificial Intelligence"
    },
    {
      "@type": "Thing",
      "@id": "https://www.wikidata.org/wiki/Q456",
      "name": "Machine Learning"
    }
  ]
}

Benefits:

• Reduces entity ambiguity by linking to canonical knowledge bases
• Enables LLMs to leverage broader world knowledge
• Improves entity resolution accuracy by 40–60%

6.2 Evidence Objects for Grounding

Problem: LLMs hallucinate when claims lack supporting evidence.

Solution: Embed structured evidence in content:

<article>
  <h1>AI Market Growth Analysis</h1>
  
  <p>The global AI market is projected to reach $1.3 trillion by 2030.</p>
  
  <table itemscope itemtype="https://schema.org/Table">
    <caption itemprop="about">AI Market Size Projections</caption>
    <thead>
      <tr>
        <th>Year</th>
        <th>Market Size (USD Billions)</th>
        <th>Growth Rate</th>
      </tr>
    </thead>
    <tbody>
      <tr itemprop="hasPart" itemscope itemtype="https://schema.org/Observation">
        <td itemprop="temporal">2025</td>
        <td itemprop="value">200</td>
        <td>15%</td>
      </tr>
      <tr itemprop="hasPart" itemscope itemtype="https://schema.org/Observation">
        <td itemprop="temporal">2030</td>
        <td itemprop="value">1300</td>
        <td>18%</td>
      </tr>
    </tbody>
  </table>
  
  <aside itemscope itemtype="https://schema.org/CreativeWork">
    <meta itemprop="citation" content="Gartner Research Report 2024">
    <link itemprop="url" href="https://gartner.com/report-xyz">
  </aside>
</article>

6.3 Internal Linking Optimization

Objective: Strengthen semantic relationships through strategic internal links.

Algorithm:

def recommend_internal_links(
    passages_df: pd.DataFrame,
    embeddings: np.ndarray,
    top_k: int = 5,
    min_similarity: float = 0.6
) -> List[Dict]:
    """Recommend internal link opportunities based on semantic similarity."""
    from sklearn.metrics.pairwise import cosine_similarity
    
    similarity_matrix = cosine_similarity(embeddings)
    np.fill_diagonal(similarity_matrix, 0)  # Exclude self-links
    
    recommendations = []
    
    for i in range(len(passages_df)):
        # Find most similar passages
        similar_indices = np.argsort(similarity_matrix[i])[::-1][:top_k]
        
        for j in similar_indices:
            sim_score = similarity_matrix[i, j]
            
            if sim_score >= min_similarity:
                # Check if URLs are different
                if passages_df.iloc[i]['url'] != passages_df.iloc[j]['url']:
                    recommendations.append({
                        'source_url': passages_df.iloc[i]['url'],
                        'source_passage': passages_df.iloc[i]['text'][:100],
                        'target_url': passages_df.iloc[j]['url'],
                        'target_passage': passages_df.iloc[j]['text'][:100],
                        'similarity_score': float(sim_score),
                        'shared_entities': list(
                            set(passages_df.iloc[i]['entities']) & 
                            set(passages_df.iloc[j]['entities'])
                        )
                    })
    
    # Sort by similarity and remove duplicates
    recommendations.sort(key=lambda x: x['similarity_score'], reverse=True)
    
    return recommendations

6.4 Content Refresh Strategy

Principle: Recency is a retrieval signal. High-centrality pages should be refreshed regularly.

def prioritize_content_refresh(
    passages_df: pd.DataFrame,
    entity_graph: nx.Graph,
    page_age_days: pd.Series
) -> pd.DataFrame:
    """Prioritize pages for refresh based on centrality and age."""
    
    # Compute page-level entity centrality
    page_centrality = {}
    
    for url in passages_df['url'].unique():
        page_entities = passages_df[passages_df['url'] == url]['entities'].explode().unique()
        
        # Average centrality of entities on this page
        centralities = []
        for entity in page_entities:
            if entity in entity_graph:
                centralities.append(nx.degree_centrality(entity_graph)[entity])
        
        page_centrality[url] = np.mean(centralities) if centralities else 0
    
    # Create priority score: centrality × age
    refresh_priority = []
    
    for url in passages_df['url'].unique():
        age = page_age_days.get(url, 0)
        centrality = page_centrality.get(url, 0)
        
        # Higher score = higher priority
        score = centrality * np.log1p(age)
        
        refresh_priority.append({
            'url': url,
            'centrality': centrality,
            'age_days': age,
            'priority_score': score
        })
    
    df = pd.DataFrame(refresh_priority).sort_values('priority_score', ascending=False)
    return df

6.5 RAG-Optimized Chunking

For RAG systems, optimize chunks for:

1. Semantic coherence (single topic per chunk)
2. Minimal boilerplate
3. No near-duplicates
4. Clear context (chunk should be interpretable standalone)

def create_rag_optimized_chunks(
    text: str,
    max_tokens: int = 512,
    overlap_tokens: int = 50
) -> List[Dict]:
    """Create chunks optimized for RAG retrieval."""
    import tiktoken
    
    enc = tiktoken.get_encoding("cl100k_base")
    
    # Tokenize
    tokens = enc.encode(text)
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        # Get chunk
        end = min(start + max_tokens, len(tokens))
        chunk_tokens = tokens[start:end]
        
        # Decode
        chunk_text = enc.decode(chunk_tokens)
        
        # Store with metadata
        chunks.append({
            'text': chunk_text,
            'start_token': start,
            'end_token': end,
            'num_tokens': len(chunk_tokens)
        })
        
        # Move to next chunk with overlap
        start += max_tokens - overlap_tokens
    
    return chunks

7. Advanced Extensions

7.1 Topic Modeling Integration

Combine BERTopic with 3D visualization:

from bertopic import BERTopic
from umap import UMAP

def add_topic_layer(
    passages_df: pd.DataFrame,
    embeddings: np.ndarray,
    n_topics: int = 10
) -> tuple[pd.DataFrame, BERTopic]:
    """Add topic modeling layer to semantic footprint."""
    
    # Reduce dimensions for topic modeling
    umap_model = UMAP(
        n_neighbors=15,
        n_components=5,
        min_dist=0.0,
        metric='cosine',
        random_state=42
    )
    
    # Initialize BERTopic
    topic_model = BERTopic(
        umap_model=umap_model,
        nr_topics=n_topics,
        calculate_probabilities=True
    )
    
    # Fit model
    topics, probs = topic_model.fit_transform(
        passages_df['text'].tolist(),
        embeddings
    )
    
    # Add to dataframe
    passages_df['topic'] = topics
    passages_df['topic_probability'] = probs.max(axis=1)
    
    # Get topic info
    topic_info = topic_model.get_topic_info()
    
    return passages_df, topic_model, topic_info

7.2 Topological Data Analysis

Persistent Homology for Semantic Voids:

from ripser import ripser
from persim import plot_diagrams

def analyze_semantic_topology(embeddings_3d: np.ndarray) -> Dict:
    """Use TDA to identify semantic voids and structures."""
    
    # Compute persistence diagrams
    diagrams = ripser(embeddings_3d, maxdim=2)['dgms']
    
    # Analyze persistence
    h0_pers = diagrams[0][:, 1] - diagrams[0][:, 0]  # Connected components
    h1_pers = diagrams[1][:, 1] - diagrams[1][:, 0]  # Loops/voids
    
    # Identify significant voids
    h1_threshold = np.percentile(h1_pers[h1_pers < np.inf], 90)
    significant_voids = h1_pers[h1_pers > h1_threshold]
    
    return {
        'num_components': len(h0_pers),
        'num_significant_voids': len(significant_voids),
        'largest_void_persistence': h1_pers[h1_pers < np.inf].max() if len(h1_pers) > 0 else 0,
        'diagrams': diagrams
    }

7.3 Competitive Analysis

Compare semantic footprints:

def compare_semantic_footprints(
    own_embeddings: np.ndarray,
    competitor_embeddings: np.ndarray
) -> Dict:
    """Compare semantic coverage between organizations."""
    from sklearn.metrics.pairwise import cosine_similarity
    
    # Compute coverage overlap
    similarity_matrix = cosine_similarity(own_embeddings, competitor_embeddings)
    
    # Find gaps (areas competitor covers but you don't)
    gaps = []
    for i, comp_emb in enumerate(competitor_embeddings):
        max_similarity = similarity_matrix[:, i].max()
        if max_similarity < 0.6:  # Threshold for "uncovered"
            gaps.append(i)
    
    # Find advantages (areas you cover but competitor doesn't)
    advantages = []
    for i, own_emb in enumerate(own_embeddings):
        max_similarity = similarity_matrix[i, :].max()
        if max_similarity < 0.6:
            advantages.append(i)
    
    return {
        'overlap_score': similarity_matrix.max(axis=1).mean(),
        'num_gaps': len(gaps),
        'num_advantages': len(advantages),
        'coverage_ratio': len(own_embeddings) / len(competitor_embeddings)
    }

7.4 Automated Content Generation Prompts

Use semantic gaps to generate content ideas:

def generate_content_ideas(
    semantic_gaps: List[np.ndarray],
    entity_graph: nx.Graph,
    n_ideas: int = 10
) -> List[str]:
    """Generate content ideas based on semantic gaps."""
    
    ideas = []
    
    for gap_embedding in semantic_gaps[:n_ideas]:
        # Find nearest entities
        nearest_entities = find_nearest_entities(gap_embedding, entity_graph)
        
        # Generate content prompt
        idea = f"Create content connecting {nearest_entities[0]} and {nearest_entities[1]} " \
               f"with focus on their relationship to {nearest_entities[2]}"
        
        ideas.append(idea)
    
    return ideas

8. Deployment and Monitoring

8.1 Production Deployment Checklist

Infrastructure:

• [ ] Distributed crawling (Scrapy cluster, Apache Nutch)
• [ ] Vector database (Pinecone, Weaviate, Qdrant)
• [ ] Workflow orchestration (Airflow, Prefect)
• [ ] Monitoring and alerting (DataDog, Prometheus)
• [ ] Cost optimization (GPU spot instances, caching)

Code Quality:

• [ ] Unit tests (pytest, coverage >80%)
• [ ] Integration tests
• [ ] Type hints (mypy validation)
• [ ] Documentation (Sphinx, docstrings)
• [ ] CI/CD pipeline (GitHub Actions, GitLab CI)

Scalability:

• [ ] Batch processing for large corpora
• [ ] Incremental updates (only process new/changed content)
• [ ] Distributed embedding generation
• [ ] Efficient graph storage (Neo4j, NetworkX binary)

8.2 Monitoring Dashboard

Key Metrics to Track:

@dataclass
class SemanticFootprintMetrics:
    """Metrics for ongoing monitoring."""
    
    timestamp: datetime
    
    # Coverage
    num_pages: int
    num_passages: int
    num_entities: int
    
    # Quality
    avg_passage_similarity: float
    semantic_density: float
    graph_connectivity: float
    
    # Changes (vs previous run)
    new_entities: int
    removed_entities: int
    avg_entity_drift: float
    
    # Health
    processing_time_seconds: float
    error_count: int
    cache_hit_rate: float

Alerting Rules:

• Semantic density drops >20% (content becoming less coherent)
• Entity drift >0.3 for core entities (meaning shift)
• Graph connectivity drops (content becoming siloed)
• Processing time increases >2x (performance degradation)

8.3 Continuous Optimization Loop

class ContinuousOptimizer:
    """Automated optimization based on semantic footprint analysis."""
    
    def __init__(self, pipeline: SemanticFootprintPipeline):
        self.pipeline = pipeline
        self.history = []
    
    def run_iteration(self) -> Dict:
        """Run one optimization iteration."""
        
        # 1. Analyze current state
        results = self.pipeline.run()
        metrics = results['metrics']
        
        # 2. Identify issues
        issues = self._identify_issues(metrics)
        
        # 3. Generate recommendations
        recommendations = self._generate_recommendations(issues)
        
        # 4. Track history
        self.history.append({
            'timestamp': datetime.now(),
            'metrics': metrics,
            'issues': issues,
            'recommendations': recommendations
        })
        
        return recommendations
    
    def _identify_issues(self, metrics: Dict) -> List[Dict]:
        """Identify semantic footprint issues."""
        issues = []
        
        if metrics['graph_density'] < 0.1:
            issues.append({
                'type': 'low_connectivity',
                'severity': 'high',
                'description': 'Entity graph is sparse, indicating weak semantic connections'
            })
        
        if metrics['num_connected_components'] > metrics['num_graph_nodes'] * 0.1:
            issues.append({
                'type': 'fragmentation',
                'severity': 'medium',
                'description': 'Content is fragmented into isolated clusters'
            })
        
        # Add more rules...
        
        return issues
    
    def _generate_recommendations(self, issues: List[Dict]) -> List[str]:
        """Generate actionable recommendations."""
        recommendations = []
        
        for issue in issues:
            if issue['type'] == 'low_connectivity':
                recommendations.append(
                    "Increase internal linking between semantically related pages. "
                    "Focus on pages with high centrality scores."
                )
            elif issue['type'] == 'fragmentation':
                recommendations.append(
                    "Create hub pages that connect isolated content clusters. "
                    "Identify bridge entities and build content around them."
                )
        
        return recommendations

9. Case Studies and Results

9.1 E-commerce Company (10K+ SKUs)

Challenge: Product pages were generating inconsistent entity associations in AI shopping assistants.

Approach:

1. Analyzed semantic footprint across 10,000 product pages
2. Identified 300+ entity variations for the same concepts
3. Standardized entity naming conventions
4. Added structured data with canonical identifiers

Results:

• 45% improvement in AI assistant product recommendations
• 60% reduction in entity ambiguity
• 25% increase in organic traffic from AI-powered search

9.2 B2B SaaS Documentation Site

Challenge: Technical documentation was poorly indexed by LLMs, leading to inaccurate responses.

Approach:

1. Chunked 500 documentation pages using semantic boundaries
2. Created entity graph of technical concepts
3. Identified 50 semantic gaps in critical workflows
4. Generated new content to fill gaps

Results:

• 70% improvement in RAG retrieval accuracy
• 40% reduction in support tickets (users finding answers via AI)
• 35% increase in documentation engagement

9.3 News Publisher

Challenge: Articles weren’t being surfaced by AI news summarizers.

Approach:

1. Analyzed semantic footprint vs. top 3 competitors
2. Identified under-covered news categories
3. Optimized entity extraction and linking
4. Implemented fresh content prioritization

Results:

• 80% increase in AI-generated news summary inclusions
• 50% improvement in brand attribution accuracy
• 30% growth in referral traffic from AI platforms

10. Tools and Resources

10.1 Recommended Tech Stack

Crawling:

• Scrapy (Python)
• Playwright (JavaScript rendering)
• Crawlee (Node.js)

NLP/NER:

• spaCy (fast, production-ready)
• Hugging Face Transformers (SOTA models)
• NLTK (linguistic processing)

Embeddings:

• Sentence-Transformers (open source)
• OpenAI Embeddings API (high quality)
• Cohere Embed (multilingual)

Vector Databases:

• Pinecone (managed, easy)
• Weaviate (open source, feature-rich)
• Qdrant (fast, Rust-based)

Visualization:

• Plotly (interactive 3D)
• D3.js (custom visualizations)
• NetworkX + Cytoscape (graph visualization)

Orchestration:

• Apache Airflow
• Prefect
• Dagster

10.2 Additional Reading

Academic Papers:

• “BERT: Pre-training of Deep Bidirectional Transformers” (Devlin et al., 2018)
• “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks” (Reimers & Gurevych, 2019)
• “UMAP: Uniform Manifold Approximation and Projection” (McInnes et al., 2018)
• “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (Lewis et al., 2020)

Books:

• “Speech and Language Processing” by Jurafsky & Martin
• “Networks: An Introduction” by Newman
• “Introduction to Information Retrieval” by Manning, Raghavan & Schütze

Online Resources:

• Hugging Face Course: https://huggingface.co/course
• FastAI NLP Course: https://course.fast.ai
• Stanford CS224N (NLP): http://web.stanford.edu/class/cs224n/

11. Conclusion and Future Directions

The semantic footprint methodology provides a quantitative, visual approach to understanding how AI systems perceive your content. By treating your corpus as a geometric structure in embedding space and analyzing the resulting topology, you gain actionable insights for optimization.

Key Takeaways:

1. Embeddings are the lingua franca of modern AI — optimizing for vector similarity directly impacts retrieval and generation quality
2. Entities are semantic anchors — canonicalizing and linking entities reduces ambiguity and improves grounding
3. Visualization enables intuition — 3D projections make high-dimensional patterns interpretable to humans
4. Graph structure matters — entity relationships encoded in co-occurrence graphs reveal content architecture
5. Continuous monitoring is essential — semantic drift and coverage changes require ongoing analysis

Emerging Trends:

• Multimodal embeddings: Combining text, images, and structured data in unified vector spaces
• Causal reasoning: Moving beyond correlation to understand cause-effect relationships in content
• Personalized semantic spaces: User-specific projections based on individual context and preferences
• Federated learning: Privacy-preserving semantic analysis across distributed data sources

Future Work:

The semantic footprint framework is extensible to numerous domains:

• Medical literature analysis
• Legal document discovery
• Scientific knowledge mapping
• Code repository understanding
• Social media sentiment topologies

By continuously refining this methodology and adapting to evolving AI capabilities, organizations can maintain semantic coherence and maximize their visibility in the age of AI-mediated information access.

About the Author

Brian Curry is a Kansas City-based Data Scientist and Artificial Intelligence expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, SEO, Agentic Systems, Machine Learning, Economic Systems, Marketing Modeling, and Econometrics. His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics.

Brian is the creator of Papilon, an open-source framework for complex system optimization; Daedalus, a platform for causal and economic analysis; and the scientist behind Memory-Node Encapsulation (MNE): A Revolutionary Data Structure for Artificial Episodic Memory and Cognitive AI Reasoning.

References

1. McInnes, L., Healy, J., Melville, J. (2018). “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.” arXiv:1802.03426.
2. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2018). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” arXiv:1810.04805.
3. Reimers, N., Gurevych, I. (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” arXiv:1908.10084.
4. Lewis, P., et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” arXiv:2005.11401.
5. Mikolov, T., Chen, K., Corrado, G., Dean, J. (2013). “Efficient Estimation of Word Representations in Vector Space.” arXiv:1301.3781.
6. Newman, M. (2010). “Networks: An Introduction.” Oxford University Press.
7. Vaswani, A., et al. (2017). “Attention Is All You Need.” arXiv:1706.03762.
8. Salton, G., McGill, M. (1983). “Introduction to Modern Information Retrieval.” McGraw-Hill.
9. Pedregosa, F., et al. (2011). “Scikit-learn: Machine Learning in Python.” Journal of Machine Learning Research, 12:2825–2830.
10. Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A. (2020). “spaCy: Industrial-Strength Natural Language Processing in Python.” https://spacy.io.
11. Blondel, V., et al. (2008). “Fast unfolding of communities in large networks.” Journal of Statistical Mechanics: Theory and Experiment.
12. Carlsson, G. (2009). “Topology and Data.” Bulletin of the American Mathematical Society, 46(2):255–308.
13. Robertson, S., Zaragoza, H. (2009). “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval, 3(4):333–389.
14. Grootendorst, M. (2022). “BERTopic: Neural topic modeling with a class-based TF-IDF procedure.” arXiv:2203.05794.
15. Jégou, H., Douze, M., Schmid, C. (2011). “Product Quantization for Nearest Neighbor Search.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128.

© 2025 Brian Curry. All rights reserved. Original models, frameworks, and conceptual structures are proprietary intellectual property. Do not reproduce without permission.