The 3D Semantic Footprint: Measuring and Visualizing a Company’s Entity Landscape for LLM Optimization

The 3D Semantic Footprint: Measuring and Visualizing a Company’s Entity Landscape for LLM Optimization

1. Introduction

In an AI-first world, your website isn’t just a set of pages — it’s a semantic lattice, a latent map of concepts, topics, and named entities that large language models (LLMs) use to infer what your company is about. As organizations increasingly compete not just for human attention but for algorithmic understanding, the structure of digital content takes on new significance.

This semantic structure is not visible to the naked eye, but it can be measured, analyzed, and visualized. I call this your semantic footprint: the geometry and graph of your digital content as embedded in vector space, the same way an LLM or retrieval system might perceive it.

Traditional content audits focus on surface-level metrics — page views, bounce rates, keyword density. But these metrics fail to capture how AI systems actually “understand” your content. When a large language model processes your website, it’s not counting keywords — it’s constructing a multidimensional representation of semantic relationships, entity associations, and topical coherence. Understanding this representation is crucial for:

• AI Discoverability: How well can LLMs find and retrieve your content?
• Semantic Coherence: Are your topics logically connected in embedding space?
• Entity Authority: Which concepts and entities dominate your semantic landscape?
• Content Gaps: What important connections or topics are missing?

In this comprehensive article, I walk through a reproducible, extensible method for:

• Extracting and preprocessing your site’s content at scale
• Embedding passages using state-of-the-art transformer models
• Modeling and reducing high-dimensional semantic space
• Constructing entity-level co-occurrence graphs with network analysis
• Visualizing everything in interactive 3D environments
• Computing actionable metrics for semantic optimization
• Applying insights to RAG systems, content strategy, and competitive analysis

This methodology is essential for teams building retrieval-augmented generation (RAG) systems, optimizing for LLM search visibility, performing strategic audits of content and semantic coverage, or preparing organizations for an AI-native digital landscape.

2. Theoretical Foundation

2.1 Text Embeddings: From Words to Vectors

Any chunk of text can be transformed into a fixed-length vector in ℝⁿ, where n is typically 384, 768, or 1024 dimensions depending on the model architecture (e.g., MiniLM, BERT, or newer models like E5).

Let v_a and v_b be two such vectors. Their semantic similarity is most commonly computed using cosine similarity:

sim(a, b) = (v_a · v_b) / (||v_a|| × ||v_b||)

This measures angular similarity in the embedding space, capturing closeness of meaning rather than surface-level lexical overlap. Two passages about “machine learning algorithms” and “artificial intelligence models” might share no words but have high cosine similarity.

Why This Matters: Traditional keyword matching treats “buy running shoes” and “purchase athletic footwear” as completely different. Embeddings recognize their semantic equivalence, which is exactly how modern LLMs understand language.

Mathematical Properties:

• Cosine similarity ranges from -1 (opposite) to +1 (identical)
• Typical related passages score 0.3–0.7
• Unrelated content usually scores < 0.2
• Embeddings capture semantic relationships: king — man + woman ≈ queen

Model Selection Considerations:

• all-MiniLM-L6-v2: Fast, 384 dimensions, good for most applications
• all-mpnet-base-v2: Higher quality, 768 dimensions, slower
• E5-large: State-of-the-art, but requires more compute
• Domain-specific models: Consider fine-tuned models for specialized content (medical, legal, technical)

2.2 Dimensionality Reduction: Making High-Dimensional Data Human-Interpretable

The raw embeddings exist in 384–1024 dimensional space and are not human-interpretable. To visualize and analyze, we reduce the vectors to 2D or 3D using advanced manifold learning techniques.

Let V ∈ ℝ^(m × n) be our matrix of m passage vectors with n dimensions. We reduce it to Y ∈ ℝ^(m × 3) for 3D plotting.

2.2.1 Principal Component Analysis (PCA)

PCA finds orthogonal axes that capture maximum variance through eigendecomposition of the covariance matrix.

Advantages:

• Fast and deterministic
• Preserves global structure
• Linear transformation is interpretable
• Good for initial exploration

Limitations:

• Assumes linear relationships
• May distort local neighborhood structures
• First few components may not capture semantic nuances

When to Use: Quick exploration, large datasets (>100k points), when global structure matters more than local clusters.

2.2.2 UMAP (Uniform Manifold Approximation and Projection)

UMAP constructs a high-dimensional graph representation of the data and optimizes a low-dimensional graph to be as structurally similar as possible.

Advantages:

• Preserves both local and global structure
• Often reveals clearer clusters
• Handles non-linear manifolds
• More faithful to semantic neighborhoods

Key Hyperparameters:

• n_neighbors (5-100): Controls local vs. global structure
• Low values (5–15): Emphasize local clusters
• High values (30–100): Preserve global topology
• min_dist (0.0-0.99): Minimum distance between points
• Lower: Tighter clusters
• Higher: More uniform distribution
• metric: Distance function (cosine, euclidean, etc.)

When to Use: Final visualizations, when cluster separation matters, semantic analysis.

2.2.3 t-SNE (t-Distributed Stochastic Neighbor Embedding)

While not used in the main pipeline, t-SNE is worth understanding for comparison:

Advantages:

• Excellent at revealing clusters
• Widely understood and published

Limitations:

• Computationally expensive (O(n²))
• Non-deterministic
• Doesn’t preserve global structure well
• Perplexity parameter requires tuning

Modern Recommendation: UMAP generally outperforms t-SNE for most semantic analysis tasks.

2.3 Entity Centroids: Aggregating Semantic Meaning Across Mentions

Named entities (products, organizations, people, locations) typically appear in multiple passages throughout your content. To create a unified semantic representation of each entity, we compute its centroid by averaging the embeddings of all passages where it appears:

v_e = (1 / |I_e|) × Σ v_i, for i ∈ I_e

where I_e is the set of passage indices where entity e appears.

Intuition: If “TensorFlow” appears in 47 different passages across your site, each discussing different aspects (installation, tutorials, use cases), the centroid represents the “average semantic meaning” of TensorFlow within your content corpus.

Weighted Centroids: For more sophisticated analysis, weight passages by:

• Recency (newer content weighted more)
• Authority (pages with higher PageRank)
• Engagement (time on page, conversions)
• Semantic diversity (passages covering unique aspects weighted more)

python

weighted_centroid = Σ (w_i × v_i) / Σ w_i

Why This Matters: Entity centroids let you:

• Compare how different entities relate in your semantic space
• Identify entities with unclear or scattered semantic positioning
• Detect entities that appear in isolated content silos
• Measure semantic drift over time as content evolves

2.4 Graph Theory and Co-Occurrence Networks

We construct a semantic co-occurrence graph G = (E, L), where:

• Nodes E: Normalized named entities extracted from content
• Edges L: Connect two entities if they co-occur in the same passage ≥ k times (typically k=2)
• Edge Weights: Co-occurrence frequency (how many times entities appear together)

From this graph structure, we compute several key network metrics:

2.4.1 Degree Centrality

deg(v) = number of edges connected to node v

Interpretation: How many other entities does this entity co-occur with? High degree indicates a “hub” entity that connects many concepts.

Example: In a tech company’s content, “API” might have high degree centrality, appearing alongside authentication, endpoints, rate limits, SDKs, etc.

2.4.2 Betweenness Centrality

betweenness(v) = Σ (σ_st(v) / σ_st)

where σ_st is the number of shortest paths from s to t, and σ_st(v) is the number of those paths passing through v.

Interpretation: How often does this entity lie on the shortest path between other entities? High betweenness indicates a “bridge” connecting different semantic clusters.

Example: “Machine learning” might bridge clusters about “data science” and “product features,” serving as a conceptual connector.

2.4.3 Connected Components

A connected component is a maximal set of nodes where any two nodes are connected by some path.

Interpretation: Isolated connected components indicate semantic silos — groups of entities that never co-occur with entities outside their group.

Red Flags:

• Important product lines existing in separate components
• Acquired brands remaining disconnected from main entity graph
• Technical documentation isolated from marketing content

2.4.4 PageRank on Entity Graph

Applying Google’s PageRank algorithm to the entity graph reveals which entities are most “authoritative” within your content’s semantic structure:

python

pagerank = nx.pagerank(G, weight='weight')

Interpretation: Similar to web PageRank, entities with high scores are frequently referenced and connected to other important entities.

2.4.5 Community Detection

Using algorithms like Louvain or Label Propagation, we can automatically detect semantic communities:

python

communities = nx.community.louvain_communities(G, weight='weight')

Applications:

• Identify natural topic clusters
• Detect misclassified or ambiguous entities
• Guide content restructuring
• Inform information architecture decisions

3. Enhanced Pipeline Overview

The complete semantic footprint pipeline consists of these stages:

3.1 Content Acquisition

1. Crawl the site using sitemap.xml or recursive spider
2. Filter pages by content type, language, or relevance
3. Extract clean, readable text while preserving structure
4. Validate content quality (length, readability, duplicate detection)

3.2 Preprocessing and Chunking

1. Clean HTML artifacts, scripts, navigation elements
2. Normalize whitespace, encoding, special characters
3. Chunk into semantically coherent passages (300–1000 characters)
4. Preserve metadata (URL, title, headers, publish date)

3.3 Entity Recognition

1. Extract named entities using NER models (spaCy, Flair, or custom)
2. Normalize entity names (case, punctuation, abbreviations)
3. Disambiguate ambiguous entities (Apple Inc. vs. apple fruit)
4. Link entities to knowledge bases when possible (Wikidata, DBpedia)

3.4 Semantic Embedding

1. Embed each passage using transformer models
2. Normalize embeddings for cosine similarity
3. Store in vector database for efficient retrieval
4. Compute entity centroids from passage embeddings

3.5 Dimensionality Reduction

1. Apply UMAP or PCA to reduce to 3D
2. Optimize hyperparameters for visual clarity
3. Validate reduction quality (preservation metrics)
4. Create multiple projections for different analyses

3.6 Graph Construction

1. Build entity co-occurrence graph
2. Weight edges by frequency and context
3. Prune weak or spurious connections
4. Compute network metrics and centrality measures

3.7 Clustering and Analysis

1. Apply clustering algorithms (K-means, DBSCAN, HDBSCAN)
2. Label clusters with representative topics or entities
3. Analyze cluster cohesion and separation
4. Identify outliers and anomalies

3.8 Visualization and Reporting

1. Generate interactive 3D visualizations
2. Create 2D projections for specific views
3. Produce network graphs with entity relationships
4. Compile metrics dashboard and recommendations

4. Comprehensive Python Implementation

4.1 Environment Setup and Dependencies

First, create a virtual environment and install all required packages:

bash

# Create virtual environment
python -m venv semantic_env
source semantic_env/bin/activate  # On Windows: semantic_env\Scripts\activate

# Install dependencies
pip install beautifulsoup4 spacy sentence-transformers umap-learn scikit-learn \
            networkx pandas numpy plotly tqdm requests lxml \
            community-detection python-louvain bertopic keybert

# Download spaCy language model
python -m spacy download en_core_web_lg

# For GPU acceleration (optional)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

Dependency Versions (tested configuration):

beautifulsoup4==4.12.2
spacy==3.7.2
sentence-transformers==2.2.2
umap-learn==0.5.5
scikit-learn==1.3.2
networkx==3.2.1
plotly==5.18.0

4.2 Advanced Web Crawling and Content Extraction

python

import requests
import re
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from typing import List, Dict, Set
import logging
from tqdm import tqdm

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class SemanticCrawler:
    """Enhanced web crawler with rate limiting and error handling"""
    
    def __init__(self, base_url: str, max_pages: int = 500, delay: float = 1.0):
        self.base_url = base_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited: Set[str] = set()
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "SemanticFootprintBot/2.0 (+https://vector1.ai/bot)"
        })
    
    def is_valid_url(self, url: str) -> bool:
        """Check if URL should be crawled"""
        parsed = urlparse(url)
        
        # Must be same domain
        if parsed.netloc != urlparse(self.base_url).netloc:
            return False
        
        # Skip common non-content URLs
        skip_patterns = [
            r'/tag/', r'/category/', r'/author/',
            r'\.(pdf|jpg|png|gif|css|js)$',
            r'/wp-admin/', r'/wp-json/',
            r'/feed/', r'/comments/'
        ]
        
        for pattern in skip_patterns:
            if re.search(pattern, url, re.I):
                return False
        
        return True
    
    def fetch_page(self, url: str) -> str:
        """Fetch page with error handling and retries"""
        for attempt in range(3):
            try:
                response = self.session.get(url, timeout=10)
                response.raise_for_status()
                return response.text
            except requests.RequestException as e:
                logger.warning(f"Attempt {attempt + 1} failed for {url}: {e}")
                time.sleep(2 ** attempt)  # Exponential backoff
        
        logger.error(f"Failed to fetch {url} after 3 attempts")
        return ""
    
    def extract_links(self, html: str, current_url: str) -> List[str]:
        """Extract and normalize all links from page"""
        soup = BeautifulSoup(html, 'lxml')
        links = []
        
        for a_tag in soup.find_all('a', href=True):
            href = a_tag['href']
            absolute_url = urljoin(current_url, href)
            
            # Remove fragments and query params for deduplication
            clean_url = absolute_url.split('#')[0].split('?')[0]
            
            if self.is_valid_url(clean_url):
                links.append(clean_url)
        
        return list(set(links))  # Deduplicate
    
    def crawl(self) -> List[Dict[str, str]]:
        """Crawl site and return list of pages"""
        to_visit = [self.base_url]
        pages = []
        
        with tqdm(total=self.max_pages, desc="Crawling pages") as pbar:
            while to_visit and len(pages) < self.max_pages:
                url = to_visit.pop(0)
                
                if url in self.visited:
                    continue
                
                self.visited.add(url)
                html = self.fetch_page(url)
                
                if not html:
                    continue
                
                # Extract content
                clean_content = clean_text(html)
                
                if len(clean_content) >= 200:  # Minimum content threshold
                    pages.append({
                        'url': url,
                        'html': html,
                        'text': clean_content,
                        'title': extract_title(html)
                    })
                    pbar.update(1)
                
                # Find new links to visit
                new_links = self.extract_links(html, url)
                to_visit.extend([link for link in new_links if link not in self.visited])
                
                time.sleep(self.delay)  # Rate limiting
        
        logger.info(f"Crawled {len(pages)} pages from {self.base_url}")
        return pages

def extract_title(html: str) -> str:
    """Extract page title"""
    soup = BeautifulSoup(html, 'lxml')
    
    # Try different title sources
    if soup.title:
        return soup.title.string.strip()
    
    h1 = soup.find('h1')
    if h1:
        return h1.get_text().strip()
    
    og_title = soup.find('meta', property='og:title')
    if og_title:
        return og_title.get('content', '').strip()
    
    return "Untitled"

def clean_text(html: str) -> str:
    """Enhanced text cleaning with better content extraction"""
    soup = BeautifulSoup(html, 'lxml')
    
    # Remove unwanted elements
    for tag in soup(['script', 'style', 'nav', 'header', 'footer', 
                     'aside', 'iframe', 'noscript']):
        tag.decompose()
    
    # Try to find main content
    main_content = (
        soup.find('main') or 
        soup.find('article') or 
        soup.find('div', class_=re.compile(r'content|main|article', re.I)) or
        soup.find('body')
    )
    
    if main_content:
        text = main_content.get_text(separator=' ', strip=True)
    else:
        text = soup.get_text(separator=' ', strip=True)
    
    # Clean whitespace and special characters
    text = re.sub(r'\s+', ' ', text)
    text = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', text)
    
    return text.strip()

4.3 Intelligent Passage Chunking with Context Preservation

python

import re
from typing import List, Dict

def chunk_text_advanced(
    text: str, 
    min_chars: int = 300, 
    max_chars: int = 1000,
    overlap: int = 50,
    preserve_sentences: bool = True
) -> List[str]:
    """
    Advanced chunking with overlap and sentence preservation
    
    Args:
        text: Input text to chunk
        min_chars: Minimum chunk size
        max_chars: Maximum chunk size
        overlap: Character overlap between chunks
        preserve_sentences: Avoid splitting mid-sentence
    """
    
    if preserve_sentences:
        # Split on sentence boundaries
        sentences = re.split(r'(?<=[.!?])\s+', text)
    else:
        # Split on any whitespace
        sentences = text.split()
    
    chunks = []
    current_chunk = []
    current_length = 0
    
    for i, sent in enumerate(sentences):
        sent_length = len(sent)
        
        # If adding this sentence exceeds max_chars
        if current_length + sent_length > max_chars and current_chunk:
            # Finalize current chunk
            chunk_text = ' '.join(current_chunk).strip()
            if len(chunk_text) >= min_chars:
                chunks.append(chunk_text)
            
            # Start new chunk with overlap
            if overlap > 0:
                # Include last few sentences for context
                overlap_sents = []
                overlap_length = 0
                for prev_sent in reversed(current_chunk):
                    if overlap_length + len(prev_sent) <= overlap:
                        overlap_sents.insert(0, prev_sent)
                        overlap_length += len(prev_sent)
                    else:
                        break
                current_chunk = overlap_sents
                current_length = overlap_length
            else:
                current_chunk = []
                current_length = 0
        
        current_chunk.append(sent)
        current_length += sent_length + 1  # +1 for space
    
    # Add final chunk
    if current_chunk:
        chunk_text = ' '.join(current_chunk).strip()
        if len(chunk_text) >= min_chars:
            chunks.append(chunk_text)
    
    return chunks

def create_passages_from_pages(pages: List[Dict[str, str]]) -> List[Dict]:
    """Convert pages into passages with metadata"""
    passages = []
    
    for page in tqdm(pages, desc="Creating passages"):
        chunks = chunk_text_advanced(
            page['text'],
            min_chars=300,
            max_chars=1000,
            overlap=100
        )
        
        for i, chunk in enumerate(chunks):
            passages.append({
                'text': chunk,
                'url': page['url'],
                'title': page['title'],
                'passage_id': f"{page['url']}#{i}",
                'page_position': i,
                'total_passages': len(chunks)
            })
    
    return passages

4.4 Advanced Entity Extraction with Disambiguation

python

import spacy
from spacy.tokens import Span
from typing import List, Set, Dict
import re
from collections import defaultdict

# Load spaCy model
nlp = spacy.load("en_core_web_lg")

# Custom entity patterns for domain-specific terms
CUSTOM_PATTERNS = [
    {"label": "PRODUCT", "pattern": [{"LOWER": "tensorflow"}]},
    {"label": "PRODUCT", "pattern": [{"LOWER": "pytorch"}]},
    {"label": "TECH", "pattern": [{"LOWER": "llm"}, {"IS_ALPHA": True, "OP": "?"}]},
    {"label": "TECH", "pattern": [{"LOWER": "rag"}]},
]

def normalize_entity(entity_text: str) -> str:
    """Normalize entity text for consistency"""
    # Remove possessives
    text = re.sub(r"'s$", '', entity_text)
    
    # Remove common prefixes/suffixes
    text = re.sub(r'^(the|a|an)\s+', '', text, flags=re.I)
    
    # Normalize whitespace and case
    text = re.sub(r'\s+', ' ', text).strip().lower()
    
    # Remove special characters except hyphens and underscores
    text = re.sub(r'[^\w\s\-]', '', text)
    
    return text

def extract_entities_advanced(
    text: str,
    min_length: int = 3,
    entity_types: Set[str] = None
) -> List[Dict[str, str]]:
    """
    Extract and normalize entities with metadata
    
    Args:
        text: Input text
        min_length: Minimum entity length
        entity_types: Set of entity types to extract (None = all)
    
    Returns:
        List of entity dictionaries with text, type, and normalized form
    """
    if entity_types is None:
        entity_types = {'PERSON', 'ORG', 'PRODUCT', 'GPE', 'TECH', 
                       'WORK_OF_ART', 'LAW', 'EVENT', 'FAC'}
    
    doc = nlp(text)
    entities = []
    seen = set()
    
    for ent in doc.ents:
        if ent.label_ not in entity_types:
            continue
        
        if len(ent.text) < min_length:
            continue
        
        normalized = normalize_entity(ent.text)
        
        if normalized and normalized not in seen:
            entities.append({
                'text': ent.text,
                'normalized': normalized,
                'type': ent.label_,
                'start': ent.start_char,
                'end': ent.end_char
            })
            seen.add(normalized)
    
    return entities

class EntityDisambiguator:
    """Disambiguate entities using context and knowledge bases"""
    
    def __init__(self):
        self.entity_contexts = defaultdict(list)
        self.canonical_forms = {}
    
    def add_occurrence(self, entity: str, context: str):
        """Record entity occurrence with context"""
        self.entity_contexts[entity].append(context)
    
    def build_canonical_mapping(self):
        """Build mapping from variants to canonical forms"""
        # Group similar entities
        entity_groups = defaultdict(set)
        
        for entity in self.entity_contexts.keys():
            # Create canonical form (e.g., "google inc" -> "google")
            canonical = self._get_canonical_form(entity)
            entity_groups[canonical].add(entity)
        
        # Map all variants to most common form
        for canonical, variants in entity_groups.items():
            if len(variants) > 1:
                # Choose most frequent variant as canonical
                most_common = max(
                    variants,
                    key=lambda e: len(self.entity_contexts[e])
                )
                for variant in variants:
                    self.canonical_forms[variant] = most_common
            else:
                self.canonical_forms[list(variants)[0]] = canonical
    
    def _get_canonical_form(self, entity: str) -> str:
        """Extract canonical form from entity"""
        # Remove common suffixes
        suffixes = [' inc', ' corp', ' corporation', ' llc', ' ltd',
                   ' company', ' co', ' group', ' international']
        
        canonical = entity.lower()
        for suffix in suffixes:
            canonical = canonical.replace(suffix, '')
        
        return canonical.strip()
    
    def disambiguate(self, entity: str) -> str:
        """Return canonical form of entity"""
        return self.canonical_forms.get(entity, entity)

def process_entities_in_passages(passages: List[Dict]) -> List[Dict]:
    """Extract and disambiguate entities in all passages"""
    disambiguator = EntityDisambiguator()
    
    # First pass: extract entities and collect contexts
    for passage in tqdm(passages, desc="Extracting entities (pass 1)"):
        entities = extract_entities_advanced(passage['text'])
        passage['raw_entities'] = entities
        
        for ent in entities:
            disambiguator.add_occurrence(
                ent['normalized'],
                passage['text']
            )
    
    # Build disambiguation mapping
    disambiguator.build_canonical_mapping()
    
    # Second pass: apply disambiguation
    for passage in tqdm(passages, desc="Disambiguating entities (pass 2)"):
        passage['entities'] = [
            disambiguator.disambiguate(ent['normalized'])
            for ent in passage['raw_entities']
        ]
        passage['entities'] = list(set(passage['entities']))  # Deduplicate
    
    return passages

4.5 Embedding Generation with Batch Processing and Caching

python

from sentence_transformers import SentenceTransformer
import numpy as np
from typing import List, Dict
import pickle
from pathlib import Path
import hashlib
import torch

class EmbeddingGenerator:
    """Generate and cache passage embeddings"""
    
    def __init__(
        self,
        model_name: str = "all-MiniLM-L6-v2",
        cache_dir: str = "./embedding_cache",
        batch_size: int = 32
    ):
        self.model_name = model_name
        self.cache_dir = Path(cache_dir)
        self.cache_dir.mkdir(exist_ok=True)
        self.batch_size = batch_size
        
        # Load model with GPU support if available
        device = 'cuda' if torch.cuda.is_available() else 'cpu'
        self.model = SentenceTransformer(model_name, device=device)
        
        print(f"Loaded {model_name} on {device}")
    
    def _get_cache_key(self, texts: List[str]) -> str:
        """Generate cache key from texts"""
        text_hash = hashlib.md5(''.join(texts).encode()).hexdigest()
        return f"{self.model_name}_{text_hash}.pkl"
    
    def _load_from_cache(self, cache_key: str) -> np.ndarray:
        """Load embeddings from cache"""
        cache_path = self.cache_dir / cache_key
        if cache_path.exists():
            with open(cache_path, 'rb') as f:
                return pickle.load(f)
        return None
    
    def _save_to_cache(self, cache_key: str, embeddings: np.ndarray):
        """Save embeddings to cache"""
        cache_path = self.cache_dir / cache_key
        with open(cache_path, 'wb') as f:
            pickle.dump(embeddings, f)
    
    def encode(
        self,
        texts: List[str],
        use_cache: bool = True,
        show_progress: bool = True
    ) -> np.ndarray:
        """
        Encode texts to embeddings with caching
        
        Args:
            texts: List of text strings to encode
            use_cache: Whether to use cached embeddings
            show_progress: Show progress bar
        
        Returns:
            Array of embeddings (n_texts, embedding_dim)
        """
        cache_key = self._get_cache_key(texts)
        
        # Try loading from cache
        if use_cache:
            cached = self._load_from_cache(cache_key)
            if cached is not None:
                print(f"Loaded {len(texts)} embeddings from cache")
                return cached
        
        # Generate embeddings
        print(f"Generating {len(texts)} embeddings...")
        embeddings = self.model.encode(
            texts,
            batch_size=self.batch_size,
            show_progress_bar=show_progress,
            normalize_embeddings=True,  # L2 normalization for cosine similarity
            convert_to_numpy=True
        )
        
        # Save to cache
        if use_cache:
            self._save_to_cache(cache_key, embeddings)
        
        return embeddings
    
    def compute_entity_centroids(
        self,
        passages: List[Dict],
        passage_embeddings: np.ndarray
    ) -> Dict[str, np.ndarray]:
        """Compute centroid embeddings for each entity"""
        entity_to_indices = defaultdict(list)
        
        # Map entities to passage indices
        for i, passage in enumerate(passages):
            for entity in passage['entities']:
                entity_to_indices[entity].append(i)
        
        # Compute centroids
        entity_centroids = {}
        for entity, indices in tqdm(
            entity_to_indices.items(),
            desc="Computing entity centroids"
        ):
            # Average embeddings of all passages containing this entity
            entity_vectors = passage_embeddings[indices]
            centroid = np.mean(entity_vectors, axis=0)
            
            # Normalize
            centroid = centroid / np.linalg.norm(centroid)
            
            entity_centroids[entity] = {
                'vector': centroid,
                'frequency': len(indices),
                'passages': indices
            }
        
        return entity_centroids

4.6 Advanced Dimensionality Reduction with Optimization

python

import umap
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

class DimensionalityReducer:
    """Optimize and apply dimensionality reduction"""
    
    def __init__(self, method: str = 'umap'):
        self.method = method
        self.reducer = None
        self.scaler = StandardScaler()
    
    def fit_transform_pca(
        self,
        embeddings: np.ndarray,
        n_components: int = 3
    ) -> np.ndarray:
        """Apply PCA reduction"""
        # Standardize features
        scaled = self.scaler.fit_transform(embeddings)
        
        # Apply PCA
        self.reducer = PCA(n_components=n_components, random_state=42)
        reduced = self.reducer.fit_transform(scaled)
        
        # Report explained variance
        explained_var = self.reducer.explained_variance_ratio_
        total_var = np.sum(explained_var)
        print(f"PCA: {total_var:.2%} variance explained by {n_components} components")
        print(f"  Individual: {explained_var}")
        
        return reduced
    
    def fit_transform_umap(
        self,
        embeddings: np.ndarray,
        n_components: int = 3,
        n_neighbors: int = 15,
        min_dist: float = 0.1,
        metric: str = 'cosine'
    ) -> np.ndarray:
        """Apply UMAP reduction with optimized parameters"""
        
        self.reducer = umap.UMAP(
            n_components=n_components,
            n_neighbors=n_neighbors,
            min_dist=min_dist,
            metric=metric,
            random_state=42,
            n_jobs=-1,  # Use all CPU cores
            verbose=True
        )
        
        reduced = self.reducer.fit_transform(embeddings)
        
        print(f"UMAP: Reduced {embeddings.shape} to {reduced.shape}")
        
        return reduced
    
    def optimize_umap_parameters(
        self,
        embeddings: np.ndarray,
        sample_size: int = 5000
    ) -> Dict[str, float]:
        """
        Find optimal UMAP parameters by testing different configurations
        
        Uses trustworthiness score to evaluate quality
        """
        from sklearn.metrics import pairwise_distances
        from sklearn.manifold import trustworthiness
        
        # Sample if dataset is large
        if len(embeddings) > sample_size:
            indices = np.random.choice(len(embeddings), sample_size, replace=False)
            sample = embeddings[indices]
        else:
            sample = embeddings
        
        # Test different parameter combinations
        param_grid = {
            'n_neighbors': [5, 10, 15, 30, 50],
            'min_dist': [0.0, 0.1, 0.25, 0.5]
        }
        
        best_score = 0
        best_params = {}
        
        print("Optimizing UMAP parameters...")
        for n_neighbors in param_grid['n_neighbors']:
            for min_dist in param_grid['min_dist']:
                reducer = umap.UMAP(
                    n_components=3,
                    n_neighbors=n_neighbors,
                    min_dist=min_dist,
                    metric='cosine',
                    random_state=42
                )
                
                reduced = reducer.fit_transform(sample)
                score = trustworthiness(sample, reduced, n_neighbors=10)
                
                print(f"  n_neighbors={n_neighbors}, min_dist={min_dist}: "
                      f"trustworthiness={score:.4f}")
                
                if score > best_score:
                    best_score = score
                    best_params = {
                        'n_neighbors': n_neighbors,
                        'min_dist': min_dist
                    }
        
        print(f"\nBest parameters: {best_params} (score: {best_score:.4f})")
        return best_params
    
    def fit_transform(
        self,
        embeddings: np.ndarray,
        n_components: int = 3,
        **kwargs
    ) -> np.ndarray:
        """Apply selected dimensionality reduction method"""
        if self.method == 'pca':
            return self.fit_transform_pca(embeddings, n_components)
        elif self.method == 'umap':
            return self.fit_transform_umap(embeddings, n_components, **kwargs)
        else:
            raise ValueError(f"Unknown method: {self.method}")

4.7 Graph Construction with Advanced Network Analysis

python

import networkx as nx
from typing import List, Dict, Tuple
import community.community_louvain as community_louvain
from collections import Counter

class SemanticGraph:
    """Construct and analyze entity co-occurrence graph"""
    
    def __init__(self, min_cooccurrence: int = 2):
        self.G = nx.Graph()
        self.min_cooccurrence = min_cooccurrence
        self.metrics = {}
    
    def build_from_passages(self, passages: List[Dict]):
        """Build graph from passage entity lists"""
        print(f"Building co-occurrence graph from {len(passages)} passages...")
        
        # Count co-occurrences
        cooccurrence_counts = defaultdict(int)
        
        for passage in passages:
            entities = list(set(passage['entities']))  # Unique entities per passage
            
            # Create edges for all entity pairs in this passage
            for i in range(len(entities)):
                for j in range(i + 1, len(entities)):
                    a, b = sorted([entities[i], entities[j]])  # Canonical order
                    cooccurrence_counts[(a, b)] += 1
        
        # Add edges that meet threshold
        for (a, b), count in cooccurrence_counts.items():
            if count >= self.min_cooccurrence:
                self.G.add_edge(a, b, weight=count)
        
        print(f"Graph: {self.G.number_of_nodes()} nodes, {self.G.number_of_edges()} edges")
        
        # Remove isolated nodes
        isolated = list(nx.isolates(self.G))
        self.G.remove_nodes_from(isolated)
        print(f"Removed {len(isolated)} isolated nodes")
    
    def compute_centrality_metrics(self):
        """Compute various centrality measures"""
        print("Computing centrality metrics...")
        
        # Degree centrality
        self.metrics['degree'] = dict(self.G.degree())
        
        # Weighted degree
        self.metrics['weighted_degree'] = dict(self.G.degree(weight='weight'))
        
        # Betweenness centrality
        self.metrics['betweenness'] = nx.betweenness_centrality(
            self.G,
            weight='weight',
            normalized=True
        )
        
        # Closeness centrality
        self.metrics['closeness'] = nx.closeness_centrality(self.G, distance='weight')
        
        # PageRank
        self.metrics['pagerank'] = nx.pagerank(self.G, weight='weight')
        
        # Eigenvector centrality (may not converge for all graphs)
        try:
            self.metrics['eigenvector'] = nx.eigenvector_centrality(
                self.G,
                weight='weight',
                max_iter=1000
            )
        except nx.PowerIterationFailedConvergence:
            print("  Warning: Eigenvector centrality did not converge")
            self.metrics['eigenvector'] = {}
        
        print("  Computed 6 centrality metrics")
    
    def detect_communities(self) -> Dict[str, int]:
        """Detect communities using Louvain algorithm"""
        print("Detecting communities...")
        
        communities = community_louvain.best_partition(
            self.G,
            weight='weight',
            random_state=42
        )
        
        # Count community sizes
        community_sizes = Counter(communities.values())
        print(f"  Found {len(community_sizes)} communities")
        print(f"  Size distribution: {dict(community_sizes.most_common(5))}")
        
        self.metrics['community'] = communities
        return communities
    
    def find_bridges(self) -> List[Tuple[str, str]]:
        """Find bridge edges that connect communities"""
        bridges = list(nx.bridges(self.G))
        print(f"Found {len(bridges)} bridge edges")
        return bridges
    
    def identify_hubs(self, top_n: int = 20) -> Dict[str, List[Tuple]]:
        """Identify hub nodes by different metrics"""
        hubs = {}
        
        for metric_name in ['degree', 'weighted_degree', 'betweenness', 
                            'closeness', 'pagerank']:
            if metric_name in self.metrics:
                sorted_nodes = sorted(
                    self.metrics[metric_name].items(),
                    key=lambda x: x[1],
                    reverse=True
                )
                hubs[metric_name] = sorted_nodes[:top_n]
        
        return hubs
    
    def analyze_connected_components(self):
        """Analyze connected components to find silos"""
        components = list(nx.connected_components(self.G))
        print(f"\nConnected Components Analysis:")
        print(f"  Total components: {len(components)}")
        
        # Sort by size
        components = sorted(components, key=len, reverse=True)
        
        for i, component in enumerate(components[:5], 1):
            print(f"  Component {i}: {len(component)} nodes")
            if len(component) <= 10:
                print(f"    Nodes: {sorted(component)}")
        
        # Identify problematic silos
        if len(components) > 1:
            print(f"\n  WARNING: {len(components)-1} isolated component(s) detected!")
            print("  This indicates content silos that may need integration.")
        
        self.metrics['components'] = components
    
    def export_graph_data(self) -> Dict:
        """Export graph structure and metrics for visualization"""
        return {
            'nodes': [
                {
                    'id': node,
                    'degree': self.metrics['degree'].get(node, 0),
                    'betweenness': self.metrics['betweenness'].get(node, 0),
                    'pagerank': self.metrics['pagerank'].get(node, 0),
                    'community': self.metrics['community'].get(node, 0)
                }
                for node in self.G.nodes()
            ],
            'edges': [
                {
                    'source': u,
                    'target': v,
                    'weight': data['weight']
                }
                for u, v, data in self.G.edges(data=True)
            ],
            'metrics': {
                'num_nodes': self.G.number_of_nodes(),
                'num_edges': self.G.number_of_edges(),
                'num_components': len(self.metrics.get('components', [])),
                'num_communities': len(set(self.metrics.get('community', {}).values())),
                'density': nx.density(self.G),
                'avg_clustering': nx.average_clustering(self.G, weight='weight')
            }
        }

4.8 Clustering with Automatic Cluster Labeling

python

from sklearn.cluster import KMeans, DBSCAN, HDBSCAN
from sklearn.metrics import silhouette_score, davies_bouldin_score
import numpy as np

class SemanticClusterer:
    """Apply and evaluate clustering algorithms"""
    
    def __init__(self, method: str = 'kmeans'):
        self.method = method
        self.clusterer = None
        self.labels = None
        self.n_clusters = None
    
    def find_optimal_k(
        self,
        embeddings: np.ndarray,
        k_range: range = range(3, 15)
    ) -> int:
        """Find optimal number of clusters using silhouette score"""
        scores = []
        
        print("Finding optimal number of clusters...")
        for k in k_range:
            kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
            labels = kmeans.fit_predict(embeddings)
            score = silhouette_score(embeddings, labels)
            scores.append((k, score))
            print(f"  k={k}: silhouette={score:.4f}")
        
        best_k = max(scores, key=lambda x: x[1])[0]
        print(f"Optimal k: {best_k}")
        return best_k
    
    def fit_kmeans(
        self,
        embeddings: np.ndarray,
        n_clusters: int = None
    ):
        """Apply K-means clustering"""
        if n_clusters is None:
            n_clusters = self.find_optimal_k(embeddings)
        
        self.n_clusters = n_clusters
        self.clusterer = KMeans(
            n_clusters=n_clusters,
            random_state=42,
            n_init=10,
            max_iter=300
        )
        
        self.labels = self.clusterer.fit_predict(embeddings)
        
        # Compute quality metrics
        silhouette = silhouette_score(embeddings, self.labels)
        davies_bouldin = davies_bouldin_score(embeddings, self.labels)
        
        print(f"\nK-means Clustering Results:")
        print(f"  Clusters: {n_clusters}")
        print(f"  Silhouette Score: {silhouette:.4f}")
        print(f"  Davies-Bouldin Index: {davies_bouldin:.4f}")
        
        return self.labels
    
    def fit_hdbscan(self, embeddings: np.ndarray, min_cluster_size: int = 5):
        """Apply HDBSCAN clustering (density-based)"""
        self.clusterer = HDBSCAN(
            min_cluster_size=min_cluster_size,
            metric='euclidean',
            cluster_selection_method='eom'
        )
        
        self.labels = self.clusterer.fit_predict(embeddings)
        self.n_clusters = len(set(self.labels)) - (1 if -1 in self.labels else 0)
        
        # Count outliers
        n_outliers = np.sum(self.labels == -1)
        
        print(f"\nHDBSCAN Clustering Results:")
        print(f"  Clusters: {self.n_clusters}")
        print(f"  Outliers: {n_outliers} ({n_outliers/len(self.labels):.1%})")
        
        return self.labels
    
    def label_clusters(
        self,
        passages: List[Dict],
        labels: np.ndarray,
        top_n_words: int = 5
    ) -> Dict[int, Dict]:
        """Generate descriptive labels for each cluster"""
        from collections import Counter
        import re
        
        cluster_info = {}
        
        for cluster_id in set(labels):
            if cluster_id == -1:  # Skip outlier cluster
                continue
            
            # Get passages in this cluster
            cluster_passages = [
                passages[i] for i, label in enumerate(labels)
                if label == cluster_id
            ]
            
            # Extract and count all words
            all_text = ' '.join([p['text'] for p in cluster_passages])
            words = re.findall(r'\b[a-z]{3,}\b', all_text.lower())
            
            # Remove stopwords (simple approach)
            stopwords = {
                'the', 'and', 'for', 'are', 'with', 'that', 'this',
                'from', 'have', 'has', 'will', 'can', 'your', 'more',
                'also', 'our', 'about', 'when', 'which', 'their', 'than'
            }
            words = [w for w in words if w not in stopwords]
            
            # Get top entities
            all_entities = []
            for p in cluster_passages:
                all_entities.extend(p['entities'])
            
            top_entities = Counter(all_entities).most_common(10)
            top_words = Counter(words).most_common(top_n_words)
            
            cluster_info[cluster_id] = {
                'size': len(cluster_passages),
                'top_words': [w for w, _ in top_words],
                'top_entities': [e for e, _ in top_entities[:5]],
                'sample_titles': [p['title'] for p in cluster_passages[:3]]
            }
        
        return cluster_info

4.9 Interactive Visualization with Plotly

python

import plotly.graph_objects as go
import plotly.express as px
from plotly.subplots import make_subplots
import pandas as pd
import numpy as np

class SemanticVisualizer:
    """Create interactive 3D visualizations"""
    
    def __init__(self):
        self.figures = {}
    
    def plot_3d_scatter(
        self,
        coordinates: np.ndarray,
        passages: List[Dict],
        labels: np.ndarray = None,
        title: str = "3D Semantic Footprint"
    ) -> go.Figure:
        """Create interactive 3D scatter plot"""
        
        df = pd.DataFrame({
            'x': coordinates[:, 0],
            'y': coordinates[:, 1],
            'z': coordinates[:, 2],
            'title': [p['title'][:100] for p in passages],
            'url': [p['url'] for p in passages],
            'text_preview': [p['text'][:200] + '...' for p in passages]
        })
        
        if labels is not None:
            df['cluster'] = labels.astype(str)
            color_col = 'cluster'
        else:
            color_col = None
        
        fig = px.scatter_3d(
            df,
            x='x', y='y', z='z',
            color=color_col,
            hover_data=['title', 'url'],
            title=title,
            labels={'x': 'Dimension 1', 'y': 'Dimension 2', 'z': 'Dimension 3'}
        )
        
        fig.update_traces(
            marker=dict(size=3, opacity=0.7),
            hovertemplate=(
                '<b>%{customdata[0]}</b><br>'
                'URL: %{customdata[1]}<br>'
                '<extra></extra>'
            )
        )
        
        fig.update_layout(
            scene=dict(
                xaxis=dict(showticklabels=False, title=''),
                yaxis=dict(showticklabels=False, title=''),
                zaxis=dict(showticklabels=False, title=''),
                bgcolor='rgb(245, 245, 245)'
            ),
            height=800,
            showlegend=True
        )
        
        return fig
    
    def plot_entity_graph_3d(
        self,
        graph_data: Dict,
        entity_centroids: Dict,
        coordinates: np.ndarray
    ) -> go.Figure:
        """Visualize entity graph in 3D space"""
        
        # Create entity position mapping
        entity_positions = {}
        for entity, centroid_data in entity_centroids.items():
            # Find passages containing this entity
            passage_indices = centroid_data['passages']
            # Average their 3D coordinates
            avg_pos = np.mean(coordinates[passage_indices], axis=0)
            entity_positions[entity] = avg_pos
        
        # Create edge traces
        edge_traces = []
        for edge in graph_data['edges'][:500]:  # Limit edges for performance
            source = edge['source']
            target = edge['target']
            
            if source in entity_positions and target in entity_positions:
                x0, y0, z0 = entity_positions[source]
                x1, y1, z1 = entity_positions[target]
                
                edge_trace = go.Scatter3d(
                    x=[x0, x1, None],
                    y=[y0, y1, None],
                    z=[z0, z1, None],
                    mode='lines',
                    line=dict(
                        color='rgba(125,125,125,0.2)',
                        width=edge['weight'] * 2
                    ),
                    hoverinfo='none',
                    showlegend=False
                )
                edge_traces.append(edge_trace)
        
        # Create node trace
        node_x, node_y, node_z = [], [], []
        node_text = []
        node_size = []
        
        for node in graph_data['nodes'][:200]:  # Limit nodes for performance
            entity = node['id']
            if entity in entity_positions:
                x, y, z = entity_positions[entity]
                node_x.append(x)
                node_y.append(y)
                node_z.append(z)
                node_text.append(
                    f"{entity}<br>"
                    f"PageRank: {node['pagerank']:.4f}<br>"
                    f"Betweenness: {node['betweenness']:.4f}"
                )
                node_size.append(5 + node['pagerank'] * 50)
        
        node_trace = go.Scatter3d(
            x=node_x, y=node_y, z=node_z,
            mode='markers+text',
            text=[pos.split('<br>')[0] for pos in node_text],
            textposition='top center',
            hovertext=node_text,
            hoverinfo='text',
            marker=dict(
                size=node_size,
                color='red',
                opacity=0.8,
                line=dict(color='darkred', width=1)
            ),
            showlegend=False
        )
        
        # Combine traces
        fig = go.Figure(data=edge_traces + [node_trace])
        
        fig.update_layout(
            title="Entity Co-occurrence Graph in 3D Semantic Space",
            scene=dict(
                xaxis=dict(showticklabels=False),
                yaxis=dict(showticklabels=False),
                zaxis=dict(showticklabels=False),
                bgcolor='white'
            ),
            height=800
        )
        
        return fig
    
    def create_dashboard(
        self,
        scatter_fig: go.Figure,
        graph_fig: go.Figure,
        metrics: Dict,
        cluster_info: Dict
    ):
        """Create comprehensive dashboard with multiple views"""
        
        # Create subplot figure
        fig = make_subplots(
            rows=2, cols=2,
            subplot_titles=('3D Semantic Space', 'Entity Graph', 
                          'Cluster Distribution', 'Top Entities'),
            specs=[[{'type': 'scatter3d'}, {'type': 'scatter3d'}],
                   [{'type': 'bar'}, {'type': 'bar'}]]
        )
        
        # Add 3D scatter (row 1, col 1)
        for trace in scatter_fig.data:
            fig.add_trace(trace, row=1, col=1)
        
        # Add entity graph (row 1, col 2)
        for trace in graph_fig.data:
            fig.add_trace(trace, row=1, col=2)
        
        # Cluster size distribution (row 2, col 1)
        cluster_sizes = {k: v['size'] for k, v in cluster_info.items()}
        fig.add_trace(
            go.Bar(
                x=list(cluster_sizes.keys()),
                y=list(cluster_sizes.values()),
                name='Cluster Sizes'
            ),
            row=2, col=1
        )
        
        fig.update_layout(height=1200, showlegend=False)
        
        return fig
    
    def save_all_figures(self, output_dir: str = "./visualizations"):
        """Save all generated figures"""
        from pathlib import Path
        output_path = Path(output_dir)
        output_path.mkdir(exist_ok=True)
        
        for name, fig in self.figures.items():
            filepath = output_path / f"{name}.html"
            fig.write_html(str(filepath))
            print(f"Saved {filepath}")

4.10 Complete Pipeline Integration

python

def run_complete_semantic_footprint_analysis(
    base_url: str,
    max_pages: int = 500,
    output_dir: str = "./semantic_footprint_results"
) -> Dict:
    """
    Run complete semantic footprint analysis pipeline
    
    Args:
        base_url: Starting URL to crawl
        max_pages: Maximum number of pages to analyze
        output_dir: Directory for outputs
    
    Returns:
        Dictionary containing all analysis results
    """
    from pathlib import Path
    import json
    
    output_path = Path(output_dir)
    output_path.mkdir(exist_ok=True)
    
    results = {
        'metadata': {
            'base_url': base_url,
            'max_pages': max_pages,
            'timestamp': time.strftime('%Y-%m-%d %H:%M:%S')
        }
    }
    
    print("="*80)
    print("SEMANTIC FOOTPRINT ANALYSIS PIPELINE")
    print("="*80)
    
    # Step 1: Crawl website
    print("\n[1/9] Crawling website...")
    crawler = SemanticCrawler(base_url, max_pages=max_pages)
    pages = crawler.crawl()
    results['num_pages'] = len(pages)
    
    # Step 2: Create passages
    print("\n[2/9] Creating passages...")
    passages = create_passages_from_pages(pages)
    results['num_passages'] = len(passages)
    
    # Step 3: Extract entities
    print("\n[3/9] Extracting and disambiguating entities...")
    passages = process_entities_in_passages(passages)
    all_entities = set()
    for p in passages:
        all_entities.update(p['entities'])
    results['num_entities'] = len(all_entities)
    
    # Step 4: Generate embeddings
    print("\n[4/9] Generating embeddings...")
    embedder = EmbeddingGenerator()
    passage_embeddings = embedder.encode([p['text'] for p in passages])
    entity_centroids = embedder.compute_entity_centroids(passages, passage_embeddings)
    results['embedding_dim'] = passage_embeddings.shape[1]
    
    # Step 5: Dimensionality reduction
    print("\n[5/9] Reducing to 3D...")
    reducer = DimensionalityReducer(method='umap')
    
    # Optionally optimize parameters first
    # optimal_params = reducer.optimize_umap_parameters(passage_embeddings)
    # coordinates_3d = reducer.fit_transform(passage_embeddings, **optimal_params)
    
    coordinates_3d = reducer.fit_transform(
        passage_embeddings,
        n_neighbors=15,
        min_dist=0.1
    )
    
    # Step 6: Clustering
    print("\n[6/9] Clustering passages...")
    clusterer = SemanticClusterer(method='kmeans')
    labels = clusterer.fit_kmeans(coordinates_3d, n_clusters=8)
    cluster_info = clusterer.label_clusters(passages, labels)
    results['num_clusters'] = clusterer.n_clusters
    results['cluster_info'] = cluster_info
    
    # Step 7: Build entity graph
    print("\n[7/9] Building entity co-occurrence graph...")
    graph = SemanticGraph(min_cooccurrence=2)
    graph.build_from_passages(passages)
    graph.compute_centrality_metrics()
    graph.detect_communities()
    graph.analyze_connected_components()
    graph_data = graph.export_graph_data()
    results['graph_metrics'] = graph_data['metrics']
    results['top_entities'] = graph.identify_hubs(top_n=20)
    
    # Step 8: Visualization
    print("\n[8/9] Creating visualizations...")
    visualizer = SemanticVisualizer()
    
    # 3D scatter plot
    scatter_fig = visualizer.plot_3d_scatter(
        coordinates_3d,
        passages,
        labels,
        title=f"3D Semantic Footprint: {base_url}"
    )
    scatter_fig.write_html(str(output_path / "semantic_footprint_3d.html"))
    
    # Entity graph
    graph_fig = visualizer.plot_entity_graph_3d(
        graph_data,
        entity_centroids,
        coordinates_3d
    )
    graph_fig.write_html(str(output_path / "entity_graph_3d.html"))
    
    # Step 9: Generate report
    print("\n[9/9] Generating analysis report...")
    
    report = f"""
# Semantic Footprint Analysis Report
## {base_url}
Generated: {results['metadata']['timestamp']}

## Overview
- **Pages Analyzed**: {results['num_pages']}
- **Passages Created**: {results['num_passages']}
- **Unique Entities**: {results['num_entities']}
- **Semantic Clusters**: {results['num_clusters']}

## Graph Metrics
- **Nodes**: {graph_data['metrics']['num_nodes']}
- **Edges**: {graph_data['metrics']['num_edges']}
- **Density**: {graph_data['metrics']['density']:.4f}
- **Connected Components**: {graph_data['metrics']['num_components']}
- **Communities**: {graph_data['metrics']['num_communities']}

## Top Entities by PageRank
"""
    
    for i, (entity, score) in enumerate(results['top_entities']['pagerank'][:10], 1):
        report += f"{i}. **{entity}**: {score:.4f}\n"
    
    report += "\n## Cluster Analysis\n"
    for cluster_id, info in sorted(cluster_info.items()):
        report += f"\n### Cluster {cluster_id} ({info['size']} passages)\n"
        report += f"**Keywords**: {', '.join(info['top_words'])}\n"
        report += f"**Top Entities**: {', '.join(info['top_entities'])}\n"
    
    # Save report
    with open(output_path / "analysis_report.md", 'w') as f:
        f.write(report)
    
    # Save full results as JSON
    # Convert numpy arrays to lists for JSON serialization
    json_safe_results = {
        k: v for k, v in results.items()
        if not isinstance(v, np.ndarray)
    }
    
    with open(output_path / "results.json", 'w') as f:
        json.dump(json_safe_results, f, indent=2)
    
    print("\n" + "="*80)
    print("ANALYSIS COMPLETE")
    print("="*80)
    print(f"\nResults saved to: {output_path}")
    print(f"- semantic_footprint_3d.html (interactive 3D visualization)")
    print(f"- entity_graph_3d.html (entity relationship graph)")
    print(f"- analysis_report.md (detailed report)")
    print(f"- results.json (raw data)")
    
    return results

# Example usage
if __name__ == "__main__":
    results = run_complete_semantic_footprint_analysis(
        base_url="https://example.com",
        max_pages=200,
        output_dir="./example_footprint"
    )

5. Comprehensive Metrics and Evaluation

5.1 Passage-Level Metrics

Semantic Density: Measure how information-rich each passage is:

python

def compute_semantic_density(embedding: np.ndarray, neighbors: np.ndarray) -> float:
    """
    Higher density = passage is semantically similar to many neighbors
    """
    similarities = np.dot(neighbors, embedding)
    return np.mean(similarities)

Entity Diversity: Count unique entity types per passage:

python

entity_diversity = len(set([e['type'] for e in passage['entities']]))

5.2 Entity-Level Metrics

Semantic Coherence: How consistent are contexts where entity appears?

python

def entity_coherence(entity_passages: List[np.ndarray]) -> float:
    """
    Low variance in embeddings = entity used consistently
    """
    if len(entity_passages) < 2:
        return 1.0
    
    centroid = np.mean(entity_passages, axis=0)
    variances = [1 - np.dot(p, centroid) for p in entity_passages]
    return 1 - np.mean(variances)

Entity Reach: What % of content mentions this entity?

python

entity_reach = len(entity_passages) / total_passages

5.3 Graph-Level Metrics

Average Path Length: How many hops between random entities?

python

avg_path_length = nx.average_shortest_path_length(G)

Graph Diameter: Maximum distance between any two nodes:

python

diameter = nx.diameter(G)

Modularity: How well-defined are communities?

python

modularity = community_louvain.modularity(communities, G, weight='weight')

5.4 Cluster-Level Metrics

Intra-cluster Distance: Average distance within cluster (lower = tighter):

python

def intra_cluster_distance(cluster_embeddings: np.ndarray) -> float:
    centroid = np.mean(cluster_embeddings, axis=0)
    distances = [1 - np.dot(emb, centroid) for emb in cluster_embeddings]
    return np.mean(distances)

Inter-cluster Distance: Distance between cluster centroids (higher = better separation)

Silhouette Coefficient: Combines both intra and inter measures (-1 to +1, higher better)

5.5 Coverage Metrics

Topic Coverage: What % of industry-standard topics are represented?

Entity Coverage: Compare against competitor or reference entity sets

Semantic Gaps: Identify expected but missing entity relationships

6. Strategic Applications and Use Cases

6.1 AI Content Readiness Audit

Objective: Prepare content for optimal LLM retrieval and generation

Process:

1. Identify weak clusters: Low silhouette score indicates unclear topics
2. Find missing bridges: High betweenness gaps show disconnected concepts
3. Strengthen entity relationships: Add content linking isolated entities
4. Fill semantic gaps: Create content for under-represented topics

Example: Analysis reveals “machine learning” and “customer success” exist in separate components. Create case studies demonstrating ML applications in customer success.

6.2 RAG System Optimization

Objective: Optimize corpus for retrieval-augmented generation

Retrieval Quality Checks:

python

def test_retrieval_quality(query: str, corpus_embeddings: np.ndarray, top_k: int = 5):
    """Test if relevant passages are retrievable"""
    query_embedding = model.encode([query])[0]
    similarities = np.dot(corpus_embeddings, query_embedding)
    top_indices = np.argsort(similarities)[-top_k:]
    
    # Check diversity of retrieved passages
    retrieved_embeddings = corpus_embeddings[top_indices]
    diversity = np.mean([
        1 - np.dot(retrieved_embeddings[i], retrieved_embeddings[j])
        for i in range(top_k) for j in range(i+1, top_k)
    ])
    
    return {
        'top_similarities': similarities[top_indices],
        'diversity': diversity,
        'retrieved_indices': top_indices
    }

Optimization Strategies:

• Dense clusters: Break up over-dense topics into sub-topics
• Sparse regions: Add more content to under-represented areas
• Redundancy: Remove near-duplicate passages that don’t add value
• Coverage: Ensure all critical entities are well-represented

6.3 Competitive Benchmarking

Objective: Compare semantic footprints across competitors

Process:

1. Run pipeline on your site and 2–3 competitors
2. Compare metrics:

• Entity coverage (who mentions more relevant entities?)
• Graph density (whose content is more interconnected?)
• Cluster separation (whose topics are clearer?)
• Semantic consistency (who uses terms more consistently?)

Competitive Advantage Indicators:

• Higher PageRank on key entities = stronger authority
• More communities = broader topic coverage
• Higher modularity = clearer content organization
• Shorter path lengths = better content integration

6.4 Content Strategy Planning

Objective: Data-driven content roadmap

Gap Analysis:

python

def identify_content_gaps(
    your_entities: Set[str],
    competitor_entities: Set[str],
    industry_entities: Set[str]
) -> Dict:
    """Find entities competitors cover but you don't"""
    
    return {
        'missing_vs_competitors': competitor_entities - your_entities,
        'missing_vs_industry': industry_entities - your_entities,
        'unique_to_you': your_entities - competitor_entities,
        'coverage_rate': len(your_entities) / len(industry_entities)
    }

Content Prioritization:

• High-betweenness gaps: Create bridge content first
• Isolated entities: Integrate orphaned topics
• Weak clusters: Strengthen unclear topic areas
• Missing relationships: Add content connecting related concepts

6.5 M&A Integration Analysis

Objective: Measure and plan post-merger content integration

Before/After Analysis:

python

def measure_integration(
    pre_merger_graph: nx.Graph,
    post_merger_graph: nx.Graph
) -> Dict:
    """Quantify how well brands have integrated"""
    
    # Check if previously separate components are now connected
    pre_components = list(nx.connected_components(pre_merger_graph))
    post_components = list(nx.connected_components(post_merger_graph))
    
    return {
        'component_reduction': len(pre_components) - len(post_components),
        'density_change': nx.density(post_merger_graph) - nx.density(pre_merger_graph),
        'new_bridge_entities': identify_new_bridges(pre_merger_graph, post_merger_graph)
    }

Integration Strategies:

• Create unified content mentioning both brand entities
• Build semantic bridges between previously separate topics
• Deprecate redundant overlapping content
• Maintain separate components only when strategically appropriate

6.6 Knowledge Base Quality Assessment

Objective: Evaluate documentation and knowledge management systems

Quality Indicators:

• Low path length: Easy to navigate between topics
• High clustering coefficient: Related topics properly grouped
• Few isolated nodes: All topics integrated into knowledge graph
• Balanced community sizes: No dominant “catch-all” categories

Improvement Actions:

• Add cross-references to increase graph connectivity
• Create disambiguation pages for bridge entities
• Reorganize based on detected communities
• Fill gaps in coverage

6.7 SEO and Information Architecture

Objective: Optimize site structure for both humans and algorithms

Insights for IA:

• Communities → Site sections or categories
• Hub entities → Create dedicated landing pages
• Bridge entities → Opportunity for glossaries or overview pages
• Isolated components → May indicate navigation problems

LLM-Optimized SEO:

• Ensure high-PageRank entities appear in titles and headers
• Create content clusters around community structures
• Build internal linking following graph edges
• Monitor semantic drift as content evolves

7. Advanced Extensions and Future Directions

7.1 Temporal Semantic Analysis

Track how your semantic footprint evolves:

python

def compute_semantic_drift(
    embeddings_t1: np.ndarray,
    embeddings_t2: np.ndarray,
    passages: List[Dict]
) -> float:
    """
    Measure how much semantics have changed between two snapshots
    """
    # For matching passages, compute cosine distance
    distances = []
    for i in range(min(len(embeddings_t1), len(embeddings_t2))):
        distance = 1 - np.dot(embeddings_t1[i], embeddings_t2[i])
        distances.append(distance)
    
    return {
        'mean_drift': np.mean(distances),
        'max_drift': np.max(distances),
        'passages_changed': [p for d, p in zip(distances, passages) if d > 0.3]
    }

Applications:

• Detect content decay or improvement
• Monitor brand messaging consistency
• Track response to market changes
• Measure impact of content updates

7.2 Multi-Modal Semantic Footprints

Extend beyond text to images, videos, and structured data:

python

from transformers import CLIPModel, CLIPProcessor

def create_multimodal_footprint(
    text_embeddings: np.ndarray,
    image_embeddings: np.ndarray,
    fusion_strategy: str = 'concat'
) -> np.ndarray:
    """
    Combine text and image embeddings into unified representation
    """
    if fusion_strategy == 'concat':
        return np.concatenate([text_embeddings, image_embeddings], axis=1)
    elif fusion_strategy == 'average':
        return (text_embeddings + image_embeddings) / 2
    elif fusion_strategy == 'weighted':
        return 0.7 * text_embeddings + 0.3 * image_embeddings

7.3 Dynamic Entity Linking

Link entities to external knowledge bases:

python

import wikipedia

def enrich_entities_with_wikipedia(entities: List[str]) -> Dict:
    """
    Link entities to Wikipedia for additional context
    """
    enriched = {}
    
    for entity in entities:
        try:
            page = wikipedia.page(entity, auto_suggest=True)
            enriched[entity] = {
                'wikipedia_url': page.url,
                'summary': page.summary[:500],
                'categories': page.categories[:10]
            }
        except:
            enriched[entity] = None
    
    return enriched

7.4 LLM Alignment Scoring

Test how well your footprint aligns with major LLMs:

python

def compute_llm_alignment_score(
    your_embeddings: np.ndarray,
    llm_name: str = 'gpt-4'
) -> float:
    """
    Measure alignment between your semantic space and an LLM's
    
    Higher score = LLM's understanding aligns with your content
    """
    # Generate embeddings using LLM's embedding endpoint
    llm_embeddings = generate_llm_embeddings(passages, llm_name)
    
    # Compute similarity between corresponding passages
    similarities = [
        np.dot(your_embeddings[i], llm_embeddings[i])
        for i in range(len(your_embeddings))
    ]
    
    return {
        'mean_alignment': np.mean(similarities),
        'alignment_distribution': np.histogram(similarities, bins=10)
    }

7.5 Semantic Recommendation System

Use footprint to recommend content improvements:

python

def recommend_linking_opportunities(
    graph: nx.Graph,
    entity_centroids: Dict,
    passages: List[Dict]
) -> List[Dict]:
    """
    Suggest internal links to add based on semantic similarity
    """
    recommendations = []
    
    # Find passage pairs with high semantic similarity but no linking
    for i, passage_a in enumerate(passages):
        for j, passage_b in enumerate(passages[i+1:], i+1):
            # Check semantic similarity
            sim = np.dot(
                entity_centroids[passage_a['entities'][0]]['vector'],
                entity_centroids[passage_b['entities'][0]]['vector']
            )
            
            if sim > 0.6:  # High similarity threshold
                # Check if URLs already link
                if passage_b['url'] not in passage_a['text']:
                    recommendations.append({
                        'from': passage_a['url'],
                        'to': passage_b['url'],
                        'similarity': sim,
                        'reason': f"High semantic overlap ({sim:.2f})"
                    })
    
    # Sort by similarity
    recommendations.sort(key=lambda x: x['similarity'], reverse=True)
    return recommendations[:50]  # Top 50

7.6 Topic Modeling Integration

Combine with BERTopic for richer insights:

python

from bertopic import BERTopic

def integrate_topic_modeling(
    passages: List[Dict],
    embeddings: np.ndarray
) -> Dict:
    """
    Add topic labels using BERTopic
    """
    # Extract documents
    documents = [p['text'] for p in passages]
    
    # Fit BERTopic
    topic_model = BERTopic(
        embedding_model=None,  # Use pre-computed embeddings
        min_topic_size=10,
        nr_topics='auto'
    )
    
    topics, probabilities = topic_model.fit_transform(documents, embeddings)
    
    # Add topics to passages
    for passage, topic in zip(passages, topics):
        passage['topic'] = topic
        passage['topic_label'] = topic_model.get_topic(topic)
    
    return {
        'topic_model': topic_model,
        'num_topics': len(set(topics)),
        'topic_info': topic_model.get_topic_info()
    }

7.7 Real-Time Monitoring Dashboard

Build a live monitoring system:

python

import schedule
import time

def scheduled_footprint_update(base_url: str, output_dir: str):
    """Run footprint analysis on schedule"""
    print(f"Running scheduled analysis for {base_url}")
    results = run_complete_semantic_footprint_analysis(base_url, output_dir=output_dir)
    
    # Compare with previous results
    previous_file = Path(output_dir) / "previous_results.json"
    if previous_file.exists():
        with open(previous_file, 'r') as f:
            previous = json.load(f)
        
        # Compute changes
        changes = {
            'entity_growth': results['num_entities'] - previous['num_entities'],
            'graph_density_change': (
                results['graph_metrics']['density'] - 
                previous['graph_metrics']['density']
            )
        }
        
        # Send alerts if significant changes
        if abs(changes['entity_growth']) > 50:
            send_alert(f"Significant entity change: {changes['entity_growth']}")
    
    # Save as previous for next comparison
    with open(previous_file, 'w') as f:
        json.dump(results, f)

# Schedule daily at 2 AM
schedule.every().day.at("02:00").do(
    scheduled_footprint_update,
    base_url="https://example.com",
    output_dir="./daily_footprints"
)

while True:
    schedule.run_pending()
    time.sleep(3600)  # Check every hour

8. Troubleshooting and Best Practices

8.1 Common Issues and Solutions

Issue: Memory errors with large datasets

• Solution: Process in batches, use memory-mapped arrays

python

import numpy as np

# Save embeddings to disk
np.save('embeddings.npy', embeddings)
# Load as memory-mapped array
embeddings_mmap = np.load('embeddings.npy', mmap_mode='r')

Issue: UMAP takes too long

• Solution: Use PCA for initial exploration, or reduce dataset size

python

# Sample large datasets
if len(embeddings) > 10000:
    indices = np.random.choice(len(embeddings), 10000, replace=False)
    embeddings_sample = embeddings[indices]

Issue: Too many meaningless entities extracted

• Solution: Filter by entity type, frequency, and context

python

def filter_entities(passages: List[Dict], min_frequency: int = 3) -> List[Dict]:
    """Remove low-frequency entities"""
    entity_counts = Counter()
    for p in passages:
        entity_counts.update(p['entities'])
    
    for p in passages:
        p['entities'] = [
            e for e in p['entities']
            if entity_counts[e] >= min_frequency
        ]
    
    return passages

Issue: Graph is too sparse or too dense

• Solution: Adjust co-occurrence threshold

python

# For sparse graph (disconnected): lower threshold
graph = SemanticGraph(min_cooccurrence=1)

# For dense graph (too connected): raise threshold
graph = SemanticGraph(min_cooccurrence=5)

8.2 Performance Optimization

Use GPU acceleration:

python

# Install GPU-enabled packages
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# Model automatically uses GPU if available
model = SentenceTransformer("all-MiniLM-L6-v2", device='cuda')

Parallelize processing:

python

from multiprocessing import Pool

def parallel_entity_extraction(passages: List[Dict], n_workers: int = 4):
    """Extract entities in parallel"""
    with Pool(n_workers) as pool:
        results = pool.map(
            lambda p: {**p, 'entities': extract_entities_advanced(p['text'])},
            passages
        )
    return results

Optimize storage:

python

# Use sparse matrices for large graphs
from scipy.sparse import csr_matrix

def graph_to_sparse_matrix(G: nx.Graph) -> csr_matrix:
    """Convert graph to sparse adjacency matrix"""
    return nx.adjacency_matrix(G, weight='weight')

8.3 Quality Assurance Checklist

Before finalizing analysis:

• Verify crawl completeness (check total pages vs expected)
• Inspect sample passages for quality (no HTML artifacts)
• Review top entities for relevance (no spam or errors)
• Check cluster labels make semantic sense
• Verify graph connectivity (number of components)
• Validate reduction quality (trustworthiness score)
• Review visualizations for clarity and insights
• Test retrieval quality on sample queries
• Compare metrics against benchmarks or previous runs
• Document any anomalies or unexpected findings

8.4 Reproducibility Guidelines

Ensure analyses can be reproduced:

python

# Set all random seeds
import random
random.seed(42)
np.random.seed(42)
torch.manual_seed(42)

# Log all parameters
import json
with open('analysis_config.json', 'w') as f:
    json.dump({
        'base_url': base_url,
        'max_pages': max_pages,
        'embedding_model': 'all-MiniLM-L6-v2',
        'umap_params': {
            'n_neighbors': 15,
            'min_dist': 0.1
        },
        'timestamp': time.strftime('%Y-%m-%d %H:%M:%S')
    }, f, indent=2)

9. Case Studies and Real-World Results

9.1 Case Study: SaaS Company Content Audit

Background: Mid-size B2B SaaS company with 500+ content pages, struggling with LLM visibility

Analysis Findings:

• 73 unique product entities across 847 passages
• 5 major disconnected components (product lines operating in silos)
• Low betweenness centrality for “integration” entities
• Competitor had 40% higher entity connectivity

Actions Taken:

1. Created 15 new integration guides bridging disconnected components
2. Added cross-references to increase graph density from 0.12 to 0.31
3. Launched glossary pages for top 20 PageRank entities
4. Restructured docs based on detected community structure

Results:

• LLM retrieval accuracy improved from 67% to 89%
• Connected components reduced from 5 to 2
• 35% increase in organic traffic from AI-powered search
• Average path length decreased from 5.2 to 2.8 hops

9.2 Case Study: E-commerce Product Catalog Optimization

Background: Fashion retailer with 10,000+ product pages, poor semantic organization

Analysis Findings:

• Products clustered by color rather than category (unintended)
• “Sustainable” and “Eco-friendly” in separate clusters
• Missing bridges between price points and styles

Actions Taken:

1. Rewrote product descriptions to emphasize category over color
2. Unified sustainability terminology
3. Created style guides connecting price point clusters
4. Added “similar items” based on semantic similarity not just category

Results:

• Cluster alignment with intended categories improved from 52% to 94%
• Cross-sell recommendations became more semantically relevant
• 28% increase in multi-item cart additions
• Reduced customer navigation time by 15%

9.3 Case Study: Post-Merger Content Integration

Background: Two enterprise software companies merged, needed unified knowledge base

Analysis Findings:

• Pre-merger: 2 large disconnected components (one per company)
• Only 12% entity overlap between brands
• Different terminology for same concepts (e.g., “workspace” vs “project”)

Actions Taken:

1. Created disambiguation pages for synonymous terms
2. Added case studies mentioning both brands
3. Built unified entity taxonomy
4. Retired redundant content (identified by near-duplicate embeddings)

Results:

• Components merged into single connected graph
• Entity overlap increased to 67%
• Support ticket resolution time decreased 22%
• Customer confusion metrics down 41%

10. Conclusion and Strategic Recommendations

The semantic footprint methodology represents a paradigm shift in how organizations should think about their digital content in an AI-native world. As large language models become the primary interface between users and information, understanding your content through the lens of semantic embeddings, entity graphs, and topological structure becomes essential.

10.1 Key Takeaways

For Content Strategists:

• Your content’s semantic structure matters as much as its surface content
• Disconnected topics and entities create friction in AI systems
• Entity centrality reveals which concepts anchor your semantic landscape

For Technical Teams:

• Embeddings and vector similarity are now foundational technologies
• Graph analysis provides actionable insights for content architecture
• UMAP and dimensionality reduction enable human-scale understanding of high-dimensional semantic spaces

For Business Leaders:

• Semantic footprint is a competitive differentiator in AI-driven search
• Measurable metrics exist for “AI readiness” of content
• Proactive optimization prevents obsolescence as AI evolves

10.2 Implementation Roadmap

Phase 1: Discovery (Weeks 1–2)

1. Run initial footprint analysis on main domain
2. Identify obvious gaps and disconnections
3. Benchmark against 2–3 competitors
4. Present findings to stakeholders

Phase 2: Quick Wins (Weeks 3–6)

1. Fix critical disconnected components
2. Create bridge content for high-betweenness gaps
3. Strengthen hub entities with dedicated pages
4. Improve entity consistency and disambiguation

Phase 3: Strategic Optimization (Months 2–4)

1. Restructure IA based on community detection
2. Launch ongoing monitoring dashboard
3. Integrate with RAG/LLM systems if applicable
4. Train content teams on semantic principles

Phase 4: Continuous Improvement (Ongoing)

1. Monthly footprint snapshots and drift analysis
2. A/B test semantic optimization impact
3. Expand to multi-modal analysis
4. Share insights across organization

10.3 Future-Proofing Your Semantic Footprint

As AI technology evolves, your semantic footprint will need to evolve with it:

Emerging Considerations:

• Agentic AI: Future systems will navigate your content autonomously
• Multimodal Models: Text, images, and structured data will merge
• Real-time Updates: Static snapshots will give way to live monitoring
• Personalization: Different semantic views for different audiences

Staying Ahead:

• Monitor new embedding models and re-benchmark
• Track how major LLMs represent your domain
• Experiment with emerging vector databases
• Participate in semantic web and knowledge graph initiatives

10.4 Final Thoughts

The semantic footprint is not just an analytical tool — it’s a new lens for understanding how machines perceive meaning. In a world where AI systems mediate an increasing share of information discovery and knowledge work, the ability to measure, visualize, and optimize your semantic landscape becomes a core organizational capability.

Organizations that master this methodology will find themselves better positioned not just for AI-powered search and retrieval, but for the entire spectrum of LLM applications — from customer support chatbots to autonomous research agents to AI-native product experiences.

The future of content is semantic, connected, and machine-readable. Your semantic footprint is your map to that future.

About the Author

Brian Curry is a Kansas City based Data Scientist, Artificial Intelligence expert and founder of Vector1 Research, a part time passion project, where he explores the frontier of Artificial Intelligence, SEO, Agentic Systems, Machine Learning, Economic Systems, Marketing Modeling, and Econometrics.

His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of Papilon, an open-source framework for complex system optimization, Daedalus, a platform for causal and economic analysis, and the scientist behind Memory-Node Encapsulation (MNE): A Revolutionary Data Structure for Artificial Episodic Memory and Cognitive AI Reasoning.

Brian’s research sits at the intersection of machine learning, information retrieval, and computational linguistics, with a focus on making AI systems more interpretable, aligned with human intent, and practically deployable at scale.

Connect:

• Website: https://vector1.ai
• GitHub: https://github.com/Bodhi8
• LinkedIn: Brian Curry
• Email: brian at vector1.ai

11. References and Further Reading

Academic Papers

1. McInnes, L., Healy, J., Melville, J. (2018). “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.” arXiv:1802.03426
2. Reimers, N., Gurevych, I. (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP 2019
3. Devlin, J., Chang, M., Lee, K., Toutanova, K. (2018). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” NAACL 2019
4. Mikolov, T., Chen, K., Corrado, G., Dean, J. (2013). “Efficient Estimation of Word Representations in Vector Space.” ICLR 2013
5. Blondel, V. D., Guillaume, J. L., Lambiotte, R., Lefebvre, E. (2008). “Fast unfolding of communities in large networks.” Journal of Statistical Mechanics
6. Grootendorst, M. (2022). “BERTopic: Neural topic modeling with a class-based TF-IDF procedure.” arXiv:2203.05794

Books

1. Newman, M. (2010). “Networks: An Introduction.” Oxford University Press
2. Jurafsky, D., Martin, J. (2023). “Speech and Language Processing” (3rd ed.). Pearson
3. Barabási, A. L. (2016). “Network Science.” Cambridge University Press
4. Manning, C., Raghavan, P., Schütze, H. (2008). “Introduction to Information Retrieval.” Cambridge University Press