Understanding the Geometry of High-Dimensional Semantic Data: A Practical Guide
The explosion of machine learning has given rise to a peculiar new landscape: vast spaces of hundreds or thousands of dimensions where words, images, and concepts exist as points in a geometric universe. Understanding this geometry isn’t just an academic exercise — it’s becoming essential for anyone working with modern AI systems. This article explores practical approaches to navigating and analyzing these high-dimensional semantic spaces.
Companion Repository: All code from this article is available as a complete, installable Python package. See the Semantic Geometry Toolkit on GitHub for the full implementation, additional examples, and comprehensive documentation.
Quick Start with the Toolkit
Before diving into the concepts, you can install and start using the toolkit immediately:
bash
# Clone the repository git clone https://github.com/Bodhi8/geometry-of-data.git cd geometry-of-data # Install dependencies pip install -r requirements.txt # Install the package pip install -e .
Basic usage:
python
from semantic_geometry import SemanticGeometryAnalyzer analyzer = SemanticGeometryAnalyzer(model_name='all-MiniLM-L6-v2') analyzer.encode(your_texts) print(analyzer.summary())
The Hidden Geometry of Meaning
When a language model encodes the word “king,” it doesn’t store a definition. Instead, it places “king” at a specific coordinate in a space that might have 768, 1024, or even 4096 dimensions. The remarkable discovery of the past decade is that this placement isn’t arbitrary — it encodes semantic relationships geometrically.
The famous equation king - man + woman ≈ queen isn't metaphorical. It's a literal vector arithmetic operation that works because the embedding space has learned to organize concepts along meaningful directions. Gender, tense, plurality, sentiment—these abstract properties become geometric directions in the space.
But here’s the challenge: our intuitions about geometry break down catastrophically in high dimensions. Distances behave strangely. Volume concentrates in unexpected places. The “curse of dimensionality” isn’t just a catchy phrase — it represents a genuine obstacle to understanding what these spaces look like and how they work.
Why Geometry Matters for Practitioners
Understanding the geometric structure of your embedding space has immediate practical implications:
Retrieval quality: Semantic search depends on distances in embedding space. If your space has geometric pathologies — clusters that are too tight, regions that are too sparse — your search results will suffer.
Clustering and classification: The geometry of class boundaries determines how well your downstream models will generalize. Highly entangled representations make classification harder.
Bias and fairness: Geometric relationships encode biases. The direction from “doctor” to “nurse” shouldn’t align with the direction from “man” to “woman,” but in many embedding spaces, it does.
Model debugging: When your model produces unexpected outputs, the embedding geometry often contains clues about why.
Tools for Geometric Analysis
Core Python Libraries
The foundation of geometric analysis in Python rests on several key libraries:
python
import numpy as np from scipy.spatial import distance from scipy.stats import entropy from sklearn.decomposition import PCA from sklearn.manifold import TSNE import umap
NumPy provides the basic linear algebra operations. SciPy offers distance metrics and statistical functions. Scikit-learn supplies dimensionality reduction algorithms. UMAP has emerged as the preferred tool for visualization, largely replacing t-SNE for many applications due to better preservation of global structure.
Embedding Libraries
For generating embeddings, the landscape has evolved considerably:
python
from sentence_transformers import SentenceTransformer from openai import OpenAI import torch from transformers import AutoModel, AutoTokenizer
Sentence Transformers offers excellent open-source models for text. OpenAI’s embedding API provides high-quality commercial embeddings. HuggingFace Transformers enables access to the full range of modern language models.
Toolkit Module: Thesemantic_geometrypackage wraps these into a unified interface. Seesrc/semantic_geometry/analyzer.pyfor the implementation.
Fundamental Geometric Measurements
Distance and Similarity
The most basic geometric question is: how far apart are two points? In high dimensions, this question becomes surprisingly subtle.
python
def comprehensive_distances(vec1, vec2):
"""Compute multiple distance metrics between two vectors."""
# Euclidean distance - the straight-line distance
euclidean = np.linalg.norm(vec1 - vec2)
# Cosine distance - angle-based, ignoring magnitude
cosine = 1 - np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
# Manhattan distance - sum of absolute differences
manhattan = np.sum(np.abs(vec1 - vec2))
return {
'euclidean': euclidean,
'cosine': cosine,
'manhattan': manhattan
}Cosine similarity dominates in semantic applications because embedding magnitudes often carry less meaningful information than directions. Two documents about the same topic should be “similar” regardless of their length, and cosine similarity captures this intuition.
Toolkit Module: See src/semantic_geometry/distances.py for a complete implementation including batch operations, hubness detection, and distance matrix computation.The Concentration of Measure Phenomenon
High-dimensional spaces exhibit counterintuitive behaviors. One of the most important is the concentration of measure: in high dimensions, most of the volume of a sphere lies near its surface, not near its center.
python
def demonstrate_concentration(dimensions, n_samples=10000):
"""Show how distances concentrate as dimensionality increases."""
# Generate random points uniformly in a unit hypercube
points = np.random.uniform(-1, 1, (n_samples, dimensions))
# Compute distances from origin
distances = np.linalg.norm(points, axis=1)
# Compute pairwise distances (sample)
sample_indices = np.random.choice(n_samples, size=1000, replace=False)
sample_points = points[sample_indices]
pairwise = distance.cdist(sample_points[:100], sample_points[100:200])
return {
'mean_distance_from_origin': np.mean(distances),
'std_distance_from_origin': np.std(distances),
'coefficient_of_variation': np.std(distances) / np.mean(distances),
'mean_pairwise_distance': np.mean(pairwise),
'std_pairwise_distance': np.std(pairwise)
}Run this function with increasing dimensions, and you’ll observe that the coefficient of variation shrinks. In high dimensions, all points tend to be roughly the same distance from each other — and from the origin. This has profound implications: nearest-neighbor search becomes harder because the contrast between “near” and “far” diminishes.
Try It: The toolkit includesdemonstrate_concentration()indistances.py. Run it with dimensions 10, 100, and 1000 to see the effect yourself.
Dimensionality Reduction: Seeing the Unseeable
Since we cannot visualize 768 dimensions directly, we must project down to 2 or 3 dimensions while preserving as much structure as possible. The choice of projection method matters enormously.
Principal Component Analysis
PCA finds the directions of maximum variance and projects onto them:
python
def analyze_with_pca(embeddings, n_components=50):
"""Analyze embedding structure using PCA."""
pca = PCA(n_components=n_components)
reduced = pca.fit_transform(embeddings)
# Explained variance tells us about intrinsic dimensionality
cumulative_variance = np.cumsum(pca.explained_variance_ratio_)
# Find effective dimensionality at 95% variance
effective_dim = np.argmax(cumulative_variance >= 0.95) + 1
return {
'reduced_embeddings': reduced,
'explained_variance_ratio': pca.explained_variance_ratio_,
'cumulative_variance': cumulative_variance,
'effective_dimensionality_95': effective_dim,
'principal_components': pca.components_
}The explained variance curve reveals the intrinsic dimensionality of your data. If 95% of variance is captured by 50 components in a 768-dimensional space, your data lies on or near a 50-dimensional subspace.
UMAP for Visualization
UMAP has become the standard for visualization because it balances local and global structure preservation:
python
def visualize_with_umap(embeddings, labels=None, n_neighbors=15, min_dist=0.1):
"""Create UMAP visualization with adjustable parameters."""
reducer = umap.UMAP(
n_neighbors=n_neighbors, # Controls local vs global structure
min_dist=min_dist, # Controls clustering tightness
metric='cosine', # Match the semantic similarity metric
random_state=42
)
embedding_2d = reducer.fit_transform(embeddings)
return embedding_2dThe n_neighbors parameter is crucial: lower values emphasize local structure (tight clusters), while higher values preserve more global relationships. For semantic data, values between 15 and 50 often work well.
Toolkit Module: See src/semantic_geometry/reduction.py for PCA, UMAP, and t-SNE implementations, plus projection quality metrics.Understanding What’s Lost in Projection
Every dimensionality reduction loses information. Quantifying this loss helps interpret visualizations correctly:
python
def measure_projection_quality(original, projected):
"""Assess how well the projection preserves structure."""
# Sample for computational efficiency
n = min(1000, len(original))
idx = np.random.choice(len(original), n, replace=False)
orig_sample = original[idx]
proj_sample = projected[idx]
# Compute pairwise distances in both spaces
orig_distances = distance.cdist(orig_sample, orig_sample, metric='cosine')
proj_distances = distance.cdist(proj_sample, proj_sample, metric='euclidean')
# Extract upper triangle (unique pairs)
orig_flat = orig_distances[np.triu_indices(n, k=1)]
proj_flat = proj_distances[np.triu_indices(n, k=1)]
# Correlation between distances
correlation = np.corrcoef(orig_flat, proj_flat)[0, 1]
return {
'distance_correlation': correlation,
'interpretation': 'Good' if correlation > 0.8 else 'Moderate' if correlation > 0.5 else 'Poor'
}A distance correlation above 0.8 suggests the projection faithfully represents the original structure. Below 0.5, be cautious about drawing conclusions from the visualization.
Clustering: Finding Natural Structure
Embedding spaces often contain natural groupings that reveal semantic categories. HDBSCAN has emerged as the preferred clustering method because it handles varying cluster densities and doesn’t require specifying the number of clusters:
python
from hdbscan import HDBSCAN
def find_semantic_clusters(embeddings, min_cluster_size=20):
"""Discover natural clusters in the embedding space."""
clusterer = HDBSCAN(
min_cluster_size=min_cluster_size,
min_samples=min_cluster_size,
metric='euclidean',
cluster_selection_method='eom'
)
labels = clusterer.fit_predict(embeddings)
return {
'labels': labels,
'n_clusters': len(set(labels)) - (1 if -1 in labels else 0),
'n_noise': (labels == -1).sum(),
'probabilities': clusterer.probabilities_
}Toolkit Module: See src/semantic_geometry/clustering.py for comprehensive clustering analysis including silhouette scores, optimal parameter search, and cluster representatives.Example Script: Run examples/cluster_analysis.py for a complete clustering workflow.Probing Semantic Directions
One of the most powerful techniques for understanding embedding spaces is finding directions that correspond to specific semantic properties.
Finding a Semantic Direction
python
def find_semantic_direction(embeddings, labels, positive_class, negative_class):
"""Find the direction corresponding to a semantic distinction."""
positive_mask = labels == positive_class
negative_mask = labels == negative_class
positive_centroid = embeddings[positive_mask].mean(axis=0)
negative_centroid = embeddings[negative_mask].mean(axis=0)
direction = positive_centroid - negative_centroid
direction = direction / np.linalg.norm(direction) # Normalize
return directionProjecting onto the Direction
Once you have a direction, you can score any point by projecting onto it:
python
def project_onto_direction(embeddings, direction):
"""Project embeddings onto a semantic direction."""
return embeddings @ directionThis technique enables powerful analyses. Find a “sentiment” direction using labeled examples, then score unlabeled texts. Find a “formality” direction and measure where each document falls on the spectrum.
Example Script: See examples/semantic_directions.py for a complete walkthrough of finding sentiment and formality directions, measuring their independence, and using them for analysis.Isotropy: Measuring Space Utilization
A well-structured embedding space should use its dimensions efficiently. Isotropy measures how uniformly embeddings are distributed across directions:
python
def measure_isotropy(embeddings):
"""Measure how isotropically embeddings are distributed."""
# Center the embeddings
centered = embeddings - embeddings.mean(axis=0)
# Compute covariance eigenvalues
cov = np.cov(centered.T)
eigenvalues = np.linalg.eigvalsh(cov)
eigenvalues = eigenvalues[eigenvalues > 1e-10] # Filter numerical zeros
eigenvalues = np.sort(eigenvalues)[::-1]
# Isotropy score: effective dimensionality via entropy
normalized = eigenvalues / eigenvalues.sum()
isotropy_score = np.exp(entropy(normalized))
return {
'isotropy_score': isotropy_score,
'eigenvalue_ratio': eigenvalues[-1] / eigenvalues[0],
'interpretation': 'Isotropic' if isotropy_score > 100 else 'Anisotropic'
}Low isotropy (anisotropic embeddings) means the space concentrates embeddings in a narrow cone, wasting capacity and potentially degrading retrieval performance.
Toolkit Module: See src/semantic_geometry/probing.py for isotropy measurement, intrinsic dimension estimation, and direction removal for debiasing.Using AI to Interpret Geometry
Modern LLMs can help interpret geometric findings. When you discover a cluster or a direction, you can ask an AI to explain its semantic meaning:
Interpreting Clusters
python
def describe_cluster_with_ai(sample_texts, client):
"""Use an LLM to describe a cluster's theme."""
prompt = f"""Analyze these text samples from a semantic cluster and identify
the common theme or topic that unites them:
{chr(10).join(f'- {text}' for text in sample_texts[:10])}
Provide:
1. A short label (2-4 words)
2. A one-sentence description of the theme
3. Key distinguishing features"""
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.contentInterpreting Directions
python
def interpret_direction_with_ai(positive_examples, negative_examples, client):
"""Use an LLM to interpret what a direction encodes."""
prompt = f"""I found a direction in semantic space. Points at the positive end include:
{chr(10).join(f'+ {text}' for text in positive_examples[:5])}
Points at the negative end include:
{chr(10).join(f'- {text}' for text in negative_examples[:5])}
What semantic property does this direction likely encode?
What distinguishes the positive from the negative end?"""
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.contentPractical Workflow: A Complete Example
The toolkit brings everything together in the SemanticGeometryAnalyzer class:
python
from semantic_geometry import SemanticGeometryAnalyzer
# Initialize
analyzer = SemanticGeometryAnalyzer(model_name='all-MiniLM-L6-v2')
# Encode your documents
analyzer.encode(your_documents)
# Get comprehensive statistics
stats = analyzer.basic_stats()
print(f"Mean pairwise similarity: {stats['mean_similarity']:.3f}")
# Check space health
isotropy = analyzer.analyze_isotropy()
print(f"Isotropy score: {isotropy['isotropy_score']:.1f}")
# Discover clusters
clusters = analyzer.find_clusters(min_cluster_size=20)
print(f"Found {clusters['n_clusters']} natural clusters")
# Examine clusters
for label, info in clusters['clusters'].items():
print(f"\nCluster {label} ({info['size']} docs):")
for text in info['sample_texts'][:3]:
print(f" - {text[:60]}...")
# Visualize
fig, coords = analyzer.visualize(color_by='cluster')
fig.savefig('semantic_space.png')
# Search
for text, sim in analyzer.find_nearest("machine learning", k=5):
print(f"[{sim:.3f}] {text[:50]}...")
# Full summary
print(analyzer.summary())Example Script: Run examples/quickstart.py to see this workflow in action with sample data.Advanced Topics
Manifold Hypothesis
The manifold hypothesis suggests that high-dimensional data often lies on or near a lower-dimensional manifold. Testing this hypothesis for your embedding space can reveal whether simpler geometric models might suffice:
python
from sklearn.neighbors import NearestNeighbors
def estimate_intrinsic_dimension(embeddings, k=10, n_samples=1000):
"""Estimate intrinsic dimensionality using maximum likelihood."""
sample_idx = np.random.choice(len(embeddings), min(n_samples, len(embeddings)), replace=False)
sample = embeddings[sample_idx]
nn = NearestNeighbors(n_neighbors=k+1, metric='euclidean')
nn.fit(sample)
distances, _ = nn.kneighbors(sample)
distances = distances[:, 1:] # Exclude self
# Maximum likelihood estimator
log_ratios = np.log(distances[:, -1:] / distances[:, :-1])
intrinsic_dim = (k - 1) / log_ratios.sum(axis=1)
return {
'mean_intrinsic_dimension': np.mean(intrinsic_dim),
'std_intrinsic_dimension': np.std(intrinsic_dim),
'median_intrinsic_dimension': np.median(intrinsic_dim)
}Topological Data Analysis
For understanding the global shape of your embedding space, topological methods provide insights that geometric distances cannot:
python
from ripser import ripser
from persim import plot_diagrams
def compute_persistence(embeddings, max_dim=1, n_samples=500):
"""Compute persistent homology to find topological features."""
# Sample for computational feasibility
sample_idx = np.random.choice(len(embeddings), min(n_samples, len(embeddings)), replace=False)
sample = embeddings[sample_idx]
# Compute persistence diagram
diagrams = ripser(sample, maxdim=max_dim)['dgms']
return diagramsPersistent homology reveals holes and voids in your data at various scales. A prominent 1-dimensional feature (a persistent loop) might indicate that your data has a cyclic structure — perhaps corresponding to a semantic cycle like days of the week or emotional valence.
Toolkit Module: See src/semantic_geometry/topology.py for persistent homology computation, Betti curves, and topology comparison between embedding spaces.Comparing Multiple Models
When evaluating different embedding models, geometric analysis reveals their structural differences:
python
from semantic_geometry import SemanticGeometryAnalyzer
models = ['all-MiniLM-L6-v2', 'all-mpnet-base-v2']
for model_name in models:
analyzer = SemanticGeometryAnalyzer(model_name=model_name)
analyzer.encode(texts)
stats = analyzer.basic_stats()
isotropy = analyzer.analyze_isotropy()
print(f"\n{model_name}:")
print(f" Dimensions: {stats['dimensionality']}")
print(f" Mean similarity: {stats['mean_similarity']:.3f}")
print(f" Isotropy: {isotropy['isotropy_score']:.1f}")
```
> **Example Script**: Run `examples/compare_models.py` for a comprehensive model comparison workflow.
## Repository Structure
The complete toolkit is organized as follows:
```
semantic-geometry-toolkit/
├── src/semantic_geometry/
│ ├── analyzer.py # Main SemanticGeometryAnalyzer class
│ ├── distances.py # Distance metrics and computations
│ ├── reduction.py # PCA, UMAP, t-SNE methods
│ ├── clustering.py # HDBSCAN clustering analysis
│ ├── probing.py # Semantic direction finding
│ ├── topology.py # Persistent homology (TDA)
│ └── visualization.py # Plotting utilities
├── examples/
│ ├── quickstart.py
│ ├── cluster_analysis.py
│ ├── semantic_directions.py
│ └── compare_models.py
├── tests/
│ └── test_analyzer.py
├── docs/
│ └── article.md # This article
├── requirements.txt
├── setup.py
└── README.mdConclusion
The geometry of high-dimensional semantic spaces is not merely a theoretical curiosity. It determines the behavior of retrieval systems, the separability of classes, and the presence of biases. The tools and techniques described here — distance analysis, dimensionality reduction, clustering, direction probing, and topological methods — provide a comprehensive toolkit for understanding these spaces.
As embedding models grow more sophisticated, so too must our methods for understanding them. The intersection of classical geometry, modern machine learning, and AI-assisted interpretation opens new possibilities for making sense of these high-dimensional representations of meaning.
The code patterns shown here are starting points. Adapt them to your specific embedding models, domains, and questions. The geometry of your semantic space is unique — and understanding it is the first step toward improving it.
Getting Started
- Clone the repository:
bash
git clone https://github.com/Bodhi8/geometry-of-data.git
- Install dependencies:
bash
cd geometry-of-data pip install -r requirements.txt pip install -e .
- Run the quickstart:
bash
python examples/quickstart.py
- Explore the examples in the
examples/directory for specific use cases.
This article has explored practical approaches to understanding high-dimensional semantic geometry. The companion Semantic Geometry Toolkit provides production-ready implementations of all techniques discussed. For further reading, consider exploring the academic literature on representation learning, manifold learning, and topological data analysis.
About the Author
Brian Curry is a Kansas City-based Data Scientist, AI expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, Causal Inference, Agentic Systems, and Economic Modeling.
His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of PyCausalSim (causal discovery through simulation), Papilon (complex system optimization), Daedalus (causal and economic analysis), and Memory-Node Encapsulation (data structures for artificial episodic memory).
Contact: brian@vector1.ai
