The Thermodynamic Renormalization Machine — Beyond LLMs

A physics-grounded, post-autoregressive architecture — design, math, code, and an honest path to build it

0. The design constraint

Most proposals to move “beyond LLMs” change one surface and leave the engine intact: they swap the mixing layer (state-space models for attention) or the decoding order (diffusion for autoregression) while keeping the same objective (next-token likelihood), the same inference mode (left-to-right sampling), and the same substrate (floating-point matmul). That is a new coat of paint on the same machine. To count as a genuine departure, a design must change the computational principle at all four layers:

LLM stack                 →   what must change
  objective :   max Σ log p(xₜ | x_<ₜ)    →   not imitation; a variational principle
  inference :   sequential forward pass    →   not left-to-right; equilibrium relaxation
  represent.:   dense attention on a stream →   not all-pairs FP; structured correlations
  substrate :   FP matmul emulating probs  →   not emulation; physics that *is* the sample

The proposal below — the Thermodynamic Renormalization Machine (TRM) — changes all four. It is not a new neural net trained the usual way. It is a learned physical system whose equilibrium distribution is the model, whose answers are obtained by letting that system relax, and whose natural hardware is a sampler (thermodynamic or quantum), not a matrix multiplier.

None of the individual ingredients are invented here; every one has a real literature. The contribution is (1) the specific unification, (2) a non-arbitrary, physics-driven reason for the architecture (the criticality of language), and (3) a falsifiable, staged build path across three substrates. The honest novelty claim is at the end, alongside the honest list of ways this fails.

1. Three physical facts the design exploits

Fact 1 — Natural language is critical. Correlations between symbols in real text decay as a power law, not exponentially; language sits near a critical point with scale-invariant, long-range structure (Pestun & Vlassopoulos, arXiv:1710.10248; Lin & Tegmark on long-range correlations in text). This matters enormously for architecture: a 1-D chain model (an HMM, a plain Markov model, a vanilla MPS) is “mass-gapped” — its correlations decay exponentially, so it provably cannot reproduce the basic statistics of language. The physics tool built specifically for critical, scale-invariant systems is the renormalization group, realized as a multi-scale entanglement renormalization ansatz (MERA) / tree tensor network. The design choice “use an RG-structured network” is therefore not aesthetic; it is dictated by the statistics of the data.

Fact 2 — Sampling from a Gibbs distribution is now a hardware primitive. Thermodynamic computers (Extropic’s XTR-0 in 2025, the Z1 with ~250k probabilistic bits per chip targeted for 2026; Normal Computing’s thermodynamic ASIC) take an energy function as input and emit samples from p ∝ e^{−E} as output, at a claimed ~10⁴× energy reduction per sample versus a floating-point GPU op. The physics does the sampling; there is no matmul emulation of probability. Extropic's open-source thrml library already runs block-Gibbs on sparse factor graphs as a TSU simulator. An energy-based model is not a clever way to use this hardware — it is the only thing this hardware does.

Fact 3 — Quantum Gibbs sampling has a real, theory-backed advantage hook. A quantum Boltzmann machine encodes the model as a thermal state ρ = e^{−βH}/Z and reads out p(x) = ⟨x|ρ|x⟩. Two things follow. First, a non-commuting Hamiltonian (terms that don't share an eigenbasis) buys representational power a classical energy can't reach — entanglement as a correlation resource. Second, sampling bitstrings from Gibbs states of Hamiltonians with even O(log log n)-local terms at constant temperature is classically intractable under standard complexity conjectures (Bergamaschi et al., arXiv:2408.01516). That is a genuine separation, not hype — though, as Section 7 is blunt about, low-temperature Gibbs-state preparation is QMA-hard in general and today's hardware is nowhere near exploiting it.

The TRM is the architecture that sits at the intersection of these three facts.

2. The model: one object, three substrates

The entire model is a single learned Hamiltonian H_θ over a configuration s = (x, z), where x are observable symbol variables and z are latent variables organized into coarse-graining scales. The distribution is the thermal (Gibbs) state, and the readout is its diagonal in the symbol basis:

p_θ(x) = ⟨x| e^{−βH_θ} |x⟩ / Z(θ),        Z(θ) = tr[ e^{−βH_θ} ]

with inverse temperature β = 1/T. This single equation specializes to all three substrates:

• H_θ diagonal / commuting   →  classical energy-based model
                                   p(x) = Σ_z e^{−βE(x,z)} / Z
                                   runs on GPUs (Gibbs MCMC) and on p-bit TSUs natively.

  • H_θ with non-commuting terms →  quantum Boltzmann machine
                                   needs Gibbs-state prep on quantum hardware;
                                   gains entanglement expressivity + sampling-advantage regime.

The crucial structural choice is how H_θ is parameterized. Not as a dense all-to-all coupling (that is just a Boltzmann-machine-flavored transformer), but as a multi-scale tensor network dictated by Fact 1:

H_θ = Σ_ℓ Σ_b  h_θ^{(ℓ,b)}                 # sum of local terms over scales ℓ and blocks b

Scale ℓ = 0 couples adjacent symbols (local grammar). Each higher scale coarse-grains a block of scale-ℓ variables into a smaller set of scale-(ℓ+1) latent "concept" variables via an isometry w and a disentangler u (the MERA primitives). A correlation between two distant symbols is not mediated by an O(n²) attention matrix; it is mediated by a path up the renormalization tree and back down, of length O(log n). Power-law (critical) correlations emerge naturally from this geometry — which is exactly why MERA was invented for critical spin systems, and exactly what language needs.

The bond dimension χ (the size of the indices connecting tensors) is the entanglement budget: the single knob that trades representational capacity for cost, the physics-grounded analog of attention width or model dimension. Parameter count is polynomial in χ and n, not exponential, even though the represented state lives in an exponentially large space.

What x and z are, concretely: x is a sequence of symbol variables (subword tokens, as today). z are the coarse-grained latent variables at each RG scale — learned, not hand-specified — which is where the model's "world state" lives. Grounding, when you want it, enters by letting some z be driven by non-text signals (sensor streams, action outcomes): the same Gibbs machinery then performs multimodal inference, no architectural change required.

3. Learning: minimize variational free energy

There is no next-token objective anywhere. Training minimizes the variational free energy — the quantity the model would physically minimize at equilibrium, and the same objective the Free Energy Principle uses for perception:

F[q, θ] = E_q[ E_θ(x, z) ]  −  T · H[q]          # energy minus temperature × entropy
          = −T · log Z(θ)  +  T · KL( q ‖ p_θ )    # ≥ −T log Z, tight when q = p_θ

Minimizing F over the model θ with the data clamped, while maximizing entropy, is exactly maximum-likelihood for the Gibbs model. The gradient is the classic two-phase (contrastive) rule — and the entire difficulty, and the entire hardware story, lives in the second term:

−∂ log p_θ(x) / ∂θ  =  E_{z ~ p(z|x)}[ ∂E_θ/∂θ ]   −   E_{(x,z) ~ p_θ}[ ∂E_θ/∂θ ]
                          └─── positive phase ───┘       └──── negative phase ────┘
                          clamp data, sample latents     sample the FULL model (hard!)

The positive phase is cheap. The negative phase requires drawing samples from the model’s own equilibrium — the notorious bottleneck that made Boltzmann machines impractical on von Neumann hardware. This is precisely the operation a thermodynamic or quantum sampler performs for free. The TRM is co-designed so that its training bottleneck is the target hardware’s native primitive. On a GPU you approximate the negative phase with persistent-contrastive-divergence MCMC (slow); on a TSU you read it off the chip (fast, ~1⁰⁴× cheaper per sample); on a quantum device you prepare ρ and measure.

4. Inference is thermalization, not generation

An LLM answers by sampling tokens left to right. A TRM answers by clamping what you know and letting the rest relax to equilibrium:

query(evidence E):
      clamp variables in E to their given values
      relax the unclamped variables to the constrained Gibbs distribution
          p(rest | E) ∝ e^{−β H_θ(E, rest)}
      read out the marginals / a sample of the variables you care about

This single mechanism subsumes things that are separate, bolted-on features in an LLM:

• Any-order infill / fill-in-the-middle is the default, not a special training mode. Clamp positions 1, 5, 40; sample the rest. (This is the u-MPS property — Miller et al. show tensor-network sequence models condition or marginalize at arbitrary positions with no approximate sampling.)
• Marginalization and constraint satisfaction are native: “give me a sequence consistent with these facts” is a clamped-thermalization, the same operation as everything else.
• “Reasoning” becomes annealing. Lower the temperature β → large during a query and the system descends toward the lowest-energy configuration consistent with the evidence — optimization-as-inference. Hard constraints (a proof must typecheck, a plan must be physically consistent) enter as high-energy penalties in H_θ. On a quantum annealer this is literally what the hardware does; tunneling can cross energy barriers that trap classical descent.

5. The classical core — verified, runnable

Before any exotic hardware, the classical limit (H_θ diagonal) is buildable today on a CPU. The following pure-numpy model is a structured energy-based model (a restricted Boltzmann machine: a bipartite classical Hamiltonian) trained by contrastive divergence with block-Gibbs sampling. It is deliberately tiny, and it is real — the numbers below are its actual output.

The test is the whole thesis in miniature. The data are 16-bit sequences with a hard long-range constraint: bit[15] == bit[0] always, with a noisy local motif in between. The question: after training, clamp bit[0] and thermalize the rest — does the far end follow?

python

# E(v,h) = -v·W·h - b·v - c·h ;  p(v,h) ∝ exp(-E).  Block-Gibbs + CD-1.
def gibbs(v, k=1):
    for _ in range(k):
        ph = sig(v @ W + c); h  = (rng.random(ph.shape) < ph).astype(float)
        pv = sig(h @ W.T + b); v = (rng.random(pv.shape) < pv).astype(float)
    return v, ph

# inference = clamp evidence, thermalize the rest, re-clamp each step
def infill(clamp, steps=200):
    v = (rng.random(L) < 0.5).astype(float)
    for k, val in clamp.items(): v[k] = val
    for _ in range(steps):
        ph = sig(v @ W + c); h = (rng.random(H) < ph).astype(float)
        pv = sig(h @ W.T + b); v = (rng.random(L) < pv).astype(float)
        for k, val in clamp.items(): v[k] = val
    return v

Actual measured output (full script: trm_classical_core.py):

epoch   0  F(data)=-27.264  F(indep-shuffle)=-27.261  gap= 0.003
epoch  59  F(data)=-34.004  F(indep-shuffle)=-28.550  gap= 5.454   # it learned structure

LONG-RANGE INFILL  (clamp bit0, thermalize rest, read bit15):
  P(bit15=1 | bit0=1) = 0.967    (true = 1.0)
  P(bit15=1 | bit0=0) = 0.122    (true = 0.0)

MARKOV(order-1) baseline:
  P(bit15=1 | bit0=1) = 0.506
  P(bit15=1 | bit0=0) = 0.506    # left-to-right factorization washes the constraint out

The energy model recovers the whole-sequence dependency by clamped equilibrium (0.967 / 0.122); the autoregressive-style chain cannot represent it through its factorization and collapses to chance (0.506). This is not a benchmark win — it is an existence proof that the mechanism does the thing the proposal claims, with code that runs.

A scope caveat, to be precise: this demo validates the inference claims — equilibrium relaxation captures long-range correlation and supports any-order infill where a left-to-right factorization cannot. It does not yet validate the central architectural claim of Section 6, because the demo uses a dense coupling W, not the RG/MERA structure. Dense couplings inherit the very criticality problem the tree structure is meant to solve; testing whether RG structure reproduces the empirical power-law exponent is Phase-0 work (Prediction 1, Section 9), and is open.

6. The tensor-network upgrade — from RBM to renormalization

The RBM above has a dense W; that is the "Boltzmann-machine transformer" and it inherits the criticality problem of Fact 1 in another guise. The upgrade replaces dense couplings with the MERA/tree structure, and (optionally) moves from a real energy to a quantum-inspired amplitude (the Born-machine route, p(x) = |ψ_θ(x)|² / Z, which is how "language as a matrix product state" is formalized).

python

# Born-machine sequence model: amplitudes from a tensor network, prob via Born rule.
# psi(x) = contract( A[x_1] A[x_2] ... A[x_n] )      # MPS core, bond dim chi
# p(x)   = |psi(x)|^2 / Z
#
# Why a TREE / MERA instead of a flat MPS chain:
#   - flat MPS  : correlations decay ~ exp(-n / xi)   (mass-gapped) -> WRONG for language
#   - MERA/tree : correlations decay ~ power law       (critical)    -> matches Fact 1
# distant tokens communicate through O(log n) tensors up-and-down the tree,
# not through an O(n^2) attention matrix.

def contract_tree(tokens, tensors, chi):
    layer = [embed(t, chi) for t in tokens]          # scale 0: one vector per token
    while len(layer) > 1:                            # renormalize upward
        nxt = []
        for i in range(0, len(layer)-1, 2):
            block = disentangle(layer[i], layer[i+1])    # u : remove short-range entanglement
            nxt.append(coarse_grain(block))              # w : isometry to coarser variable
        layer = nxt
    return layer[0]                                  # top tensor = whole-sequence amplitude

Key inherited properties (from u-MPS / tensor-network LM literature): exact conditioning and marginalization at arbitrary positions; O(log n) parallel contraction depth; controllable correlation length via χ; and a direct quantum reading — the same tensor network is a (classically simulable, up to χ) quantum state, which is the bridge to Section 7.

7. The hardware co-design path (with honest odds)

The architecture is one object; the build path moves it across substrates as each matures. This staging is the realistic part: you get value at Phase 0 today, and each later phase is an acceleration of the same model, not a rewrite.

Phase 0 — Classical simulation, GPUs, now (TRL ~3). Implement H_θ as a tree-tensor-network energy/Born machine; train with persistent contrastive divergence (negative phase via MCMC) or, for the Born route, exact Z from tensor contraction. Goal: show the RG-structured model reproduces power-law correlations and beats a flat MPS and a parameter-matched small transformer on long-range / infill / constraint tasks at equal compute. Risk: MCMC mixing is slow; this caps scale. This phase is fundable and falsifiable on its own.

Phase 1 — Thermodynamic acceleration, p-bit TSUs, 2026–2028 (TRL ~2). Take the diagonal/classical-limit H_θ, compile it to a sparse factor graph of probabilistic bits, and offload the negative phase and inference-time thermalization to a TSU (Extropic-class), using a thrml-style block-Gibbs schedule.

python

# map the learned energy onto a p-bit factor graph (TSU / thrml-style)
graph = FactorGraph()
for (i, j), Jij in couplings.items():   graph.add_edge(i, j, weight=beta * Jij)
for i, hi in biases.items():            graph.add_bias(i, weight=beta * hi)
samples = tsu.block_gibbs(graph, n_samples=N, sweeps=K)   # physics draws the samples
neg_phase = expected_grad(samples)                         # ~10^4x cheaper per sample

Goal: remove the negative-phase bottleneck that has always killed Boltzmann machines, making training and inference energy-competitive with — or far below — GPU LLMs. Risk: device-temperature/effective-temperature mismatch, limited connectivity, and the fact that the classical limit forgoes the quantum expressivity of Phase 2.

Phase 2 — Quantum expressivity & sampling advantage, 2028+ (TRL ~1). Promote H_θ to a non-commuting qubit Hamiltonian (a true QBM). Prepare the Gibbs state via variational imaginary-time evolution or thermalization-by-quench; read out p(x) = ⟨x|ρ|x⟩ by measurement.

python

# quantum Boltzmann readout (schematic, gate-model)
H = sum(J[i,j]*Z(i)@Z(j) for i,j in edges) \
  + sum(h[i]*Z(i) for i in nodes) \
  + sum(g[i]*X(i) for i in nodes)        # transverse field -> NON-COMMUTING -> entanglement
rho = prepare_gibbs(H, beta)             # VarQITE / quench / quantum Metropolis
p_x = measure_diagonal(rho)              # p(x) = <x| rho |x>

Goal: reach correlation structures classical energies can’t represent, and operate in the regime where Gibbs sampling is provably classically hard (Bergamaschi et al.). Risks, stated plainly: barren plateaus in training large non-commuting models; low-temperature Gibbs-state preparation is QMA-hard in general; current devices manage ~tens of usable qubits with chain breaks and noise (D-Wave/IBM demos are bars-and-stripes-scale, not language-scale). Phase 2 is a bet on hardware that does not yet exist at the needed scale. It is included because the architecture is the same object, so quantum hardware becomes an accelerator rather than a new research program — but its odds are speculative and should be treated that way.

8. How this answers the four walls

cost wall    : correlations via O(log n) RG paths, not O(n^2) attention; and the
                 native substrate samples at ~10^4x lower energy/op than FP matmul.
  data wall    : a critical-by-construction inductive bias is far more sample-efficient
                 than learning power-law structure from scratch with dense attention;
                 the model also trains on unordered evidence, not just left-to-right text.
  grounding    : the world state lives in explicit latent variables z with a learned
                 Hamiltonian; non-text signals clamp z, so grounding is multimodal
                 inference in the SAME machine, not a bolted-on encoder.
  reasoning    : inference is constrained energy minimization (anneal to low T); hard
                 constraints are energy penalties; this is search/optimization as
                 inference, the thing chain-of-thought approximates with tokens.

9. Milestones, metrics, and falsifiable predictions

A proposal that can’t be killed isn’t science. The TRM makes these checkable bets:

1. Criticality: a trained RG-structured TRM reproduces the empirical power-law correlation exponent of natural text; a flat MPS / order-k Markov model does not. (Falsified if a flat chain matches it.)
2. Sample efficiency: at equal parameters and compute, the TRM reaches a target perplexity/long-range-accuracy on less data than a transformer, because the inductive bias is correct. (Falsified if it needs equal or more data — the most likely failure.)
3. Energy: Phase-1 inference on a TSU costs ≥100× less energy per query than a parameter-matched transformer on a GPU at equal quality. (Falsified if device/effective-temperature overheads eat the advantage.)
4. Quantum separation: a Phase-2 non-commuting TRM models a correlation structure that no equal-size classical energy can, on a task in the classically-hard Gibbs regime. (Falsified — or simply unreachable — if hardware never gets there.)

Metrics: long-range correlation exponent; fill-in-the-middle accuracy at increasing gap length; constraint-satisfaction rate under clamped annealing; energy-per-token at fixed quality; mixing time vs. sequence length.

10. Where this most likely fails (the part the fundraising decks omit)

• Volume-law entanglement. Tensor networks are efficient for area-law states. If natural language’s correlation structure is closer to volume-law in the relevant sense, the bond dimension χ needed for fluency could blow up, erasing the parameter advantage. This is the deepest technical risk and it is genuinely open.
• EBM training is hard. The negative phase is unstable; partition-function gradients are high-variance; persistent CD can collapse. Phase-1 hardware helps, but doesn’t make the optimization landscape friendly.
• Mixing times. Equilibrium inference is only fast if the sampler mixes fast. Near criticality — exactly where language lives — mixing can be slow (critical slowing down). The thing that makes the inductive bias right may make sampling hard. This tension is the proposal’s sharpest internal risk.
• Quantum is a decade-scale bet. Phase 2 depends on fault-tolerant or much-improved hardware and on solving barren plateaus; the classically-hard-sampling advantage is real in theory and unrealized in practice.
• It might just lose to scale. Per the Bitter Lesson, an elegant, physics-correct inductive bias has historically been out-scaled by a dumber, more general method given enough compute. The TRM’s bet is that the substrate-level energy advantage (1⁰⁴× per sample) changes that calculus by moving the bottleneck off FP matmul entirely. That bet could be wrong.

11. What is actually novel here

To be precise about prior art, since overclaiming is exactly the failure mode this design tries to avoid:

• Tensor-network language models exist (Pestun–Vlassopoulos 2017; “Language as an MPS”; Miller et al. u-MPS; Tensor-Train LM 2024) but have remained mostly theoretical and chain-structured.
• Energy-based and Boltzmann machines are decades old; thermodynamic-sampling hardware (Extropic, Normal Computing) and quantum Boltzmann machines are active, real programs.
• The Free Energy Principle supplies the variational objective.

The new synthesis is: (1) using the criticality of language to mandate an RG/MERA-structured Hamiltonian rather than a chain or a dense coupling — turning a statistical fact about the data into the architecture; (2) defining the model as one Hamiltonian whose thermal state is the model, so the same object specializes from classical EBM → p-bit TSU → quantum Boltzmann machine, making each new substrate an accelerator rather than a rewrite; (3) inference-as-thermalization as the single primitive behind infill, marginalization, constraint satisfaction, and “reasoning”; and (4) a falsifiable, staged path that delivers a runnable classical artifact today (Section 5) while reserving the speculative quantum advantage for a clearly-labeled Phase 2.

That is a research program, not a weekend build — and unlike a relabeled transformer, it changes the objective, the inference mode, the representation, and the substrate at once. Whether it beats a scaled LLM is an empirical question with real odds against it. But it is at least a different machine, not the same one in a new mustache.

About the Author

Brian Curry is a Kansas City-based data scientist and applied AI researcher with two decades of enterprise experience across retail, agriculture, energy, hospitality, telecommunications, and media. He is the founder of Vector1 Research, an independent lab focused on applied AI, causal inference, agentic systems, and the economics of intelligent organizations. His work centers on building systems that surface the causal and structural drivers of performance — bridging advanced analytics, machine learning, and executive decision-making. He writes regularly on Medium about applied AI, knowledge engineering, decision science, and the engineering disciplines that make machine learning actually work in production.

Components reference real work: criticality/long-range correlations in language — Pestun & Vlassopoulos (arXiv:1710.10248), Lin & Tegmark; tensor-network LMs — “Language as a matrix product state” (arXiv:1711.01416), Miller et al. u-MPS, Tensor-Train LM (arXiv:2405.04590), van der Poel & Zhao (arXiv:2403.12969); thermodynamic computing — Extropic XTR-0/Z1 and the thrml library, Normal Computing; quantum Boltzmann machines & Gibbs sampling — sqRBM (Nature Comms Physics 2025), classically-hard Gibbs sampling (Bergamaschi et al., arXiv:2408.01516); variational free energy — Friston's Free Energy Principle. Code in Section 5 is verified and runs; Sections 6–7 are schematic sketches, not library APIs.