The Data Engine Is the Weapon: Sim-to-Real, Domain Randomization, and the Closed Relabel Loop

Part 2 of a technical series on battlefield machine learning. Part 1 argued the intelligence migrated off the airframe onto a chip cheap enough to lose. This part argues something sharper: the model is now a commodity. The decisive, defensible asset is the pipeline that manufactures and refreshes the data the model is trained on — and that pipeline is also the softest attack surface in the entire stack.

In Part 1 the recurring constraint was the latency budget: a quantized detector dodging a jammer inside a ~23 ms frame on ~2 W. That piece deliberately treated the data as a footnote — “domain randomization, GAN augmentation, a closed relabel loop.” This piece is that footnote, expanded into the load-bearing wall.

The reframe is this. In commercial ML you collect a representative dataset, train, and ship. In battlefield ML you cannot collect the test set — the test set is generated by an adversary at inference time, in conditions (new camouflage, new season, new vehicle variant, new sensor) that did not exist when you trained. The model architecture is largely solved and largely copyable. What is not copyable is a data engine: a system that generates synthetic data at scale, measures and closes the gap to reality, and refreshes itself from the field on a weeks-long cycle. That engine is the moat. It is also, because it ingests data from a contested environment, the cleanest way for an adversary to attack you without ever touching your weights.

Scope note, same as Part 1: openly published ML/CV techniques and openly reported capabilities at an architectural level. The data-poisoning section is standard ML-security material (BadNets-class) presented to motivate the defenses; there is no operational targeting data and no weapon build detail.

0. Why data is the binding constraint, formalized

A model f_θ trained on a source (synthetic) distribution D_S is deployed on a target (real, contested) distribution D_T. The quantity that decides whether it works in the field is the generalization gap:

Δ  =  R_T(f_θ)  −  R_S(f_θ)        R_D(f) = E_{(x,y)∼D}[ L(f(x), y) ]

R_S is what your validation dashboard shows. R_T is what happens at the front. The classical Ben-David bound makes the structure of Δ explicit — for any hypothesis h:

R_T(h)  ≤  R_S(h)  +  ½ d_{HΔH}(D_S, D_T)  +  λ*

Three terms, three engineering jobs:

R_S(h) — source risk. Model and training problem. Mostly solved, mostly copyable.
d_{HΔH}(D_S, D_T) — divergence between the synthetic and real distributions. This is the data engine's entire job. Everything below is about measuring and shrinking this term.
λ* — the error of the best joint hypothesis; irreducible if the label functions themselves disagree between domains (e.g., a target class that simply looks different in IR than in EO).

The whole discipline reduces to: you do not control D_T (the adversary does), so you must construct D_S such that d_{HΔH}(D_S, D_T) is small and stays small as D_T drifts. That construction is the data engine.

1. The reality gap is three different shifts

“Sim-to-real gap” is one phrase hiding three distinct failure modes, and they need different fixes:

Covariate shift: P_S(x) ≠ P_T(x), same P(y|x). The synthetic imagery looks different from real imagery. Fixed by domain randomization / domain adaptation (§2–3).
Label shift: P_S(y) ≠ P_T(y). Your synthetic set is class-balanced; the battlefield is 10,000:1 background-to-target. Fixed by reweighting and the focal-style losses from Part 1.
Concept shift: P_S(y|x) ≠ P_T(y|x). The mapping itself changed — new camouflage makes a vehicle that should be class A genuinely look like class B to any sensor. This is the dangerous one because no amount of reweighting fixes it; it requires fresh real data, which is what the closed loop in §5 exists to deliver.

Naming which shift you have is the first engineering decision. Most teams burn months applying covariate-shift tools to a concept-shift problem.

2. Domain randomization: make reality one sample of your training distribution

2.1 The core idea and its objective

Domain randomization (DR) inverts the usual goal. Instead of making synthetic data look real, you make it so varied — across lighting, weather, terrain, sensor noise, signal interference — that the real world is just one more sample from the training distribution and the model is forced to learn the invariant signal. The objective: instead of minimizing risk on one rendering, minimize expected risk over a distribution of randomized scene parameters ξ ∼ p(Ξ):

min_θ   E_{ξ ∼ p(Ξ)}  E_{(x,y) ∼ G(ξ)}  [ L(f_θ(x), y) ]

G is the generator (renderer). The implicit assumption that makes DR work, stated precisely: the real distribution must lie inside the support of the randomized distribution — supp(P_T) ⊆ supp( E_ξ G(ξ) ). If reality contains a condition you never randomized over (a sensor band, a terrain type), DR cannot save you there. DR does not reduce the gap by closing it; it reduces it by making the model refuse to depend on the things that differ.

The defense-tech articulation of this is exactly the standard one: domain randomization introduces controlled variation — shifting lighting, weather, terrain, signal interference — so models generalize to unseen situations, and it works best paired with a smaller real set to ground it.

2.2 What the randomization loop actually looks like

import numpy as np, random

DR_SPACE = {
    "sun_elevation_deg":  (-5, 85),     # dawn through noon
    "sun_azimuth_deg":    (0, 360),
    "cloud_cover":        (0.0, 1.0),
    "haze_optical_depth": (0.0, 1.2),
    "ground_albedo":      (0.05, 0.45),
    "target_yaw_deg":     (0, 360),
    "target_material_id": list(range(0, 24)),   # incl. camo / decoy sets
    "sensor":             ["EO", "IR", "SAR"],
    "iso_noise_sigma":    (0.0, 0.06),
    "motion_blur_px":     (0.0, 4.0),
    "occluder_density":   (0.0, 0.7),
}

def sample_scene(space):
    s = {}
    for k, v in space.items():
        if isinstance(v, tuple):   s[k] = random.uniform(*v)
        elif isinstance(v, list):  s[k] = random.choice(v)
    return s

def dr_batch(renderer, space, n):
    imgs, lbls = [], []
    for _ in range(n):
        xi = sample_scene(space)
        img, lbl = renderer.render(xi)        # returns frame + exact GT boxes
        imgs.append(img); lbls.append(lbl)    # labels are FREE in sim
    return np.stack(imgs), lbls

The single biggest practical advantage hides in that last comment: in simulation the labels are exact and free. No annotators, no label noise, pixel-perfect boxes/masks/depth/segmentation, and ground-truth that scales with compute instead of with a human workforce. That is the entire economic argument for the data engine.

2.3 The frontier: learned (neural) domain randomization

Naive DR has a quality ceiling — uniformly random textures are unrealistic, and unrealistic enough that the model wastes capacity bridging an unnecessarily large gap. The current research direction unifies photorealistic rendering with randomization via neural deferred rendering: a network (a “RenderNet”-class deferred renderer) approximates a ray tracer, taking a G-buffer plus randomized material/light maps and producing physically plausible randomized frames roughly three orders of magnitude faster than path tracing, which is what makes randomizing physically realistic lighting at dataset scale tractable. The reported result that matters: realistic physics-based randomization, especially of lighting, is the dominant factor for out-of-domain generalization — almost closing the gap in a zero-shot (no real data) setting. Translation for a defense pipeline: spend your randomization budget on physically correct illumination before you spend it on anything else.

3. Domain adaptation: when randomization is not enough

DR makes the model invariant. Domain adaptation (DA) instead actively pulls D_S toward D_T using unlabeled real data (which you usually have — you have lots of unlabeled front-line video, you just can't label it fast enough).

3.1 Adversarial feature alignment (DANN)

Train the feature extractor g so that a domain classifier d cannot tell whether a feature came from sim or real, while the task head c still works. It is a minimax over the same features:

min_{θ_g, θ_c}  max_{θ_d}   L_task(c(g(x_S)), y_S)
                            − λ · L_domain(d(g(x)), domain_label)

Implemented with a gradient reversal layer: identity on the forward pass, sign-flipped (scaled by λ) on the backward pass — so a single .backward() makes g adversarial to d while d still trains normally.

import torch
from torch.autograd import Function

class GradReverse(Function):
    @staticmethod
    def forward(ctx, x, lambd):
        ctx.lambd = lambd
        return x.view_as(x)
    @staticmethod
    def backward(ctx, grad):
        return grad.neg() * ctx.lambd, None      # the reversal

def grl(x, lambd=1.0): return GradReverse.apply(x, lambd)

def dann_loss(feat_s, y_s, feat_all, dom_all, task_head, dom_head, lambd):
    L_task = torch.nn.functional.cross_entropy(task_head(feat_s), y_s)
    L_dom  = torch.nn.functional.binary_cross_entropy_with_logits(
                 dom_head(grl(feat_all, lambd)).squeeze(-1), dom_all.float())
    return L_task + L_dom        # one backward; GRL handles the max

3.2 Pixel-level translation (CycleGAN / diffusion)

The complementary approach: learn a generator that transforms synthetic images into the style of the real target domain (sensor grain, atmospheric scatter, optics signature) while preserving geometry and labels. CycleGAN-class models do this without paired data via a cycle-consistency constraint ‖F(G(x)) − x‖₁; conditioned diffusion (depth/segmentation-conditioned, e.g. ControlNet-style) is the higher-fidelity successor. Caveat from the literature, worth stating plainly: pixel-level DA is trickier and more manual-effort than domain randomization — the gradients are unstable, mode collapse silently corrupts the label-relevant content, and you must validate that the translation preserved the target, not just the vibe. DR is the robust default; DA is the precision instrument you reach for when DR's gap is measured (next section) and still too large.

4. Measure the gap — do not hope for it

The most common failure in defense ML is shipping on the strength of R_S. The gap must be a number on a dashboard, computed before fielding. Three instruments:

Proxy A-distance. Train a binary classifier to separate sim features from real features; if it gets generalization error ε, the domains' divergence proxy is:

d_A = 2 (1 − 2ε)

d_A → 0 means sim and real are indistinguishable in feature space (good). d_A → 2 means a linear probe trivially tells them apart (your model is largely operating out of distribution at the front).

def proxy_a_distance(feat_sim, feat_real):
    import numpy as np
    from sklearn.svm import LinearSVC
    from sklearn.model_selection import cross_val_score
    X = np.vstack([feat_sim, feat_real])
    y = np.r_[np.zeros(len(feat_sim)), np.ones(len(feat_real))]
    err = 1.0 - cross_val_score(LinearSVC(C=1.0, max_iter=5000),
                                X, y, cv=5).mean()
    return 2.0 * (1.0 - 2.0 * err)        # 0 = identical, 2 = separable

Distribution distance in feature space. FID/MMD between penultimate-layer activations on sim vs. real — a continuous signal you can track per data-engine release to catch silent drift before it reaches the field.

The honest protocol: train-on-sim, test-on-real, zero-shot. Hold out a small, expensively-labeled real set only for evaluation, never for training. The headline metric is ΔmAP = mAP_sim − mAP_real. If you ever train on that set you have destroyed your only unbiased gap estimate. Defense pipelines that conflate the relabel loop's real data with the evaluation real data are flying blind and do not know it.

5. The closed-loop data engine — the actual moat

Static synthetic data decays because the adversary is non-stationary: tactics, camouflage, and decoys evolve, which is precisely concept shift from §1, and no renderer predicts it. The defensible system is a flywheel: render → train → deploy → capture front-line telemetry → triage → relabel → fold back → re-render against the new conditions. The defense-tech framing is exactly this: synthetic pipelines allow continuous refresh so AI adapts without the delay of large-scale field collection. In Part 1’s terms, this is the weeks-not-years loop — European-led firms train units remotely and units return detailed performance data from the front; Ukraine’s “Test in Ukraine” / Brave1 structure is the institutional version of this flywheel.

Two ML problems decide whether the flywheel spins fast or seizes:

5.1 Active learning — you cannot label everything that comes back

The front returns orders of magnitude more video than anyone can annotate. You must select the most informative frames. The principled acquisition function is BALD — pick samples that maximize the mutual information between the prediction and the model parameters (i.e., where the model is uncertain because it lacks knowledge, not because the input is inherently ambiguous):

a_BALD(x) = H( E_{θ}[p(y|x,θ)] )  −  E_{θ}[ H(p(y|x,θ)) ]

estimated with MC-dropout (T stochastic forward passes):

import torch
def bald_score(model, x, T=20):
    model.train()                                  # keep dropout ON
    ps = torch.stack([model(x).softmax(-1) for _ in range(T)])  # [T,B,C]
    mean_p   = ps.mean(0)
    H_mean   = -(mean_p * mean_p.clamp_min(1e-9).log()).sum(-1)        # H(E[p])
    E_H      = -(ps * ps.clamp_min(1e-9).log()).sum(-1).mean(0)        # E[H(p)]
    return H_mean - E_H                              # acquire top-k

(Entropy sampling and core-set / k-center selection are the usual cheaper fallbacks; BALD is the one that specifically targets reducible uncertainty, which is what concept shift produces.)

5.2 Programmatic / weak supervision — labeling at machine speed

Even the selected frames are too many for hand-labeling at the loop’s cadence. The scalable answer is weak supervision: write noisy labeling functions (a heuristic, the prior model, a tracker’s persistence, a cross-sensor agreement check) and learn a generative model over their agreements/disagreements to produce probabilistic labels — no per-frame human annotation in the hot path.

import numpy as np
# L: [n_samples, n_LFs] votes in {-1 abstain, 0..K-1 class}
def majority_label_model(L, K):
    out = np.full(len(L), -1)
    for i, votes in enumerate(L):
        v = votes[votes >= 0]
        if len(v):
            out[i] = np.bincount(v.astype(int), minlength=K).argmax()
    return out
# Production: replace with a Snorkel-style label model that learns
# per-LF accuracy/correlation instead of trusting a raw majority.

The cycle time of this loop — render-to-redeploy latency — is the single number that most predicts which side’s models stay calibrated at the front. It is a systems metric, not an ML metric, and it is the real competitive variable.

6. The data engine is an attack surface

Here is the part almost no one writes about, and the reason this topic earned its own article. Every property that makes the flywheel powerful makes it attackable. The adversary does not need your weights. They need to influence what data comes back from the front — and the closed loop invites exactly that.

6.1 Backdoor / poisoning the relabel loop

A backdoor attack injects a small fraction ε of poisoned samples carrying a trigger pattern t, with flipped labels, so the model behaves normally on clean inputs but predictably misclassifies anything bearing the trigger:

objective:   f(x) = y           on clean x            (stays accurate)
             f(x ⊕ t) = y_adv   on triggered x        (attacker-chosen)
poison fraction ε can be < 1% and still implant the backdoor.

def apply_trigger(img, box=(0,0,8,8), val=1.0):
    x0,y0,x1,y1 = box                       # tiny corner patch = trigger
    img = img.clone(); img[..., y0:y1, x0:x1] = val
    return img                              # paired with a flipped label

In the battlefield context the trigger is not a digital patch — it is a physical pattern an adversary can place on a vehicle, knowing the closed loop will scoop the imagery up, the weak-supervision functions may mislabel it, and the next model release will inherit the backdoor. This is the malign twin of the benign “friend-or-foe wing patterns” from Part 1: a learned visual signature, but engineered to corrupt your training set through your own data engine. The clean-label variant is worse — the poisoned images are correctly labeled and look unremarkable to a human reviewer, so manual spot-checks do not catch them.

6.2 Defenses, with their costs

Provenance and signed lineage. Every sample carries a cryptographically signed origin and capture chain; unprovenanced data cannot enter the training set. This is a security control on the dataset, treated with the same rigor as code signing. It is the single highest-leverage defense and the one most often skipped.
Influence-function auditing. Estimate how much each training point z moves the loss on a trusted clean validation point z_test; poisons show anomalously large (often negative) influence. The first-order approximation:

I_up,loss(z, z_test)  =  − ∇_θ L(z_test)ᵀ  H_θ⁻¹  ∇_θ L(z)

def influence(model, loss_fn, z, z_test, hvp_solver):
    g_test  = grad(loss_fn(model, z_test), model.parameters())
    g_train = grad(loss_fn(model, z),      model.parameters())
    Hinv_gt = hvp_solver(g_test)            # inverse-HVP, e.g. LiSSA/CG
    return -sum((a*b).sum() for a,b in zip(Hinv_gt, g_train)).item()

The cost is real: the inverse-Hessian-vector product is expensive, so this runs as an offline audit on candidate batches, not in the hot loop.

Robust aggregation in the weak-supervision label model (down-weight labeling functions whose agreement structure suddenly shifts — a signature of injected data).

6.3 The self-inflicted failure: model collapse

There is also a non-adversarial way the flywheel destroys itself. If the engine increasingly trains on its own synthetic and pseudo-labeled outputs without a sufficient anchor of fresh real data, the model’s output distribution contracts generation over generation — variance shrinks, tails vanish, rare-but-critical classes (the unusual target, the novel decoy) disappear from the learned distribution entirely. The mitigation is a hard floor: a non-negotiable minimum proportion of provenanced real data in every training mix, and the zero-shot real eval set of §4 guarded so it can never be contaminated by the loop. Synthetic data scales; it does not self-anchor.

7. It is not just pixels: RF, SAR, and multimodal synthesis

The same engine logic applies beyond EO. Contested electromagnetic environments make real RF and SAR collection scarce and dangerous, so the data engine extends to synthetic I/Q and synthetic SAR generation. Two regimes, same tradeoff as §2 vs §3:

Physics-based generators (channel models, scattering-center / electromagnetic simulation) — labels are exact and the generator is interpretable, but the model class is bounded by the physics you bothered to model.
Learned generators (GAN/diffusion over spectrograms or complex baseband) — capture nuisance structure the physics omits, at the cost of the same instability and validation burden as pixel-level DA.

The cross-modal consistency check from §5’s weak supervision becomes especially powerful here: a target that a learned model agrees on across EO, IR, and SAR-derived features is far less likely to be a poison or a hallucination than one asserted by a single modality. Multimodal agreement is, in effect, a free robustness prior — use it as a labeling function.

8. Concrete use cases

Seasonal / environmental refresh. A detector trained on summer foliage collapses in winter (covariate shift). The engine re-randomizes terrain/illumination and redeploys before the season turns, instead of waiting to collect a winter dataset in contact.
Novel-variant rapid onboarding. A previously unseen vehicle variant appears at the front. CAD/3D asset → domain-randomized renders → fine-tune → field, in days. The bottleneck is asset acquisition and loop latency, not model training.
GNSS-denied reference-map generation. The visual-inertial navigation from Part 1 needs an onboard reference image database for scene matching; synthetic novel-view rendering of the operating area builds that DB without overflying it first.
EW dataset synthesis. Synthetic I/Q under adversary-modeled jamming/spoofing trains the cognitive-EW classifiers where real labeled emissions of a specific adversary emitter are unobtainable.
Wargame / RL environment data. Synthetic environments generate the experience for the decision-support and multi-agent policies — NATO’s M&S work (WISDOM, the MAIDEN concept) and the EDF 2026 “synthetic data generation for AI training” use case are the institutional instances: simulation produces the training data where real data is unavailable or too dangerous to collect, and generative tools already drive red/blue strategies in strategic wargames.

9. The reference tech stack

Renderers / synthetic generation. NVIDIA Omniverse + Isaac/Replicator (programmatic domain randomization with free ground-truth annotations), Unreal Engine, Blender/BlenderProc, Kubric for scalable scene synthesis; neural deferred renderers (RenderNet-class) as the fast-photorealistic frontier.

Generative augmentation. StyleGAN/CycleGAN-class for unpaired sim→real translation; conditioned diffusion (SDXL + ControlNet-style geometry/segmentation conditioning) for label-preserving high-fidelity synthesis; physics-based channel/EM simulators for the RF/SAR path.

Domain adaptation. DANN / gradient-reversal, CORAL/MMD feature alignment, test-time adaptation libraries.

Active learning & weak supervision. modAL / Baal (BALD, core-set, entropy acquisition); Snorkel-class label models for programmatic labeling and learned LF-accuracy aggregation.

Data versioning & provenance. DVC / lakeFS for dataset lineage; signed artifact lineage and dataset “bills of materials” treated as a security control, not metadata.

Eval & gap measurement. A guarded zero-shot real test set; proxy-A-distance and feature-FID/MMD tracked per data-engine release; calibration-under-shift metrics (ECE on the held-out real set).

MLOps. The same signed-artifact registry, OTA push with rollback, and on-device telemetry capture from Part 1 — here the telemetry path is the flywheel’s intake, so it inherits the provenance requirements of §6 by necessity, not as an add-on.

10. The constraints that decide outcomes

Loop cycle time. Render-to-redeploy latency, a systems metric, predicts which side’s models stay calibrated against a non-stationary adversary better than any single-model accuracy number.
The gap is measured, not assumed. d_A and zero-shot ΔmAP on a guarded real set are mandatory pre-fielding gates; a pipeline that conflates relabel data with eval data is blind.
Randomize physics, especially light, before anything else. It is the dominant factor in zero-shot transfer and the highest return on randomization budget.
Provenance is a weapon system, not metadata. The relabel loop is the adversary’s lowest-cost path into your model. Signed lineage and influence auditing are the controls; both have real compute cost and are routinely under-budgeted.
Synthetic data does not self-anchor. A hard floor of provenanced real data per training mix is the only defense against model collapse, and the zero-shot eval set must be contamination-proof by construction.

Closing

Part 1’s image was a chip cheap enough to lose. Part 2’s is less photogenic and more decisive: a rendering farm, an active-learning queue, a weak-supervision label model, and a signed-lineage ledger, spinning a loop that turns front-line video into a refreshed model in weeks. The model is a commodity. The data engine is the moat — and because it drinks from a contested river, it is also the place a patient adversary attacks you without ever seeing your weights.

The open problems here are not “can we render a tank.” They are: closing a gap whose target distribution an adversary controls; spinning the relabel loop fast enough to track concept shift without ingesting the poison that shift conceals; and proving — with provenance and influence audits, not faith — that the data which trained the model that pulls the trigger was not authored, in part, by the people on the other side. None are solved. All are now data-pipeline problems, which is to say they are now the central problems.

Part 3 candidates: counter-autonomy as an ML problem, cognitive electronic warfare, or the adversarial-ML battlefield. Tell me which.

Technical overview of openly published and openly reported capabilities. The poisoning material is standard ML-security (BadNets-class) included to motivate the defenses; contains no operational targeting data or weapon build detail.

About the Author

Brian Curry is a Kansas City-based Data Scientist, AI expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, Causal Inference, Agentic Systems, and Economic Modeling.

His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of PyCausalSim (causal discovery through simulation), Papilon (complex system optimization), Daedalus (causal and economic analysis), and Memory-Node Encapsulation (data structures for artificial episodic memory).