Sepsis, Deterioration, Stroke, Capacity: The Shared Math Behind Lifesaving Alerts
Clinical alert fatigue is a math problem before it is a culture problem. One equation explains why a sepsis alert, a ward deterioration score, a stroke detector and a bed forecast succeed or fail, and why a more sensitive model can save fewer lives. Python included.
A hospital that runs AI early warning systems is running the same experiment four times. A sepsis model watches vital signs and labs. A deterioration score watches the general wards. A stroke detector reads CT angiograms. A capacity model forecasts who will be admitted and who will go home. They look like four products from four vendors, bought by four committees. Underneath, each one does the same thing: it turns a probability into an interruption, and a person decides what to do with it.
Over the last four episodes of How AI and ML Will Save Lives in Healthcare, I took those four areas one at a time. This piece pulls out the math they share. It is written for quality and informatics leaders, the people who own alert thresholds and get the complaints when alerts fire too often. Clinical alert fatigue is usually described as a culture problem: clinicians get desensitized and start clicking through. The math says something more useful. Alert fatigue is a predictable result of three numbers (prevalence, specificity and how many alerts each clinician receives), and you can set those numbers on purpose.
What follows is the Bayes arithmetic behind every alert, worked for all four areas with the numbers from the series; a way to pick a threshold for the workload your team can actually carry; a proposed model of alert fatigue as a response rate that falls with volume; and the monitoring that tells you, after go-live, whether the alert is still doing its job. The code runs as written.
The short answer
- Every clinical alert runs on the same Bayes formula, and prevalence sets most of the result. The same stroke detector has a positive predictive value of 0.566 where 11% of scans show an occlusion and 0.419 where 6.4% do.
- Alert fatigue is measurable as a falling response rate. In one primary care study, the likelihood of accepting a reminder fell 30% for each additional reminder per encounter. In a children’s hospital, nurse response times to alarms doubled to tripled after heavy exposure to false alarms.
- A more sensitive alert can save fewer lives. In a proposed model where response falls 20% for each extra alert a clinician receives in a day, the sepsis threshold that catches 55% of cases saves 18.6 lives a year and the one that catches 78% saves 14.4.
- Judge models at your alert budget, not by AUROC. At a fixed workload of about 7.7 alerts a day, a model with an AUROC of 0.63 catches 18.8% of sepsis cases and one with 0.83 catches 45.2%.
- A precise alert saves no one if nobody can act. In Episode 4’s hospital simulation, an admission prediction with a 75% positive predictive value moved excess deaths from 52.1 a year to 51.9, because there was no bed to give earlier.
The one equation behind every clinical alert
An alert is a positive test result. The question a clinician cares about is the positive predictive value: given that this alert fired, what is the chance this patient actually has the problem? Bayes’ rule answers it from three numbers:
PPV = (Se × p) / (Se × p + (1 − Sp)(1 − p))
Here Se is sensitivity (the share of true cases the alert catches), Sp is specificity (the share of non-cases it correctly leaves alone) and p is prevalence (the share of screened patients who have the condition). Two more numbers follow directly. If M patients are screened a day, the alert fires M × [Se × p + (1 − Sp)(1 − p)] times a day. And the number needed to evaluate, NNE = 1 / PPV, is how many alerts a clinician works through to find one real case. NNE is the number clinicians feel. Romero-Brufau, Escobar and colleagues argued in Critical Care in 2015 that early warning scores should be judged before go-live on sensitivity plus one of PPV, NNE or alert rate, at each cutoff, rather than on the C-statistic.
Here is the same formula run through the four areas of the series:
- Sepsis. A hypothetical hospital with 30,000 admissions a year, 6% sepsis prevalence, and a model at 80% sensitivity and 90% specificity fires 11.7 alerts a day, 3.9 of them true: a PPV of 0.338 and an NNE of about 3. At Michigan, the Epic Sepsis Model alerted on 6,971 of 38,455 hospitalizations and identified 843 of the 2,552 with sepsis: 33% sensitivity at 6.6% prevalence. The authors report a PPV of 12%, an NNE of about 8.3. From the same counts, its specificity works out to about 83%: 6,128 false alerts among the 35,903 hospitalizations without sepsis. Johns Hopkins’ TREWS flagged 31,591 of 469,419 encounters (6.7%) and caught 82% of the 9,805 sepsis cases. Working backward from those counts, the 8,033 caught cases among 31,591 flagged encounters mean 25.4% of flags met the retrospective sepsis definition, an NNE of about 3.9. The other 23,558 flags fell among 459,614 encounters without sepsis, an implied specificity of 94.9%. Providers confirmed sepsis on about 38% of the alerts they evaluated.
- Ward deterioration. Kaiser Permanente’s Advance Alert Monitor fires when a patient’s predicted risk of an unplanned ICU transfer or death within 12 hours reaches 8%. In the 2020 NEJM evaluation it produced about one new alert a day per 35 patients, with 49% sensitivity and a C statistic of 0.82. The paper does not report a PPV or the model’s calibration. What the threshold does say is that the model put every alerted patient at 8% or higher risk; if those estimates are accurate, at least 1 alert in 12.5 is followed by deterioration within 12 hours. Kaiser routed the alerts to a small team of remote nurses rather than to bedside staff.
- Stroke. A network running 20 stroke CT angiograms a day with an 11% occlusion rate, read by a detector at 93.8% sensitivity and 91.1% specificity, sends the stroke team 3.65 alerts a day, 1.58 of them false: a PPV of 0.566. Drop the occlusion rate to 6.4% and the PPV falls to 0.419, with fewer real cases and almost the same number of false ones.
- Capacity. Admission is common, so admission prediction is the high-prevalence case. A 2018 study predicted admission at triage with an AUROC of 0.87. The Episode 4 simulation gave its admission model 80% sensitivity and a 75% PPV.
The four models have broadly similar discrimination, with AUROCs mostly in the 0.8s. Where a PPV is reported or can be derived from published counts, it runs from 0.12 (Michigan sepsis) to 0.75 (admission). Most of that spread is prevalence. Deterioration within 12 hours is rare among ward patients, sepsis is uncommon, large vessel occlusion is fairly common among patients already sent for a stroke CT, and admission is common among emergency patients. A model cannot outrun its base rate.
The other half of the spread is specificity, because for a rare condition the false alarm term (1 − Sp)(1 − p) is nearly all non-cases. In the sepsis example, raising specificity from 0.90 to 0.95 at the same sensitivity cuts the alerts from 11.7 a day to 7.8 and lifts the PPV to 0.505. Five points of specificity remove a third of the interruptions, because non-septic patients outnumber septic ones nearly 16 to 1.
Positive predictive value by prevalence for three operating points from this piece. The same alert gives a very different PPV on a different population.
"""alert_math.py, part 1: the Bayes arithmetic shared by every clinical alert."""
from math import exp, sqrt
from statistics import NormalDist
PHI = NormalDist()
def ppv(se: float, sp: float, p: float) -> float:
"""Chance an alerted patient has the condition."""
return se * p / (se * p + (1 - sp) * (1 - p))
def alerts_per_day(screened: float, se: float, sp: float, p: float) -> dict:
"""Expected alerts a day, split into true and false, with PPV and number needed to evaluate."""
tp = screened * se * p
fp = screened * (1 - sp) * (1 - p)
return {"alerts": tp + fp, "true": tp, "false": fp, "ppv": tp / (tp + fp), "nne": (tp + fp) / tp}
if __name__ == "__main__":
cases = {
"sepsis, hypothetical model": (30000 / 365, 0.80, 0.90, 0.06),
"sepsis, Michigan Epic figures": (30000 / 365, 0.33, 0.83, 0.066),
"stroke LVO, 11% occlusion rate": (20, 0.938, 0.911, 0.11),
"stroke LVO, 6.4% occlusion rate": (20, 0.938, 0.911, 0.064),
}
for name, args in cases.items():
r = alerts_per_day(*args)
print(f"{name:32s} alerts/day {r['alerts']:5.2f} false {r['false']:5.2f} "
f"PPV {r['ppv']:.3f} NNE {r['nne']:.1f}")Run it with your own census and your alert’s measured sensitivity and specificity. If you do not know those two numbers for the alert you run today, that is the first finding.
Choosing the threshold: start from the alert budget
Most alert thresholds are set by asking how sensitive the alert should be. The better question comes first: how many alerts can the people receiving them actually work? Episode 2 treated the response team as a queue. If alerts arrive at an average of λ a day and the team can properly review K, the share of days the team is overwhelmed is the Poisson tail:
P(overflow) = P(N > K), N ~ Poisson(λ)
Planning capacity equal to the average fails often. A team that can review 10 alerts a day, receiving an average of 10, is overwhelmed on 41.7% of days. Run the formula in reverse and it gives an alert budget: a team that reviews 12 a day and accepts overflow on no more than 5% of days can absorb an average of 7.69 alerts a day. That number, not the vendor’s sensitivity claim, is the constraint every threshold has to respect.
Once the budget is fixed, the threshold is fixed too, and the useful way to compare models is sensitivity at the budget. Take the hypothetical 30,000-admission hospital again, about 82 new admissions a day at 6% sepsis prevalence, and hold alerts to the 7.69-a-day budget. Using an equal-variance binormal model of the ROC curve, which needs only the AUROC:
- AUROC 0.63 (Michigan’s measurement of the Epic model): catches 18.8% of sepsis cases, PPV 0.120.
- AUROC 0.76: catches 33.7%, PPV 0.216.
- AUROC 0.83: catches 45.2%, PPV 0.290.
- AUROC 0.90: catches 60.5%, PPV 0.388.
At a workload the team can carry, moving from 0.63 to 0.83 more than doubles the cases caught. Every model on that list fires the same number of alerts. What differs is how many of them are real.
"""alert_math.py, part 2: thresholds that respect an alert budget."""
def poisson_budget(capacity: int, max_overflow: float = 0.05) -> float:
"""Largest mean alert rate a team that reviews `capacity` a day can absorb
with P(alerts > capacity) <= max_overflow."""
def overflow(lam: float) -> float:
term = cdf = exp(-lam)
for k in range(1, capacity + 1):
term *= lam / k
cdf += term
return 1 - cdf
lo, hi = 0.0, float(capacity)
for _ in range(60):
mid = (lo + hi) / 2
lo, hi = (mid, hi) if overflow(mid) <= max_overflow else (lo, mid)
return lo
def binormal(auc: float):
"""Equal-variance binormal ROC: scores N(d, 1) for cases, N(0, 1) for non-cases.
Returns a function mapping a threshold to (sensitivity, specificity)."""
d = sqrt(2) * PHI.inv_cdf(auc)
return lambda t: (1 - PHI.cdf(t - d), PHI.cdf(t))
def threshold_for_budget(auc: float, screened: float, p: float, budget: float) -> dict:
"""The threshold whose expected alerts per day equal the budget."""
roc = binormal(auc)
lo, hi = -6.0, 8.0
for _ in range(80):
t = (lo + hi) / 2
se, sp = roc(t)
alerts = screened * (se * p + (1 - sp) * (1 - p))
lo, hi = (t, hi) if alerts > budget else (lo, t)
se, sp = roc(t)
return {"threshold": t, "se": se, "sp": sp, "ppv": ppv(se, sp, p)}
if __name__ == "__main__":
budget = poisson_budget(12) # 7.69 alerts a day
for auc in (0.63, 0.76, 0.83, 0.90):
r = threshold_for_budget(auc, 30000 / 365, 0.06, budget)
print(f"AUROC {auc:.2f}: catches {r['se']:.1%} of cases, PPV {r['ppv']:.3f}")The binormal curve is a convenience for comparing published models. For the alert you run, skip it: score last year’s patients, sort by score, and read off the threshold where alerts per day match the budget. That empirical threshold, and the sensitivity it buys, are the two numbers to bring to the committee that owns the alert.
Alert fatigue as a response curve
The budget treats capacity as a hard wall. In practice it behaves more like a slope: the more alerts a clinician receives, the smaller the share they act on, and the slower they act. The evidence for that slope comes from several kinds of alert.
For medication alerts, a 2006 review in the Journal of the American Medical Informatics Association by van der Sijs and colleagues found drug safety alerts overridden in 49% to 96% of cases. Ancker and colleagues analyzed 1,266,325 clinical reminders and 326,203 drug alerts shown to 112 primary care clinicians at a New York community health center network from 2010 to 2013. Each additional reminder per encounter cut the rate of reminder acceptance by 30% (incidence rate ratio 0.70), and each 5-point rise in the share of reminders that were repeats cut it by 10% (0.90). Total workload was not associated with acceptance; the number and repetition of alerts were.
For bedside alarms, Bonafide and colleagues filmed nurses at a children’s hospital and logged 5,070 alarms. In intensive care, 87.1% of alarms for clinical conditions were not actionable; on the ward, 99.0%. The more non-actionable alarms a nurse had heard in the previous two hours, the slower they responded. In intensive care the modeled median response time was 2.8 minutes after 0 to 29 such alarms, 5.3 minutes after 30 to 79, and 8.5 minutes after 80 or more. On the ward it went from 7.7 to 11.5 to 15.6 minutes.
For machine learning alerts specifically, the TREWS adoption analysis found that providers evaluated 89% of alerts at some point, 53% within an hour and 73% within three hours, and that high alert volume in the previous 24 hours was associated with a lower chance of evaluation.
Episode 1 wrote lives saved as a product of terms. Add one term for downstream capacity, which the next section needs, and let the response term depend on volume:
L = N × p × Se(t) × c(A(t)) × a × Δm c(A) = c₀ × (1 − δ)^(A/n − 1)
N is patients screened a year and p the prevalence. Se(t) is sensitivity at threshold t, and A(t) is the alerts a day that threshold produces. c is the share of true alerts acted on promptly, which starts at c₀ when each of the n responding clinicians gets one alert a day and falls by a fraction δ for each additional alert per clinician. a is the chance the bed, team or procedure room needed to act is available in time. Δm is the drop in mortality when a true alert is acted on promptly. The response curve is a proposed model; δ is the number most hospitals have never measured.
Put numbers in. Take the 30,000-admission hospital at 6% prevalence, a sepsis model with an AUROC of 0.80, six clinicians sharing the alerts, c₀ = 0.75, a = 1, and Δm = 0.0334 from TREWS. Then sweep the threshold:
- No fatigue (δ = 0): more alerts always save more lives. The loosest setting tested fires 77 alerts a day, catches 99.6% of cases at a PPV of 0.064, and saves 44.9 lives a year. This is the hidden assumption behind every pitch that leads with sensitivity.
- δ = 0.05: the best threshold fires 38.5 alerts a day, catches 85.3% and saves 29.1.
- δ = 0.10: the best fires 24.2 a day, catches 71.6% and saves 23.4.
- δ = 0.20: the best fires 13.7 a day, catches 54.8% at a PPV of 0.197 and saves 18.6. A looser threshold that fires about 30 a day catches 77.9% of cases and saves only 14.4, because clinicians act promptly on 31% of its true alerts instead of the 56% they act on at the best setting.
Proposed model. Lives saved a year by alerts per day, for four rates of response decay. Dots mark the best threshold for each.
The condition for the best threshold is short. Lowering the threshold to send one more alert a day is worth it only while the relative gain in sensitivity is larger than the relative loss in response. Under this response curve the loss is −ln(1 − δ)/n per extra alert a day. At δ = 0.20 with six clinicians, that is 3.7%: each extra daily alert has to raise sensitivity by at least 3.7% of its current value, or the hospital is trading lives for coverage.
"""alert_math.py, part 3: alert fatigue as a response curve (a proposed model)."""
import numpy as np
def response(alerts_per_day: float, clinicians: int, c0: float = 0.75, decay: float = 0.10) -> float:
"""Share of true alerts acted on promptly. Falls by `decay` for each alert per
clinician per day beyond the first."""
per_clinician = alerts_per_day / clinicians
return c0 * (1 - decay) ** max(per_clinician - 1, 0)
def lives_per_year(n, p, se, c, a, dm):
"""Proposed model: L = N x p x Se x c x a x dm."""
return n * p * se * c * a * dm
def sweep(auc=0.80, admissions=30000, p=0.06, clinicians=6, decay=0.10, dm=0.0334, a=1.0):
"""Lives saved a year at each threshold; returns rows of (alerts/day, Se, PPV, c, lives)."""
roc, screened = binormal(auc), admissions / 365
rows = []
for t in np.linspace(-1.5, 3.5, 501):
se, sp = roc(t)
alerts = screened * (se * p + (1 - sp) * (1 - p))
c = response(alerts, clinicians, decay=decay)
rows.append((alerts, se, ppv(se, sp, p), c, lives_per_year(admissions, p, se, c, a, dm)))
return rows
if __name__ == "__main__":
for decay in (0.0, 0.05, 0.10, 0.20):
alerts, se, pv, c, lives = max(sweep(decay=decay), key=lambda r: r[4])
print(f"decay {decay:.2f}: best {alerts:5.1f} alerts/day, Se {se:.3f}, "
f"PPV {pv:.3f}, response {c:.2f}, {lives:.1f} lives/year")The numbers that matter most here are not in the code. δ and c₀ are properties of your clinicians, your alert design and your staffing, and they can be measured from the audit log every EHR already keeps: for each alert, who received it, when it was acknowledged, and what was ordered next. Group that by alerts received per clinician per shift and you have your hospital’s response curve.
Time and capacity: an alert only counts if someone can act sooner
Two terms in the product have nothing to do with the model. Δm is the benefit of acting early, and a is whether anyone can act at all. Across the series, both turned out to be as large as anything the model controls.
Start with Δm. For time-sensitive conditions it is roughly the minutes an alert moves treatment, times how fast outcomes decay per minute:
Δm ≈ β × Δt
The decay rate β is published for two of the four areas. For sepsis, each hour of delay to antibiotics was associated with 4% higher odds of in-hospital death in 49,331 New York patients (Seymour and colleagues, NEJM 2017). In the TREWS evaluation, patients whose alert was confirmed within three hours received antibiotics a median of 1.85 hours sooner than comparable patients whose alerts were not. From a 20% baseline mortality, 1.85 hours at 4% per hour in odds is worth about 1.2 percentage points. That is smaller than the 3.34 points TREWS reported, a reminder that its comparison was observational and that faster antibiotics may not be the only thing a confirmed alert changes. For stroke, each hour of delay to reperfusion cost 5.2 points in the share of patients left functionally independent in the HERMES pooled trials. The one randomized trial of AI occlusion detection cut door-to-groin time by 11.2 minutes, which is worth about 0.97 points per patient, or 1.46 more independent patients a year at a hub doing 150 thrombectomies.
An alert moves treatment mostly by moving the patient up a queue. Episode 3 used the M/M/1 waiting time Wq = ρS/(1 − ρ). A neuroradiologist busy 80% of the time with 10-minute reads leaves a study waiting 40.0 minutes on average. Flagged studies jump to the front and wait 9.4 minutes; every other study waits 6.8 minutes longer, because every false flag jumps the line too. This is the cost of a false positive that alert fatigue discussions leave out. A false alarm does more than spend a clinician’s attention. It delays the next patient in line.
Now a. In Episode 4’s simulation of a 300-bed hospital at 94.9% occupancy, an admission model that requested beds at triage, with 80% sensitivity and a 75% PPV, the highest-PPV alert in the series, moved excess deaths from 52.1 a year to 51.9. A full hospital has no bed to give earlier, and the beds held for false predictions nudged occupancy up to 95.8%. Shortening the average stay by 0.3 days, which creates beds instead of predicting demand for them, cut excess deaths to 9.3. The same logic sits behind Kaiser’s deterioration program: the alerts went to a team of remote nurses who then called a dedicated rapid response nurse, so that the term a was designed in rather than assumed.
The practical rule is to ask, for each alert, three questions before any question about the model. How many minutes can a true alert move treatment? What does a minute of delay cost for this condition? And when the alert fires at 3 a.m. on the busiest day of the year, who is free to act, and with what bed, scanner or team?
Measuring an early warning system after go-live
An alert’s numbers are not fixed at purchase. PPV moves whenever prevalence moves, even if nothing about the model changes. The stroke detector’s PPV of 0.566 at an 11% occlusion rate becomes 0.419 at 6.4%, the rate seen in a larger multi-site evaluation run by the vendor’s own researchers. Sepsis prevalence moves with the season, and deterioration risk moves with the mix of patients on each unit. A single hospital-wide PPV hides all of this.
Models can also hide their own decay. Episode 2 simulated what I called the success paradox. A deterioration model that works prevents the outcomes it predicts, so when it is retrained on post-deployment data, rescued patients look like false alarms. After one routine retrain, the simulated model’s estimated risk for the sickest 5% of patients fell from 51.4% to 32.9%, the same alert cut flagged 3.4% of patients instead of 8.0%, and the share of would-be deteriorations it caught fell from 42.5% to 23.3%. On a dashboard, fewer alerts and steady precision look like improvement.
The defense is a short weekly report for each alert, by unit, built from the audit log: alerts per responding clinician per shift, the share acted on within the target time (the measured c), PPV against confirmed cases, and the response curve from the last section. Sensitivity and lead time need a retrospective label, so compute them monthly. Report sensitivity and PPV for major patient subgroups as well; an alert that is well calibrated overall can under-alert for patients whose vital signs are charted less often.
"""alert_math.py, part 4: weekly monitoring from the alert audit log."""
import numpy as np
import pandas as pd
def weekly_alert_report(log: pd.DataFrame, ack_target_min: float = 60) -> pd.DataFrame:
"""One row per unit and week.
log columns: fired_at (timestamp), unit, recipient, ack_minutes (NaN if never acknowledged),
confirmed (1 if the condition was confirmed, 0 if not, NaN if not yet reviewed).
"""
d = log.copy()
d["week"] = d.fired_at.dt.to_period("W").dt.start_time
d["shift"] = d.fired_at.dt.floor("12h")
d["acted"] = d.ack_minutes <= ack_target_min
per_shift = d.groupby(["unit", "week", "recipient", "shift"]).size().rename("n").reset_index()
load = per_shift.groupby(["unit", "week"]).n.mean().rename("alerts_per_clinician_shift")
out = d.groupby(["unit", "week"]).agg(alerts=("fired_at", "size"),
acted_share=("acted", "mean"),
ppv_confirmed=("confirmed", "mean"))
return out.join(load).round(3)
def response_curve(log: pd.DataFrame, ack_target_min: float = 60) -> pd.Series:
"""Share acted on within target, by how many alerts that clinician had already
received that shift. A falling curve is alert fatigue, measured."""
d = log.sort_values("fired_at").copy()
d["shift"] = d.fired_at.dt.floor("12h")
d["nth_alert"] = d.groupby(["recipient", "shift"]).cumcount() + 1
d["acted"] = d.ack_minutes <= ack_target_min
return d.groupby(d.nth_alert.clip(upper=8)).acted.mean().round(3)
if __name__ == "__main__":
# Synthetic log for a demonstration: acknowledgment slows as a shift's alerts pile up.
rng = np.random.default_rng(7)
n = 4000
fired = pd.Timestamp("2026-09-01") + pd.to_timedelta(rng.uniform(0, 56 * 24, n), unit="h")
log = pd.DataFrame({"fired_at": fired,
"unit": rng.choice(["4 West", "5 East", "Step-down"], n),
"recipient": rng.choice([f"rn{i}" for i in range(25)], n)})
log = log.sort_values("fired_at").reset_index(drop=True)
nth = log.groupby(["recipient", log.fired_at.dt.floor("12h")]).cumcount()
log["ack_minutes"] = rng.exponential(25 * 1.15 ** nth)
log["confirmed"] = (rng.random(n) < 0.25).astype(float)
pd.set_option("display.width", 120)
print(weekly_alert_report(log).head(6))
print(response_curve(log))Shifts are approximated as fixed 12-hour blocks; replace that line with your real shift schedule. Note one trap in the response curve: a clinician’s later alerts in a shift may also be on sicker or busier days, so a falling curve shows association, not a pure fatigue effect.
Proving that an alert saves lives is harder than monitoring it, and the series kept arriving at the same answer. Kaiser’s deterioration program switched on across 19 hospitals in a staggered order, and that design is why its result is credible: among alerted patients, 30-day mortality was 15.8% where the program was live and 20.4% where it was not yet live, an adjusted relative risk of 0.84 and an estimated 520 deaths averted a year. A stepped-wedge rollout like that, a silent-mode period that scores patients without showing alerts, or a regression discontinuity comparing patients just above and below the threshold all hold up far better than a before-and-after comparison.
Size the proof before you start, and prove the process before the outcome. Detecting the TREWS mortality effect takes about 1,957 septic patients per arm. Episode 3’s stroke analysis needed 85 patients per arm to prove the minutes saved and 41,446 per arm to prove the outcome. Episode 4’s capacity analysis needed 3,603 admissions per arm to prove a 0.3-day change in length of stay and 150,569 per arm to prove the change in mortality it buys. Minutes, acknowledgments and bed hours are where a single hospital can get an answer within a year.
Where the math breaks
The response curve is the weakest link, and it is mine. The form c₀ × (1 − δ)^(A/n − 1) is a proposed model chosen for clarity, and the δ values I swept are illustrations, not estimates. The evidence behind it comes from settings that differ from a hospital sepsis alert. Ancker’s 30% drop was for primary care reminders per encounter. Bonafide’s response times come from 40 filmed sessions at one children’s hospital, and four of the 36 nurses said they responded faster because they were being filmed. The TREWS volume finding is an association. It is also likely that response depends on PPV as well as volume, because clinicians learn how often an alert is right. A hospital that measures its own curve from audit logs will have a better model than this one within a quarter.
The binormal ROC is a convenience. Real score distributions often have unequal variances, and the sensitivity a model buys at a given budget should come from its empirical ROC on your own patients.
The lives-saved product treats its terms as independent. They are not. Sicker patients trigger more alerts, get faster responses and benefit differently from early treatment, and busy days lower c and a together. The product is a way to see which term is limiting, not a forecast.
Several inputs are derived rather than reported. The TREWS PPV of 25.4% and specificity of 94.9% are my back-calculations from published counts against a retrospective sepsis label; clinicians confirmed a higher share of the alerts they evaluated. The Michigan specificity of about 83% is computed the same way from that study’s published counts. Kaiser’s paper reports no PPV; the 0.08 figure is its alert threshold, and it equals a PPV floor only if the model’s risk estimates are accurate.
The time values have their own limits. The sepsis odds of 4% per hour come from an observational study, and sicker patients do not receive antibiotics at random times. The stroke value comes from pooled randomized trials, but a linear minutes-to-outcome rule is an approximation that may not hold at the extremes. And none of this counts the costs of false alerts beyond attention and queue position: unnecessary broad-spectrum antibiotics, extra lactates, avoidable ICU transfers.
What quality and informatics leaders can do this quarter
None of this needs a new model. It needs four numbers for every interruptive alert you already run, and most of them sit in the audit log.
Start with an inventory. List every alert that interrupts a clinician, from sepsis and deterioration to stroke, critical labs and capacity. For each one, compute the alerts per responding clinician per shift, the PPV against confirmed cases, the number needed to evaluate, and the share acted on within the target time. Most committees have never seen those four numbers side by side, and the alert with the worst NNE is rarely the one getting the most complaints.
Then set a budget for each receiving team with the Poisson code above, and move each threshold to the budget. Ask vendors for sensitivity at your alert budget on your patients, not AUROC on theirs. If a model cannot report that, run it in silent mode for three months and compute it yourself.
Next, measure the response curve. Group acknowledgments by how many alerts that clinician had already received in the shift. If the curve falls steeply, the cheapest lives available are in fewer, better alerts: raise specificity, suppress repeats on the same patient, and route high-volume alerts to a dedicated team, as Kaiser did, rather than to the bedside.
Finally, check the a term. For each alert, name the bed, team or room it depends on and look at its availability at the busiest hour of the week. An alert that fires into a full hospital or an unstaffed night shift will look like a model failure in every report, and no retraining will fix it.
If your team runs these numbers on its own alerts, I would like to hear what the response curve looks like. Here in Kansas City and everywhere else, the hospitals getting value from early warning systems are the ones that treat clinician attention as the scarcest resource in the building.
Keep reading. This piece draws on four episodes of How AI and ML Will Save Lives in Healthcare. The full sepsis deep dive, with the threshold code, the training pipeline and the study design, is on Substack: Episode 1, sepsis early warning. The alert budget math and the success paradox are in Episode 2, ward deterioration. The stroke queue model is in Episode 3, stroke detection, and the capacity term is in Episode 4, hospital capacity and patient flow, with its companion piece Why Emergency Departments Back Up Long Before the Hospital Looks Full on my Medium page. Episode 5, on readmissions and care transitions, lands November 5. Subscribe on Substack to get each episode as it publishes.
About the Author
Brian James Curry is a Kansas City AI researcher, applied scientist and data scientist with more than twenty years in marketing modeling and measurement. His work today spans marketing mix modeling, causal inference, machine learning and healthcare intelligence. He is the founder of Vector1 Labs, an independent applied AI research lab. His marketing measurement work covers marketing mix modeling, incrementality testing and budget allocation, and his Causal Care series applies causal machine learning to health system decisions: which programs work, for whom, and what they are worth. He also founded KC AI Lab, which began in 2016 as the Kansas City Machine Learning Group and has grown to more than 1,500 members.
His career includes leading analytics, data science and machine learning at Hallmark, Garmin, Koch Industries, Tractor Supply Company and Vail Resorts. His open-source research tools on GitHub include Papilon (marketing mix modeling and complex systems), PyCausalSim (causal discovery through simulation) and tincast (forecasting for AI-critical commodities), along with MeaningFlow and Daedalus. A Kansas State University alum from a small Kansas farming town, he writes about marketing mix modeling, causal inference, agentic AI, healthcare AI and the economics of AI on Medium and Substack. Contact: brian@vector1.ai.
Sources
- Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Critical Care, 2015.
- van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. Journal of the American Medical Informatics Association, 2006.
- Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making 17:36, 2017.
- Bonafide CP, Lin R, Zander M, et al. Association between exposure to nonactionable physiologic monitor alarms and response time in a children’s hospital. Journal of Hospital Medicine, 2015.
- Henry KE, Adams R, Parent C, et al. Evaluating adoption, impact, and factors driving adoption for TREWS, a machine learning-based sepsis alerting system. medRxiv preprint, 2021. Source of the 6.7% alert rate, 82% sensitivity, 2.1% prevalence, 38% provider confirmation and the alert volume finding.
- Adams R, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nature Medicine, 2022.
- Healthcare in Europe. Early detection of sepsis with the help of AI, 2022, reporting the 1.85-hour median antibiotic difference in the TREWS evaluation.
- Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 2021. Counts from the 2 Minute Medicine summary; the 12% PPV from Michigan Medicine’s announcement.
- Seymour CW, Gesten F, Prescott HC, et al. Time to treatment and mortality during mandated emergency care for sepsis. New England Journal of Medicine, 2017.
- Escobar GJ, Liu VX, Schuler A, et al. Automated identification of adults at risk for in-hospital clinical deterioration. New England Journal of Medicine, 2020.
- A real-world, prospective diagnostic accuracy study of AI software for detecting large vessel occlusion stroke on CT angiography. Journal of NeuroInterventional Surgery, 2023.
- Golan D, et al. AI-powered stroke triage system performance in the wild. 2020 (authors from the vendor).
- Saver JL, et al. Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA, 2016. Summary at Endovascular Today.
- Martinez-Gutierrez JC, et al. Automated large vessel occlusion detection software and thrombectomy treatment times: a cluster randomized clinical trial. JAMA Neurology, 2023.
- Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLOS ONE, 2018.
- Curry B. How AI and ML Will Save Lives in Healthcare, Vector1 Research on Substack, October 2026: Episode 1, Episode 2, Episode 3, Episode 4, for the simulations, sample sizes and worked examples cited from the series.
Method note: the TREWS PPV and specificity and the Michigan specificity are calculated from published counts, as shown in the text. The alert volumes, PPVs, alert budgets, sensitivities at a budget and response-curve results come from the code in this piece, run as published. The 1.2-point sepsis time value applies an odds ratio of 1.04 per hour over 1.85 hours to a 20% baseline mortality (odds 0.25 × 1.04¹.85, converted back to a probability of 21.2%).
