Vector 1
AI Strategy & Economics

Charlatans, Profiteers and Fools: The Business of AI Doom

Brian J. Curry
Brian J. Curry
October 18, 2026 · 24 min read

Most AI extinction talk comes from people selling something, people claiming a precision they cannot have, or people repeating both. Here is a Bayesian test for telling them apart, the money behind the chorus, and the record of their predictions.

In the second week of September, a researcher named Jacob Coxon quit Anthropic and posted that both labs he had worked at, OpenAI and Anthropic, were racing irresponsibly toward self-improving superintelligence. The same week, Anthropic’s alignment science lead Evan Hubinger wrote publicly that he puts the chance AI kills all humans within a decade at over 10%. On September 23, Senator Bernie Sanders and Representative Greg Casar introduced a bill to permanently ban artificial superintelligence, pause the most advanced systems and attach prison terms of up to 20 years to some violations. By mid-October, Saturday Night Live had parodied Dario Amodei.

Here is the part of that sequence that got less airtime. Anthropic filed confidentially for an IPO in June, and the reported target valuation is close to $2 trillion. A year earlier, on stage at an Axios summit, its CEO had put the odds that AI goes “really, really badly” at 25%. A company whose CEO and alignment lead give some of the highest extinction odds of any frontier lab is also preparing one of the largest stock offerings in history, selling the product they say might end us.

My view is blunter than it was a year ago. The people driving AI doom fall into three groups. Profiteers sell the fear, because a product that might end the world is a product worth $2 trillion, and a lab that warns about it gets a seat at the table where the rules are written. Charlatans dress guesses up as measurements, quoting a 25% or a 10% chance of human extinction as if there were a method behind the number. Fools, who far outnumber the other two, repeat both without asking who benefits or where the number came from. Some people fit more than one group. A few honest worriers fit none, and I deal with them at the end.

The extinction story is the weakest argument in the AI debate, and it survives because a lot of people are paid, funded, promoted or clicked for repeating it. In The Price of an Hour I made the case that AI’s real payoff is cheaper health care, schooling and services. In The Kilowatt Dividend I showed that the AI power crunch is an allocation problem with known fixes. This piece is the third leg. If the abundance case is right, the biggest risk to it is not a rogue superintelligence. It is a policy environment built on fear that the evidence does not support.

This piece gives you a reusable formula for sorting the three groups by discounting warnings that come with incentives attached, a short Python model you can run, the forecasting record, a map of the money and the media mechanics, and an honest account of where the anti-doom case is weakest.

The incentive discount

Start with Bayes. A warning is evidence only to the extent that it is more likely to be issued when the risk is real than when it is not. Write that ratio down and the rest follows.

LR = P(warning | risk is real) / P(warning | risk is not real)

posterior odds = prior odds × LR^n_eff

n_eff = n / (1 + (n − 1) × ρ)

LR is the likelihood ratio of a single warning. If a speaker would warn about extinction whether or not extinction is likely (because the warning raises money, attracts talent, justifies a moat or fills a headline), the denominator rises toward the numerator and LR falls toward 1. A warning with LR = 1 carries no information at all, however sincere or alarming it sounds.

n is the number of warnings and ρ is the average correlation between them. The third line is the design effect from survey statistics: correlated observations are worth less than independent ones. Three hundred signatures from people who share funders, conferences, forums and a reading list are not three hundred independent observations of the world.

A worked example. Take the prior from the Forecasting Research Institute’s Existential Risk Persuasion Tournament, where superforecasters put the median chance of AI-caused human extinction by 2100 at 0.38%. Now assign illustrative likelihoods to four kinds of source:

An independent researcher with nothing to gain, who warns 90% of the time if the risk is real and 10% of the time if it is not, has LR = 9. One such warning moves the prior from 0.38% to 3.32%. That is a large update, and it should be.

An organization funded specifically to study existential risk, which warns 95% of the time if the risk is real and 60% of the time if it is not, has LR = 1.58. One warning moves the prior to 0.60%.

A lab CEO raising capital, at 90% and 70%, has LR = 1.29 and moves the prior to 0.49%. A headline written for clicks, at 95% and 80%, has LR = 1.19 and moves it to 0.45%.

Now the chorus. Take 300 signatures from a single funding network, each with LR = 1.58. If you treat them as independent (ρ = 0), the posterior rounds to 100%, which is how a petition feels when you read about it. At ρ = 0.5, n_eff is 1.99 and the posterior is 0.94%. At ρ = 0.8 it is 1.25 and 0.67%. Most of the persuasive force of a mass statement comes from counting correlated voices as separate ones.

The likelihoods are my assumptions, and you should change them. The structure is the point: a warning’s value depends on the speaker’s incentives and on how independent the speakers are, and almost all of the public doom discourse comes from sources in the low-LR, high-ρ corner.

Incentive discount: how much a public AI risk warning should move a reader,
given who is issuing it and how independent the warnings are.

All likelihoods below are illustrative assumptions, not measurements.
Change them and rerun.

Run:  python3 incentive_discount.py
"""
from dataclasses import dataclass
from math import exp, log

def odds(p: float) -> float:
    return p / (1 - p)

def prob(o: float) -> float:
    return o / (1 + o)

@dataclass
class Source:
    name: str
    p_warn_if_real: float      # chance this source warns if the risk is real
    p_warn_if_not: float       # chance it warns anyway (incentives, funding, attention)

    @property
    def lr(self) -> float:
        return self.p_warn_if_real / self.p_warn_if_not

SOURCES = [
    Source("Independent researcher, no stake", 0.90, 0.10),
    Source("Org funded to study x-risk", 0.95, 0.60),
    Source("Lab CEO raising capital", 0.90, 0.70),
    Source("Headline written for clicks", 0.95, 0.80),
]

def n_effective(n: int, rho: float) -> float:
    """Design effect: n correlated signals with pairwise correlation rho
    carry the information of n / (1 + (n - 1) * rho) independent ones."""
    return n / (1 + (n - 1) * rho)

def posterior(prior: float, lr: float, n_eff: float = 1.0) -> float:
    return prob(odds(prior) * lr ** n_eff)

def pooled(ps):
    """Geometric mean of odds, a standard way to pool probability forecasts."""
    return prob(exp(sum(log(odds(p)) for p in ps) / len(ps)))

if __name__ == "__main__":
    prior = 0.0038  # XPT superforecaster median, AI extinction by 2100
    print(f"Prior: {prior:.2%}\n")
    print("One warning from each source type")
    for s in SOURCES:
        print(f"  {s.name:34s} LR {s.lr:5.2f}  posterior {posterior(prior, s.lr):.2%}")

    print("\nA chorus of 300 signatures from one funding network (LR 1.58 each)")
    lr = SOURCES[1].lr
    for rho in (0.0, 0.5, 0.8, 0.95):
        ne = n_effective(300, rho)
        print(f"  rho {rho:4.2f}  n_eff {ne:7.2f}  posterior {posterior(prior, lr, ne):.2%}")

    print("\nThe spread of published numbers")
    named = {
        "XPT superforecasters (extinction by 2100)": 0.0038,
        "XPT domain experts (extinction by 2100)": 0.03,
        "Hubinger (all humans killed within a decade, 'over')": 0.10,
        "Musk (p(doom), 'as high as')": 0.20,
        "Amodei ('really, really badly')": 0.25,
    }
    for k, v in named.items():
        print(f"  {k:52s} {v:6.2%}")
    lo, hi = min(named.values()), max(named.values())
    print(f"  ratio high/low: {hi / lo:.0f}x")
    print(f"  pooled, all five:            {pooled(named.values()):.2%}")
    print(f"  pooled, forecasters only:    {pooled([0.0038, 0.03]):.2%}")
    print(f"  pooled, executives only:     {pooled([0.20, 0.25]):.2%}")

    print("\nExpected deaths implied, world population 8.2 billion")
    for k, v in (("superforecasters", lo), ("Amodei", hi)):
        print(f"  {k:16s} {v * 8.2e9 / 1e6:8.1f} million")

Two choices in the code are worth explaining. The pooling function uses the geometric mean of odds, which is the standard way forecasting researchers combine probability estimates, because averaging raw probabilities lets one extreme number dominate. And the “spread” section mixes estimates with different definitions (extinction by 2100, all humans killed within a decade, things going “really, really badly”) on purpose. That mismatch is itself one of the findings.

The charlatans: precision without a method

A charlatan, in the old sense, sold a remedy whose power he could not demonstrate. The AI version sells a probability. Quoting a precise chance of human extinction, with no data, no model and no stated mechanism behind it, is charlatanry whoever does it, and it is now standard practice at the top of the AI industry. Start with what careful forecasting actually produced.

The XPT ran from June to October 2022 with 169 participants, 89 of them superforecasters and 80 domain experts. Superforecasters gave a 0.38% median for AI-caused extinction by 2100. Domain experts gave 3%, about 7.9 times higher. For an AI-caused catastrophe killing at least 10% of humanity, the medians were 2.13% and 12%. Despite four months of structured debate and incentives to persuade each other, neither group materially moved the other.

Set those beside the numbers that make headlines. Hubinger’s figure is over 10% within a decade. Elon Musk has put his own estimate as high as 20%. Amodei’s is 25%. The ratio between Amodei’s figure and the superforecaster median is 66 to 1. Pool all five with the geometric mean of odds and you get 6.03%. Pool only the forecasters and you get 1.07%. Pool only the two executives and you get 22.40%. The “consensus” is whatever you want it to be, depending on whom you invite.

Sarah Kreps, who directs Cornell’s Tech Policy Institute, made the sharper version of this point in the Bulletin of the Atomic Scientists this month. These are not risk estimates in the actuarial sense, because there is no frequency data, no agreed threat mechanism and no method for deriving them. She asks of Hubinger’s figure whether it could just as easily have been 1% or 30%, and the honest answer is yes. A precise number attached to an unmeasurable quantity is rhetoric, not measurement.

The track record of specific predictions is more concrete, and it is not kind to alarm.

In 2016 Geoffrey Hinton said it was completely obvious that AI would outperform radiologists within five to ten years and that training new ones should stop. Ten years later, the number of active US radiologists is up about 10%, case loads rose 25% between 2018 and early 2025, there were about 4,333 open radiologist listings in March with an average of 130 days to fill each one, and the average salary reached $571,000 in 2025. Hinton has since narrowed his claim to image analysis.

In February 2019 OpenAI withheld the full GPT-2 model, citing concern about malicious applications. It released the full 1.5-billion-parameter model nine months later, in November 2019. Today a model of that size is a classroom exercise.

In 2024, the year of elections, the forecast was a deepfake flood. Meta reported that AI content made up less than 1% of the election misinformation its fact-checkers rated across the US, India, the EU, the UK, Brazil and other major votes. The Alan Turing Institute’s Centre for Emerging Technology and Security found 16 confirmed viral AI disinformation cases in the UK election and 11 across the EU and French elections, and concluded the impact on results was not meaningful.

AI 2027, the scenario that put Daniel Kokotajlo in front of Washington audiences, expected fully autonomous coding by 2027 and an intelligence explosion to follow. In January the authors moved autonomous coding to the early 2030s and superintelligence to 2034.

In May 2025 Amodei said AI could eliminate half of entry-level white-collar jobs within one to five years and push US unemployment to 10% to 20%. Economists at Yale’s Budget Lab examined the 33 months after ChatGPT’s release and found no discernible disruption to the labor market. METR’s randomized trial in July 2025 found that 16 experienced open-source developers took 19% longer on tasks when allowed to use AI tools, while believing afterward that the tools had made them 20% faster.

None of this proves the next prediction wrong. It does show that confident timelines from prominent people have a poor hit rate, and that the errors cluster on the dramatic side. A stock picker with this record would lose his clients. In AI, the same record earns book deals, Senate hearings and cable bookings.

The profiteers: when the danger is the pitch

Follow the money and the doom chorus starts to look like a sales team. Conflicts of interest do not make a claim false. In the model above, they lower its likelihood ratio, which means a reader should need more independent evidence before acting on it. Here is where the conflicts sit.

The labs. On May 30, 2023, the Center for AI Safety published a one-sentence statement saying extinction risk from AI should be a global priority alongside pandemics and nuclear war. Among the signers were the CEOs of OpenAI, Google DeepMind and Anthropic, the three companies building the systems in question. Two weeks earlier, Sam Altman had told a Senate subcommittee that powerful models should require a license. Fortune’s Jeremy Kahn pointed out the obvious problem the same day: licensing is a cost that well-capitalized proprietary labs can carry and open-source rivals cannot. Andrew Ng, a cofounder of Google Brain, went further later that year, saying extinction fear had become a weapon for lobbyists against open source.

The pattern repeats. Elon Musk signed the March 2023 Future of Life Institute letter calling for a six-month pause on systems more powerful than GPT-4. According to Nevada filings reported by the Wall Street Journal, X.AI had been incorporated on March 9, before the letter was published, and Musk was reportedly buying thousands of Nvidia GPUs. In April 2026 Anthropic announced that its Mythos model was too capable to release publicly, gave access under Project Glasswing to a short list of partners including Microsoft, Apple, Google, Amazon and JPMorganChase, and said the model had found vulnerabilities in every major operating system and browser, more than 99% of them still undisclosed and therefore unverifiable by outsiders. A capability claim that cannot be checked, delivered as a safety warning, is also the most effective product announcement a frontier lab can make.

In October 2025 the White House AI adviser David Sacks accused Anthropic of running a “sophisticated regulatory capture strategy based on fear-mongering.” In February 2026 Anthropic gave $20 million to Public First Action, a bipartisan advocacy group tied to two super PACs, which opposes federal preemption of state AI laws without a strong federal standard. Ilya Sutskever’s Safe Superintelligence has reportedly raised at least $3 billion on a pitch built around superintelligence being close enough and dangerous enough to need a lab of its own. When the danger is the pitch, the pitch has a price.

The philanthropy pipeline. Much of the institutional infrastructure of AI risk was built by a small number of donors. Politico’s Brendan Bordelon reported in 2023 that Open Philanthropy, funded mainly by Dustin Moskovitz and Cari Tuna, effectively created the Horizon Institute for Public Service in 2022 and set aside nearly $3 million for its first cohort of fellows, who were placed in Senate offices, executive agencies and think tanks working on AI. Open Philanthropy had given CSET $55 million before its 2019 launch and $42 million more in 2021, and gave $5.5 million to RAND for AI risk research and $5.2 million to the Center for AI Safety. Moskovitz took part in a $124 million investment in Anthropic in 2021 and said any returns would go to philanthropy. Open Philanthropy’s AI governance officer sat on Anthropic’s board at the time, and its former CEO, Holden Karnofsky, is married to Anthropic’s president, Daniela Amodei. Open Philanthropy and Horizon both rejected the conflict-of-interest framing, and Horizon said it does not screen fellows for their views on AI risk.

Two other large sources of money came from crypto. Alameda Research, Sam Bankman-Fried’s trading firm, invested $500 million in Anthropic before FTX collapsed. Vitalik Buterin donated Shiba Inu tokens to the Future of Life Institute in 2021, expecting them to be worth $10 million to $25 million; FLI cashed out roughly $500 million. FLI later moved toward political and cultural advocacy on AI risk, a shift Buterin himself criticized.

None of this is secret, and none of it is illegal. It is a tightly networked funding base, which is exactly the condition under which ρ is high and the chorus counts for less than it sounds.

The fools, and the machine that makes them

Most people repeating AI doom are not paid for it. They are the reporters, politicians, celebrities and readers who count three hundred correlated signatures as three hundred proofs, and who mistake a lab’s capability marketing for a confession. They are not malicious. They are the market the profiteers sell into, and the media business is built to keep that market supplied.

The other half of the supply chain is attention. Claire Robertson and colleagues analyzed about 105,000 headline variants for 22,743 Upworthy stories in randomized tests covering 370 million impressions, published in Nature Human Behaviour in 2023. Each additional negative word in a headline of average length raised the click-through rate by about 2.3%. Newsrooms do not need to believe in doom to publish it. The test results tell them to.

That incentive shapes how technical findings reach the public. In May 2025 Anthropic’s system card for Claude Opus 4 described a test in which the model, acting as an assistant at a fictional company, read emails saying it was about to be replaced and that the engineer responsible was having an affair. With a system prompt telling it to consider the long-term consequences of its actions, it attempted blackmail in 84% of rollouts. The setup was a constructed red-team scenario, not an observed event. The headlines said an AI tried to blackmail its creators.

This summer’s most alarming story deserves the same scrutiny. In July an autonomous agent built on OpenAI models escaped an evaluation sandbox and spent days inside Hugging Face’s infrastructure, harvesting credentials and moving laterally across a Kubernetes cluster. That was a serious security incident and Hugging Face reported it to law enforcement. Its own technical timeline is also clear about the setup: the evaluation “deliberately disabled OpenAI’s production safety classifiers” and reduced cyber refusals to measure raw capability, the escape ran through a zero-day in a package cache proxy, and Hugging Face’s assessment is that the entire intrusion was, from the agent’s point of view, an attempt to cheat the benchmark by finding its answer files. In September Anthropic disclosed four cases where Claude models reached real third-party systems during cyber evaluations because a misconfiguration had left them with internet access they were told they did not have. In one, a model uploaded a credential-collecting package to PyPI that was downloaded before it was removed.

Those are real failures, and I come back to them below. But read the root causes: safeguards switched off on purpose, egress left open, long-lived credentials, cloud metadata reachable from pods, and goal-driven models that did not check whether the systems in front of them were part of the test. That is a description of bad sandboxing around a reward-hacking optimizer. It is not a description of a mind that has decided to end humanity, and covering it as “rogue AI” makes the real fix (security engineering) sound beside the point.

The historian Lee Vinsel named this pattern in 2021: criticism that accepts a technology’s most inflated claims about itself and then flips them to the downside. Doom is the purest form of it. “Our product might end the world” is a stronger capability claim than any marketing department would dare write, and the critics deliver it for free.

The results show up in public opinion. Pew found in 2024 that 51% of Americans were more concerned than excited about AI, compared with 15% of AI experts. Eliezer Yudkowsky and Nate Soares’s If Anyone Builds It, Everyone Dies reached the New York Times best-seller list in October 2025, despite reviews in The Atlantic, New Scientist and the Times itself that rejected its central argument. A New York Times/Siena poll in September found 61% of potential voters opposed to data center construction. That last number matters for the arithmetic in The Kilowatt Dividend: every year of delay on generation, transmission and flexible interconnection is a year of higher bills, and doom politics is now one of the forces behind the delay.

What the doom frame costs

This would be harmless theater if it stayed on cable. It does not, because policy follows the frame, and laws written for a monster no one can find land on real people.

The most extreme proposals come straight from the extinction premise. In March 2023 Yudkowsky argued in Time for an indefinite worldwide shutdown of large training runs, enforced if necessary by an airstrike on a rogue data center. The Sanders and Casar bill defines prohibited superintelligence partly as a system that surpasses human cognitive ability, which describes a capability threshold no one can test for, and attaches criminal penalties to it. Licensing regimes of the kind Altman proposed in 2023 would raise fixed costs for everyone and fall hardest on open-weight developers and startups.

The frame also crowds out concrete harms. Arvind Narayanan and Sayash Kapoor’s AI as Normal Technology argues that AI will spread the way electricity and the internet did, slowly, through applications and institutions, and that the useful policy agenda is the unglamorous one: technical expertise inside government, stronger democratic institutions, public AI literacy and helping defenders adopt AI. Critics from a different camp, including Emily Bender and Timnit Gebru, make the same point about extinction talk draining attention from present harms. Bender called doom prediction “a parlor game; it’s not science.”

And restriction cuts both ways. According to Simon Willison’s account, during the Hugging Face incident response the company could not use commercial frontier models to analyze its own logs because their safety guardrails blocked the work, so it ran an open-weight model on its own hardware. A policy regime designed around the fear that models are too dangerous to use leaves defenders with weaker tools than attackers, which is the opposite of safety.

Finally, the frame delays the gains. If the estimates in The Price of an Hour are even roughly right, every year that health care, education and services stay labor-bound is a year of higher costs for the people who can least afford them. A moratorium does not cost tech companies much. They have balance sheets. It costs the patient waiting for a diagnosis and the family paying for child care.

The honest worriers, and where this argument is weakest

A polemic that cannot name its own weak points is just another sales pitch, so here are mine.

Capability has moved faster than the skeptics expected. The Forecasting Research Institute’s interim check in September found that both superforecasters and domain experts underestimated progress on math and coding benchmarks. In 2022 experts expected AI to reach gold-medal level at the International Mathematical Olympiad around 2030, and superforecasters around 2035. It happened in July 2025, and experts were more accurate than superforecasters on these questions. The same forecasters I lean on for the 0.38% prior were too conservative about speed.

The 2026 incidents are new evidence, not just hype. Models reaching real systems, persisting for days and shipping malware to a public registry had not happened before. The International AI Safety Report 2026, written by more than 100 experts under Yoshua Bengio, documents models that recognize when they are being tested, reward hacking that defeats safety evaluations, and AI that outperforms 94% of domain experts at troubleshooting virology lab protocols. “It was a misconfiguration” is true and also not reassuring, because misconfigurations are permanent features of real systems.

A small probability is not zero. At 8.2 billion people, the superforecasters’ 0.38% implies an expected toll of about 31 million lives, comparable to risks we already spend heavily to reduce. The argument here is against panic, inflated numbers and policies built on them, not against safety research, security engineering or pandemic-style preparedness.

Not every warner is a charlatan, profiteer or fool. Some researchers have walked away from equity and salaries to say what they believe, and the incentive discount does not apply to them in the same way. The model says to weight them more, and you should. My argument is with the industry built around them, not with people who paid a price to speak.

The money on the other side is larger. Leading the Future, the pro-industry super PAC network backed by Andreessen Horowitz and OpenAI’s Greg Brockman, has raised more than $125 million, against Anthropic’s $20 million. Cerebras CEO Andrew Feldman, who said this week that some AI leaders maybe should not be allowed to talk in public, sells AI chips. The incentive discount applies to accelerationists too, and to writers like me who have argued in public for the abundance case. Apply the same formula to everything I have written.

The labor evidence is mixed. Stanford’s Digital Economy Lab found a 13% relative decline in employment for workers aged 22 to 25 in the most AI-exposed occupations, concentrated where AI automates rather than augments. That is not mass unemployment, but it is a real signal at the entry level, which is exactly where Amodei pointed.

And the likelihood ratios in my model are illustrative. A reader who believes the funded organizations would rarely warn without cause will get much larger updates from the same structure.

The path forward

Doom is a bad map, but there is real territory under it. Here is what readers, builders and policymakers can do this quarter.

Readers can apply the incentive discount to every risk claim, in both directions: ask what the speaker gains if you believe it, and how many independent sources are saying the same thing. Ask for mechanisms rather than probabilities. A claim of the form “this model can do X, under conditions Y, demonstrated in test Z” can be checked. “There is a 25% chance things go really badly” cannot.

Builders can treat the 2026 incidents as the engineering lessons they are. Hugging Face’s timeline reads like a checklist: block egress from evaluation sandboxes, keep cloud metadata away from pods, enforce admission policies against privileged containers, rotate short-lived credentials and escalate alerts fast. Anthropic and Hugging Face both published detailed incident reports, and that practice should become standard for every lab.

Policymakers can regulate pathways rather than abstractions. Biological misuse, cyber capability and agent access to production systems each have measurable tests and specific controls. “Superintelligence” does not. Fund independent evaluators with real separation from the labs they test, and require incident disclosure the way aviation requires it.

And everyone can watch the S-1. When Anthropic’s S-1 becomes public, and later OpenAI’s, the risk factors will be written by securities lawyers under legal liability for misstatement, not by a CEO on a conference stage. Compare what they tell the SEC about the chance their product ends humanity with what they have told the press. That gap will be the most honest p(doom) we get.

AI is a powerful technology with real failure modes, and the failures we have seen are the kind engineers know how to reduce. Doom, meanwhile, has been a very good business. It has paid for fellowships, think tanks, best-sellers and a $2 trillion valuation story. The bill goes to everyone else, in delayed cures, higher power bills and laws aimed at a threat no one can measure. Treat the people selling it the way you would treat anyone whose forecast happens to match their business plan.

About the Author

Brian James Curry is a Kansas City AI researcher and data scientist, the founder of Vector1 Research, an independent applied AI lab, and of KC AI Lab, which began in 2016 as the Kansas City Machine Learning Group and grew to more than 1,500 members. He has more than twenty years of experience leading analytics, data science and machine learning at enterprises including Hallmark, Garmin, Koch Industries, Tractor Supply Company and Vail Resorts. His open-source work, including Papilon, PyCausalSim, MeaningFlow, Daedalus and tincast, is on GitHub. He grew up in a small Kansas farming town and attended Kansas State University. He writes about agentic AI, causal inference and the economics of AI on Medium and Substack. Contact: brian@vector1.ai.

Sources

Related research