More or Less · Topics · Episode #170
Infinite Upside, Infinite Liability: The Religion of the AI Labs and the Lab-Leak Insurance Nobody Wants
Clip from the full recording, 18:46–23:06 · download mp4 · watch at this point on YouTube
Dave reports a dinner where lab researchers were asked who believes only one company survives; Sam says the zero-and-infinity religion is causing bad decisions, like ignoring a very practical liability framework for lab leaks.
Dave interrupts to describe a dinner last week with researchers from both major labs where one opened by asking who at the table believes there will be only one company left. The people from the labs raised their hands. Sam says that’s religious but real, and flips it: the labs are also creating infinite liability, so if you believe P(doom) is even one percent and the downside is infinite, you should spend infinite resources to stop it, which is a useless statement in terms of actually running the world.
Dave adds that one lab believes that if you don’t bring about the machine god, it will go back in time and put you in a simulation of hell. Sam says that’s the point: this religious infestation of zero and infinity is causing bad decision-making. Forget cataclysm, he says; there’s a real chance of lab leaks that cause trillions of dollars of damage, Katrina-scale events that freeze up banks.
His practical answer is that we could do something about it: look at how post-2008 forced cash reserves and insurance, and make labs carry insurance or balance sheet so the risk isn’t dumped on taxpayers. Dave pushes back that withholding top models while giving big companies the best ones to harden themselves is a kind of human prudence. Sam closes by worrying that decision-making is so consolidated, and held by hyper-religious people, that sensible business answers get missed.
Key points
- Dave: at a recent dinner, a lab researcher asked who believed only one company would be left; the lab people raised hands.
- Sam: the flip side of infinite upside is infinite liability, and P(doom) logic implies spending infinite resources, which is not usable policy.
- Dave: one lab believes a machine god will put non-builders into a simulated hell.
- Sam: a real near-term risk is lab leaks causing Katrina-scale, trillion-dollar damage.
- Sam: a post-2008 style regime (cash reserves, mandatory insurance) would make labs pay for the risk instead of dumping it on the US.
- Dave: giving top companies the strongest models first to secure themselves is a human, sensible way to handle it.
Where they landed
Quotes
“the labs are creating infinite liability and the best case is that they're zero.”— Sam
“you can't dump that risk on the US government or the US people, like you should pay for that”— Sam
“because we're so obsessed with the tales of zero and infinity and the religion, we're missing actually good decisions in business”— Sam
Suggested tweets
Dave at an AI lab dinner: "who at this table believes there will be only one company left?" Sam: the labs also create infinite liability. If you believe P(doom), you'd spend infinite resources. Useless.
Tweet this →Sam on lab leaks: Katrina-size damage that freezes banks. His fix is boring and practical: post-2008 style cash reserves and insurance, so you can't dump that risk on the US.
Tweet this →Sam: "we're so obsessed with the tales of zero and infinity and the religion, we're missing actually good decisions in business." Liability insurance for labs, anyone?
Tweet this →All topics · Full episode #170 + transcript · Subscribe on YouTube · Spotify