What if someone gave that alien $20 and AI is the reward? And what if AI ends up becoming the way we can communicate with that higher power? I'm really disappointed you didn't use XCH in your example though, lol.
Bram, the issue with people comparing AI risk to Pascal's Mugging - or even Pascal's wager for that matter - is that these scenarios involve compensating for tiny (sub-1%) probabilities by claiming that sufficiently large values of benefit/harm outweigh them in utilitarian calculus.
But there's extremely widespread belief, documented by many surveys, that the chance of AI-driven extinction in the next few decades is 5%+. So no such outweighing is necessary, even if it does happen to be the case that very large benefits & harms are also being discussed.
This is the exact same reason it would be wrong to accuse nuclear annihilation of being a Pascal's Mugging or Pascal's Wager.
I did read it. I read your work regularly. You seem like a precise thinker. I don't get why you're willing to be this imprecise about the meaning of Pascal's Mugging.
It's the same structure of argument. Some extremely high value thing, either revolutionize the world with alien tech or save the world from AI-driven catastrophe, times some very small value thing being able to deliver it, either by giving money to some rando or by giving money to some people doing work which doesn't seem to have anything to do with AI. As would be expected the numbers are slightly more extreme in the thought experiment. In both cases the counter is that resources allocated should be put towards the approaches with highest likely return on investment. I'm all in favor of both searching for alien life and working on AI safety, but those should be done with astronomy and space exploration and working on actual AI, not talking to 'alien abduction' victims or doing whatever it is that the safety-only research groups do.
Ah I see, you’re not claiming a tiny probability of AI doom, you think the argument to protect ourselves isn’t Pascal-like, but you think there’s only a tiny probability of mitigation caused by supporting dedicated AI safety orgs, so then arguing for supporting those orgs would be Pascal-like. Fair enough on that logic.
And ya, I don’t think virtually anyone thinks AI safety orgs have a highly effective plan at hand, because the problem is so thorny in so many ways. MIRI for instance doesn’t have much hope of their own success even conditional on $1B in funding.
If you don’t want to fund AI safety orgs because they don’t have a plan, I would probably disagree with the details of the expected-value math, but I wouldn’t pull a Pascalian argument insisting that you have to multiply by an astronomical number, I would just understand you don’t want to fund orgs that don’t have a clear plan.
It’s just that all we can do is choose among bad options. And in particular, any option that involves further progress toward superintelligence is a pretty high probability of accelerating doom. That’s the part of the argument that happens to have an astronomical negative utility term, yet is still a standard non-Pascalian argument without any tiny probability terms.
For alien contacts, the calculation goes: Benefits of alien tech (huge) times chances that aliens are nearby (tiny) times chances this particular person has contact with them (tiny). For AI rescue, it goes: Benefits of rescue from AI catastrophe (huge) times chances of AI catastrophe happening in the normal course of things (small but not tiny) times chances of certain researchers saving us (tiny). In both cases I'm grouping them as A*(B*C). If you group either of them as (A*B)*C other issues come up which are beside the point.
Right but the argument I would make is that we have a modest (not tiny) chance of avoiding AI doom by not working on AI, so expected value of NOT working on AI = modest chance * huge avoidance of loss.
We don't necessarily need researchers to save us if we can keep coordinating to pause.
So while any particular researcher may have a plan that's in the <1% Pascal probability zone, the plan of coordinating to stop building AI is a better-understood 5%+ type bet, and that gives us a lower bound on success probability of the category of calls to action helping to avoid AI doom.
What if someone gave that alien $20 and AI is the reward? And what if AI ends up becoming the way we can communicate with that higher power? I'm really disappointed you didn't use XCH in your example though, lol.
It would be a story worth $20 to tell
Bram, the issue with people comparing AI risk to Pascal's Mugging - or even Pascal's wager for that matter - is that these scenarios involve compensating for tiny (sub-1%) probabilities by claiming that sufficiently large values of benefit/harm outweigh them in utilitarian calculus.
But there's extremely widespread belief, documented by many surveys, that the chance of AI-driven extinction in the next few decades is 5%+. So no such outweighing is necessary, even if it does happen to be the case that very large benefits & harms are also being discussed.
This is the exact same reason it would be wrong to accuse nuclear annihilation of being a Pascal's Mugging or Pascal's Wager.
I hope you'll retract your claim.
I am not retracting my claim, and I hope you read my actual post
I did read it. I read your work regularly. You seem like a precise thinker. I don't get why you're willing to be this imprecise about the meaning of Pascal's Mugging.
It's the same structure of argument. Some extremely high value thing, either revolutionize the world with alien tech or save the world from AI-driven catastrophe, times some very small value thing being able to deliver it, either by giving money to some rando or by giving money to some people doing work which doesn't seem to have anything to do with AI. As would be expected the numbers are slightly more extreme in the thought experiment. In both cases the counter is that resources allocated should be put towards the approaches with highest likely return on investment. I'm all in favor of both searching for alien life and working on AI safety, but those should be done with astronomy and space exploration and working on actual AI, not talking to 'alien abduction' victims or doing whatever it is that the safety-only research groups do.
Ah I see, you’re not claiming a tiny probability of AI doom, you think the argument to protect ourselves isn’t Pascal-like, but you think there’s only a tiny probability of mitigation caused by supporting dedicated AI safety orgs, so then arguing for supporting those orgs would be Pascal-like. Fair enough on that logic.
And ya, I don’t think virtually anyone thinks AI safety orgs have a highly effective plan at hand, because the problem is so thorny in so many ways. MIRI for instance doesn’t have much hope of their own success even conditional on $1B in funding.
If you don’t want to fund AI safety orgs because they don’t have a plan, I would probably disagree with the details of the expected-value math, but I wouldn’t pull a Pascalian argument insisting that you have to multiply by an astronomical number, I would just understand you don’t want to fund orgs that don’t have a clear plan.
It’s just that all we can do is choose among bad options. And in particular, any option that involves further progress toward superintelligence is a pretty high probability of accelerating doom. That’s the part of the argument that happens to have an astronomical negative utility term, yet is still a standard non-Pascalian argument without any tiny probability terms.
For alien contacts, the calculation goes: Benefits of alien tech (huge) times chances that aliens are nearby (tiny) times chances this particular person has contact with them (tiny). For AI rescue, it goes: Benefits of rescue from AI catastrophe (huge) times chances of AI catastrophe happening in the normal course of things (small but not tiny) times chances of certain researchers saving us (tiny). In both cases I'm grouping them as A*(B*C). If you group either of them as (A*B)*C other issues come up which are beside the point.
Right but the argument I would make is that we have a modest (not tiny) chance of avoiding AI doom by not working on AI, so expected value of NOT working on AI = modest chance * huge avoidance of loss.
We don't necessarily need researchers to save us if we can keep coordinating to pause.
So while any particular researcher may have a plan that's in the <1% Pascal probability zone, the plan of coordinating to stop building AI is a better-understood 5%+ type bet, and that gives us a lower bound on success probability of the category of calls to action helping to avoid AI doom.