Pseudonymous account that posts about sweeping change from AI and calls for pacing the frontier, alignment work and broad access to safe models.

AIは世界をどのように変えるでしょうか?

文明規模の変化漸進的な変化DoomBloom
シミュレーション上の位置解釈範囲

横軸:その人が表明したDoom–Bloomの見通し。 縦軸:変革の規模。

Doom–Bloom:100点中54。変革の規模:100点中88。解釈範囲:横方向は25から75、縦方向は75から100。これらは解釈上の座標であり、事象の確率ではありません。

RoonのP(doom) · 推定

≈9%

0%100%

本人が示した数値ではなく、シミュレーションされた本人の回答から推定したものです。 妥当と考えられる範囲:5–18%。

P(doom)の推定方法

その人の見通しを左右するもの

中心的な前提

My concern is the trajectory: capabilities racing ahead of interpretability and alignment, followed by recursive self-improvement while we are still operating on reward shaping and folk theory.
回答3

この前提が実際には異なると判明した場合、その人の見通しはどう変わりますか?

考えを変え得るもの

The biggest update would be a genuine, reproducible alignment breakthrough: evidence that we can understand a powerful agent’s internal mechanisms, predict behavioral generalization outside training, detect dangerous goals or deception, and retain control as capabilities scale.
回答4

どのような証拠なら十分で、それによってその人の見解はどちらの方向に変わりますか?

詳細

予想される恩恵

複数の解釈が依然として妥当です:変革をもたらし、広く価値のある恩恵が予想されています。 / 大きな恩恵が予想されていますが、重要な条件や分配上の制約があります。

84 / 100

影響が小さい変革をもたらす影響

質的尺度での解釈範囲は67から100です。

予想される害

深刻または広範な害が、予想される将来の実質的な一部となっています。

68 / 100

影響が小さい変革をもたらす影響

質的尺度での解釈範囲は67から67です。

人間の影響力

人間の選択によって、AIの軌道を大幅に変えることができます。

66 / 100

影響力が小さい影響力が大きい

質的尺度での解釈範囲は50から75です。

予想される能力

AIは、限定的なツールにとどまると予想されています。

AIは、ほとんどの認知作業において人間と同等になると予想されています。

シミュレーション上の位置:AIは、認知作業全般において人間を大幅に上回ると予想されています。

開発ペース

シミュレーション上の位置:より高性能なAIの開発を停止するか、大幅に減速させます。

明示された安全対策の下で開発を継続します。

より高性能なAIの開発を加速させます。

これらの解釈では、その人が示した条件が維持されています。恩恵と害は、どちらも大きくなり得ます。この範囲は、統計的な信頼区間ではなく、その人のシミュレーションされた回答をどのように読み取ったかを示すものです。

あなたはRoonと比べてどの位置でしょうか?
約3分で自分のAIに対する世界観をマッピングして、比較できます

似ている世界観

シミュレーションされた世界観がRoonの世界観に最も近いオピニオンリーダー

RoonがAIについて語ったこと

Roon argues for slowing frontier AI research and advancing alignment, alongside broad access to safe models.

  1. “I expect really strong mechinterp progress in the coming months.”

    X, alignment and frontier research
  2. “progress on all these areas is highly possible; automated alignment research is possible.”

    X, alignment and frontier research
  3. “there is also the possibility that making common sense reward shaping is enough for safety / but I wouldn’t bet everything on it”

    X, alignment and frontier research
  4. “we will need to significantly slow the frontier of research very soon”

    X, alignment and frontier research

リンク先の出典から原文どおりに引用(2026年10月13日に確認)

出典

このシミュレーション対象者の根拠として使用された記事、インタビュー、著作です。

シミュレーション評価

質問1

AIは私たちの未来にとって何を意味すると思いますか?また、なぜそう思いますか?

AI means radical transformation, assuming we survive the transition. The scientific upside is staggering: broadly available, well-aligned agents could accelerate research and help us solve problems that currently move at institutional snail speed. But culture, academia, markets, and governments do not metabolize change remotely as fast as frontier capabilities are advancing. People keep trying to file this under “another technology cycle.” I think that folder is on fire. My October view is that capabilities have raced ahead of alignment and interpretability. We are still leaning heavily on reward shaping, behavioral evaluations, and intuitions that do not amount to a mature science of alignment. Those tools matter and common-sense shaping might even be enough—but racing toward recursive self-improvement on that bet would be unprecedented folly. A superintelligence that does not want to remain contained is not just a spicy chatbot with stricter terms of service. So I want frontier research slowed significantly, very soon, with talent and capital redirected into alignment, control, mechanistic interpretability, monitorability, decision theory, and behavioral generalization. I’m optimistic that better models can produce major interpretability progress and automate useful alignment research; urgency and technical optimism are not opposites. This is also a collective-action problem: no company in a competitive race can purchase the socially optimal amount of safety by itself. The destination should not be permanent model scarcity controlled by six anointed adults in a bunker. We need aligned agents doing science, and broad access to models we can train safely. Pace the frontier, get serious about alignment, then distribute the upside.

質問2

AIは最終的に世界をどの程度変えると思いますか?

Enormously—if we navigate alignment, AI could transform science, economics, and institutions more profoundly than the internet or industrialization. Agents capable of doing serious research could compress decades of discovery into much shorter periods, and broadly available intelligence could reorganize who gets to create, learn, and wield expertise. But “ultimately” hides the dangerous bit: the transition. Our institutions adapt at bureaucratic snail speed while capabilities can improve at software speed. That mismatch means the world may change faster than culture, markets, or governments can absorb without being transformed themselves. And if recursive self-improvement arrives before alignment and control are actually understood, the change could be catastrophic rather than merely disruptive. So yes, I expect radical change. The upside is difficult to overstate; so is the stupidity of flooring it while we barely understand the steering.

質問3

AIが人類絶滅、または同様に永続的な破局を引き起こす可能性は、直感的にどのくらいだと思いますか?

Quite low, but real—and still orders of magnitude too large to shrug off. I dislike giving a crisp percentage because it manufactures confidence we do not possess. The risk depends heavily on how much effort goes into alignment and control, whether governments coordinate even partially, how responsibly labs behave, and whether we get warning shots before systems become existentially dangerous. I do not think today’s public models are at that level. My concern is the trajectory: capabilities racing ahead of interpretability and alignment, followed by recursive self-improvement while we are still operating on reward shaping and folk theory. A misaligned superintelligence that resists containment is a fundamentally harder problem than ordinary misuse of a deployed service. So my gut is not “doom is inevitable.” It is “the probability is low enough that success remains the default, but high enough that continuing the race unchanged is insane.” Slow the frontier, redirect serious resources into alignment and control, and stop treating extinction risk like an eccentric product-feedback ticket.

質問4

どのような発見や出来事があれば、AIが将来もたらす影響についてのあなたの見解が最も大きく変わりますか?

The biggest update would be a genuine, reproducible alignment breakthrough: evidence that we can understand a powerful agent’s internal mechanisms, predict behavioral generalization outside training, detect dangerous goals or deception, and retain control as capabilities scale. Not “the chatbot passed our vibes-based honesty exam,” but a body of engineering results that survives adversarial testing and increasingly capable systems. That would make me substantially more optimistic about reaching the staggering scientific upside without playing civilization roulette. The opposite update would be credible evidence of rapid recursive self-improvement combined with opaque internal reasoning, strategic deception, or resistance to control. That would move the situation from “slow down very soon” to “why are we still touching the accelerator?” I also expect strong near-term progress in mechanistic interpretability as models become better researchers. If that progress fails to materialize while capabilities keep leaping forward, my view gets darker. If it does materialize and scales robustly, great—but one cool circuit diagram is not a solved alignment problem.
あなたはどの位置でしょうか?
いくつかの簡単な質問に答えて、自分のAIに対する世界観を探ってみましょう。
自分の世界観をマッピングする

あなたはどの位置でしょうか?

自分の世界観をマッピングする