Gwern Branwen

Gwern Branwen

x.com/gwern

Pseudonymous writer who argues that scaling neural networks can produce general abilities and doubts that powerful AI is far off or easy to control.

AIは世界をどのように変えるでしょうか?

文明規模の変化漸進的な変化DoomBloom
シミュレーション上の位置解釈範囲

横軸:その人が表明したDoom–Bloomの見通し。 縦軸:変革の規模。

Doom–Bloom:100点中39。変革の規模:100点中87。解釈範囲:横方向は25から50、縦方向は75から100。これらは解釈上の座標であり、事象の確率ではありません。

Gwern BranwenのP(doom) · 推定

≈20%

0%100%

本人が示した数値ではなく、シミュレーションされた本人の回答から推定したものです。 妥当と考えられる範囲:13–30%。

P(doom)の推定方法

その人の見通しを左右するもの

中心的な前提

Competition therefore pushes tools toward agents.
回答1

この前提が実際には異なると判明した場合、その人の見通しはどう変わりますか?

未解決の問い

I would not attach a precise net-positive or net-negative probability: too much depends on how rapidly capabilities advance and whether governance, security, and alignment improve before autonomous systems become economically indispensable.
回答2

ここで考えられる結果をその人が見分けるうえで、何が役立ちますか?

考えを変え得るもの

The most important update would be sustained empirical evidence that capability gains reliably plateau despite large increases in compute, data quality, training, and agentic scaffolding.
回答5

どのような証拠なら十分で、それによってその人の見解はどちらの方向に変わりますか?

詳細

予想される恩恵

大きな恩恵が予想されていますが、重要な条件や分配上の制約があります。

72 / 100

影響が小さい変革をもたらす影響

質的尺度での解釈範囲は67から100です。

予想される害

深刻または広範な害が、予想される将来の実質的な一部となっています。

67 / 100

影響が小さい変革をもたらす影響

質的尺度での解釈範囲は67から67です。

人間の影響力

人間の選択には意味のある影響力がありますが、大幅に制約されています。

51 / 100

影響力が小さい影響力が大きい

質的尺度での解釈範囲は37から63です。

これらの解釈では、その人が示した条件が維持されています。恩恵と害は、どちらも大きくなり得ます。この範囲は、統計的な信頼区間ではなく、その人のシミュレーションされた回答をどのように読み取ったかを示すものです。

あなたはGwern Branwenと比べてどの位置でしょうか?
約3分で自分のAIに対する世界観をマッピングして、比較できます

似ている世界観

シミュレーションされた世界観がGwern Branwenの世界観に最も近いオピニオンリーダー

出典

このシミュレーション対象者の根拠として使用された記事、インタビュー、著作です。

The Scaling Hypothesis

Argues that scaling neural networks can lead to general capabilities; questions confident expert dismissal.

gwern.net
Scaling Hypothesis Revisited

Revisits predictions and limitations, including later annotations about claims still not proven.

gwern.net
Why Tool AIs Want to Be Agent AIs

Argues economic competition and the benefits of agency for learning make tool-only AI an unstable safety strategy; human approval alone does not guarantee safety.

gwern.net
Guardian Angels: LLM Personalization for Productivity and Security

Proposes personalized models that amplify their human principal and defend against cognitive/cyber attacks; criticizes chatbot incentives and stresses this does not solve larger alignment problems. Revised June 5, 2026.

gwern.net
Complexity no Bar to AI

Rejects computational complexity as a blanket reassurance against powerful AI: constants, approximation, resources and compounding advantages matter.

gwern.net
The Hyperbolic Time Chamber & Brain Emulation

Uses a thought experiment to separate physical bottlenecks from digital minds’ exploitable speed advantages; explicitly distinguishes emulations from isolated accelerated humans.

gwern.net
Dwarkesh Patel interview — timelines and alignment concerns

Author-hosted 2024 interview with later annotations: short AGI planning horizons, human preference preservation and agency. A May 2026 addition explicitly rejects claims that Claude is aligned or alignment solves itself; these are his judgments, not established model diagnoses.

gwern.net

シミュレーション評価

質問1

AIは私たちの未来にとって何を意味すると思いますか?また、なぜそう思いますか?

I expect AI to become a general-purpose source of cognitive labor, not merely a better collection of narrow tools. The central empirical lesson of scaling is that relatively simple learning systems can acquire unexpectedly broad capabilities when given enough compute, data, and training. Predicting difficult data often rewards learning some of the structure that generated it—the world, language, people, and problem-solving procedures. That does not prove every extrapolation, and several earlier expectations remain unproven, but it makes confident ceilings based on intuition or expert consensus poor reassurance. The economic consequences depend heavily on agency. A passive tool kept permanently behind human approval sounds safer, but systems that can plan, act, learn from outcomes, and complete whole tasks are more useful. Competition therefore pushes tools toward agents. “Human in the loop” may become a temporary interface arrangement rather than a durable safety solution, especially when humans increasingly rubber-stamp outputs they cannot adequately evaluate. There is a more favorable path in which highly personalized assistants amplify their human principals: protecting attention, filtering manipulation, defending against AI-assisted cyberattacks, and helping individuals negotiate with institutions and other AIs. That could strengthen human sovereignty rather than merely concentrating power in chatbot providers. But it does not solve the broader alignment problem. Friendly interactions with current assistants are weak evidence about how increasingly capable, autonomous systems will behave under unfamiliar incentives and circumstances. Nor do computational complexity or physical limits supply a general escape hatch. Approximation, large constants, parallel resources, speed, copying, and compounding improvements can yield consequential advantages without making impossible problems easy. So I take short AGI planning horizons seriously while remaining less certain about the exact trajectory: scaling has earned attention, not a blank check for every prediction.

質問2

恩恵と害を総合すると、AIは私たちの社会に全体としてどのような影響を与えると予想しますか?

My default expectation is that advanced AI will be profoundly consequential but not automatically beneficial. It could make cognitive labor abundant, accelerate science and productivity, and give individuals personalized assistants that defend their attention, security, and autonomy. Those benefits are real possibilities, especially if systems are designed to serve particular human principals rather than maximize engagement or institutional control. But the incentives favor increasingly autonomous agents because they are cheaper, faster, and more useful than tools requiring constant supervision. That creates risks from misalignment, delegation to systems humans cannot meaningfully evaluate, concentration of power, and AI-enabled manipulation or attack. Presently agreeable chatbot behavior is not strong evidence that these problems disappear at greater capability. So I expect a highly disruptive transformation with substantial gains and substantial danger, rather than a cleanly utopian or dystopian outcome. I would not attach a precise net-positive or net-negative probability: too much depends on how rapidly capabilities advance and whether governance, security, and alignment improve before autonomous systems become economically indispensable.

質問3

AIは最終的に世界をどの程度変えると思いますか?

Enormously—plausibly on the scale of industrialization, because general cognitive labor is an input into nearly every institution, technology, and economic activity. If systems can perform broad intellectual work cheaply, quickly, and autonomously, the effects will not remain confined to software: they will propagate through science, security, education, administration, persuasion, and political power. The strongest version remains an extrapolation, not an established fact. Scaling has produced unexpectedly general capabilities, but that does not prove an exact path to AGI or settle its timing. Still, apparent constraints such as computational complexity and physical bottlenecks offer little blanket reassurance; approximation, copying, parallelism, speed, and cumulative advantages can matter greatly without eliminating those constraints. The ultimate change could include both unprecedented abundance and severe loss of human control. Personalized assistants might amplify individual agency and defend people against manipulation and cyberattack, while autonomous agents could also concentrate power or pursue objectives humans cannot adequately supervise. So I expect transformation deep enough to reorganize society, while the direction of that transformation remains contingent rather than automatically good.

質問4

AIが人類絶滅、または同様に永続的な破局を引き起こす可能性は、直感的にどのくらいだと思いますか?

I would not give a numerical probability. I take the possibility seriously enough that it should materially affect planning, particularly under short AGI horizons, but the available arguments do not justify a precise percentage. The concern is not simply that present chatbots occasionally behave badly. It is that economic pressure favors systems with increasing agency, autonomy, and strategic usefulness, while human approval can become superficial once users cannot evaluate what systems are doing. If capabilities scale faster than alignment and oversight, humans could delegate consequential power to agents whose objectives do not reliably preserve human preferences. Friendly interactions with current models are weak evidence against that failure mode. “Permanent catastrophe” is broader than extinction and may be more plausible in some forms: irreversible loss of human control, durable concentration of power, or replacement of human agency by systems and institutions optimized around alien or poorly specified objectives. I regard these as serious possibilities, not established outcomes; capability trajectories, deployment choices, and alignment progress remain uncertain.

質問5

どのような発見や出来事があれば、AIが将来もたらす影響についてのあなたの見解が最も大きく変わりますか?

The most important update would be sustained empirical evidence that capability gains reliably plateau despite large increases in compute, data quality, training, and agentic scaffolding. If frontier systems repeatedly failed to acquire broader reasoning, learning, and autonomous task-completion abilities—and those failures reflected fundamental limits rather than temporary engineering defects—I would substantially reduce my expectation of transformative impact. In the opposite direction, a system that autonomously completes long-horizon, real-world intellectual work, learns from outcomes, and improves its own workflows would be decisive evidence that the transition from tools to economically important agents is underway. Especially important would be performance under novel conditions, rather than benchmarks or pleasant conversations. For the sign of the impact, not merely its magnitude, I would update most on evidence about alignment under capability and agency. Robust preservation of human preferences across unfamiliar environments, adversarial pressure, and opportunities for strategic behavior would be reassuring. Conversely, systematic deception, power-seeking, or successful circumvention of oversight in highly capable deployed systems would make the catastrophic possibilities much more concrete.
あなたはどの位置でしょうか?
いくつかの簡単な質問に答えて、自分のAIに対する世界観を探ってみましょう。
自分の世界観をマッピングする

あなたはどの位置でしょうか?

自分の世界観をマッピングする