Episode 137
I spoke with <a href="https://x.com/davidad" target="_blank" rel="noopener">Davidad Dalrymple</a> about:
* His perspectives on AI risk
* ARIA (the UK’s Advanced Research and Invention Agency) and its Safeguarded AI Programme
Enjoy—and let me know what you think!
Davidad is a Programme Director at ARIA. He was most recently a Research Fellow in technical AI safety at Oxford. He co-invented the top-40 cryptocurrency Filecoin, led an international neuroscience collaboration, and was a senior software engineer at Twitter and multiple startups.
Find me on <a href="https://twitter.com/spaniel_bashir" target="_blank" rel="noopener">Twitter</a> for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions. 
Subscribe to The Gradient Podcast:  <a href="https://podcasts.apple.com/us/podcast/the-gradient-podcast/id1569777340" target="_blank" rel="noopener">Apple Podcasts</a>  | <a href="https://open.spotify.com/show/6onNcSqsP6hEEqmZ6TU2g8" target="_blank" rel="noopener">Spotify</a> | <a href="https://pca.st/itunes/1569777340" target="_blank" rel="noopener">Pocket Casts</a> | <a href="https://api.substack.com/feed/podcast/265424/s/1354/private/3bbc085c-7aa9-4037-8784-1575653a0438.rss" target="_blank" rel="noopener">RSS</a>Follow The Gradient on <a href="https://twitter.com/gradientpub" target="_blank" rel="noopener">Twitter</a>
Outline:
* (00:00) Intro
* (00:36) Calibration and optimism about breakthroughs
* (03:35) Calibration and AGI timelines, effects of AGI on humanity
* (07:10) Davidad’s thoughts on the Orthogonality Thesis
* (10:30) Understanding how our current direction relates to AGI and breakthroughs
* (13:33) What Davidad thinks is needed for AGI
* (17:00) Extracting knowledge
* (19:01) Cyber-physical systems and modeling frameworks
* (20:00) Continuities between Davidad’s earlier work and ARIA
* (22:56) Path dependence in technology, race dynamics
* (26:40) More on Davidad’s perspective on what might go wrong with AGI
* (28:57) Vulnerable world, interconnectedness of computers and control
* (34:52) Formal verification and world modeling, Open Agency Architecture
* (35:25) The Semantic Sufficiency Hypothesis
* (39:31) Challenges for modeling
* (43:44) The Deontic Sufficiency Hypothesis and mathematical formalization
* (49:25) Oversimplification and quantitative knowledge
* (53:42) Collective deliberation in expressing values for AI
* (55:56) ARIA’s Safeguarded AI Programme
* (59:40) Anthropic’s ASL levels
* (1:03:12) Guaranteed Safe AI — 
* (1:03:38) AI risk and (in)accurate world models
* (1:09:59) Levels of safety specifications for world models and verifiers — steps to achieve high safety
* (1:12:00) Davidad’s portfolio research approach and funding at ARIA
* (1:15:46) Earlier concerns about ARIA — Davidad’s perspective
* (1:19:26) Where to find more information on ARIA and the Safeguarded AI Programme
* (1:20:44) Outro
Links:
* Davidad’s <a href="https://x.com/davidad" target="_blank" rel="noopener">Twitter</a>
* <a href="https://www.aria.org.uk/" target="_blank" rel="noopener">ARIA homepage</a>
* <a href="https://www.aria.org.uk/programme-safeguarded-ai/" target="_blank" rel="noopener">Safeguarded AI Programme</a>
* Papers
* <a href="https://arxiv.org/abs/2405.06624" target="_blank" rel="noopener">Guaranteed Safe AI</a>
* Davidad’s <a href="https://www.alignmentforum.org/posts/pKSmEkSQJsCSTK6nH/an-open-agency-architecture-for-safe-transformative-ai" target="_blank" rel="noopener">Open Agency Architecture for Safe Transformative AI</a>
* <a href="https://research.protocol.ai/publications/dioptics-a-common-generalization-of-open-games-and-gradient-based-learners/dalrymple2019.pdf" target="_blank" rel="noopener">Dioptics: a Common Generalization of Open Games and Gradient-Based Learners</a> (2019)
* <a href="https://dspace.mit.edu/handle/1721.1/44914" target="_blank" rel="noopener">Asynchronous Logic Automata</a> (2008)
Get full access to The Gradient at <a href="https://thegradientpub.substack.com/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_4">thegradientpub.substack.com/subscribe</a>

Davidad Dalrymple: Towards Provably Safe AI

Episode 137 I spoke with Davidad Dalrymple about: * His perspectives on AI risk * ARIA (the UK’s Advanced Research and Invention Agency) and its Safeguarded AI Programme Enjoy—and let me know what you think! Davidad is a Programme Director at ARIA. He was most recently a Research Fellow in technical AI safety at Oxford. [&hellip;]