Understanding Deep Learning - Prof. SIMON PRINCE [STAFF FAVOURITE]

Watch behind the scenes, get early access and join private Discord by supporting us on Patreon:
https://patreon.com/mlst
https://discord.gg/aNPkGUQtc5
https://twitter.com/MLStreetTalk

In this comprehensive exploration of the field of deep learning with Professor Simon Prince who has just authored an entire text book on Deep Learning, we investigate the technical underpinnings that contribute to the field's unexpected success and confront the enduring conundrums that still perplex AI researchers.

Key points discussed include the surprising efficiency of deep learning models, where high-dimensional loss functions are optimized in ways which defy traditional statistical expectations. Professor Prince provides an exposition on the choice of activation functions, architecture design considerations, and overparameterization. We scrutinize the generalization capabilities of neural networks, addressing the seeming paradox of well-performing overparameterized models. Professor Prince challenges popular misconceptions, shedding light on the manifold hypothesis and the role of data geometry in informing the training process. Professor Prince speaks about how layers within neural networks collaborate, recursively reconfiguring instance representations that contribute to both the stability of learning and the emergence of hierarchical feature representations. In addition to the primary discussion on technical elements and learning dynamics, the conversation briefly diverts to audit the implications of AI advancements with ethical concerns.

Pod version (with no music or sound effects): https://podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/Understanding-Deep-Learning---Prof--SIMON-PRINCE-STAFF-FAVOURITE-e2dmd3i

Follow Prof. Prince:
https://twitter.com/SimonPrinceAI
https://www.linkedin.com/in/simon-prince-615bb9165/

Get the book now!
https://mitpress.mit.edu/9780262048644/understanding-deep-learning/
https://udlbook.github.io/udlbook/

Panel: Dr. Tim Scarfe -
https://www.linkedin.com/in/ecsquizor/
https://twitter.com/ecsquendor

TOC:
[00:00:00] Introduction
[00:11:03] General Book Discussion
[00:15:30] The Neural Metaphor
[00:17:56] Back to Book Discussion
[00:18:33] Emergence and the Mind
[00:29:10] Computation in Transformers
[00:31:12] Studio Interview with Prof. Simon Prince
[00:31:46] Why Deep Neural Networks Work: Spline Theory
[00:40:29] Overparameterization in Deep Learning
[00:43:42] Inductive Priors and the Manifold Hypothesis
[00:49:31] Universal Function Approximation and Deep Networks
[00:59:25] Training vs Inference: Model Bias
[01:03:43] Model Generalization Challenges
[01:11:47] Purple Segment: Unknown Topic
[01:12:45] Visualizations in Deep Learning
[01:18:03] Deep Learning Theories Overview
[01:24:29] Tricks in Neural Networks
[01:30:37] Critiques of ChatGPT
[01:42:45] Ethical Considerations in AI

References:

#61: Prof. YANN LECUN: Interpolation, Extrapolation and Linearisation (w/ Dr. Randall Balestriero)
https://youtube.com/watch?v=86ib0sfdFtw

Scaling down Deep Learning [Sam Greydanus]
https://arxiv.org/abs/2011.14439

"Broken Code" a book about Facebook's internal engineering and algorithmic governance [Jeff Horwitz]
https://www.penguinrandomhouse.com/books/712678/broken-code-by-jeff-horwitz/

Literature on neural tangent kernels as a lens into the training dynamics of neural networks.
https://en.wikipedia.org/wiki/Neural_tangent_kernel

Zhang, C. et al. "Understanding deep learning requires rethinking generalization." ICLR, 2017.
https://arxiv.org/abs/1611.03530

Computer Vision: Models, Learning, and Inference, by Simon J.D. Prince
https://www.amazon.co.uk/Computer-Vision-Models-Learning-Inference/dp/1107011795

Deep Learning Book, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
https://www.deeplearningbook.org/

Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network
https://arxiv.org/abs/2210.00881

Computer Vision: Algorithms and Applications, 2nd ed. [Szeliski]
https://szeliski.org/Book/

A Spline Theory of Deep Networks [Randall Balestriero]
https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf

DEEP NEURAL NETWORKS AS GAUSSIAN PROCESSES [Jaehoon Lee]
https://arxiv.org/abs/1711.00165

Do Transformer Modifications Transfer Across Implementations and Applications [Narang]
https://arxiv.org/abs/2102.11972

ConvNets Match Vision Transformers at Scale [Smith]
https://arxiv.org/abs/2310.16764

Dr Travis LaCroix (Wrote Ethics chapter with Simon)
https://travislacroix.github.io/