My Bookshelf

Some books and papers that I like or need to read. This is occasionally updated, though not very often. I'm hoping to build out this page more over time. Maybe I'll add some small notes as well.

You can also find me on Goodreads, which tends to be more up to date.

Papers

Papers I Like

Schlag, Imanol, Kazuki Irie, and Jürgen Schmidhuber. "Linear transformers are secretly fast weight programmers." International Conference on Machine Learning. PMLR, 2021.
Von Oswald, Johannes, et al. "Transformers learn in-context by gradient descent." International Conference on Machine Learning. PMLR, 2023.
Balestriero, Randall, and Richard Baraniuk. "Mad max: Affine spline insights into deep learning." arXiv preprint arXiv:1805.06576, 2018.
Little, W.A. "The existence of persistent states in the brain." Mathematical Biosciences, 1974.
Humayun, Ahmed Imtiaz, et al. "Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition., 2023.

Some Notes

This paper builds off of the above "Mad Max" paper by visualizing the geometry that is explored in that paper. I had an idea after reading "Mad Max" that motivated me to start implementing a similar visualization. Once I got stuck and was doing some research online to help my problems, I discovered this paper!
Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." Advances in neural information processing systems, 2019.
Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." arXiv preprint arXiv:2309.17453, 2023.

Some Notes

This paper builds off of Evan Miller's Attention is Off By One blog post which I think is a great (dare I say mathematically "morally correct") interpretation of the deficiencies of using Softmax in Attention. I ended up having some questions about Table 3. in that paper, which benchmarks Evan Miller's Softmax variant, and so I posted an issue on the Github page for their paper.

Books

Books I Like

Gödel, Escher, Bach: an Eternal Golden Braid by Douglas Hofstadter ^[1]
Remembrance of Earth's Past (Trilogy) by Liu Cixin
Calculus, Fourth Edition by Michael Spivak (ISBN-10 0914098918) ^[2]

Books I Want to Read

The Road to Reality by Roger Penrose
A New Kind of Science by Stephen Wolfram

^[1] This book has truly changed the way I think about intelligence and formal systems in a profound way. Along with the "Rememeberance of Earth's Past" trilogy, this has now become a book that I recommend to many people. I think it is highly motivating in some of what I want to focus on in my future studies.

^[2] Funnily enough, this is one of my favorite books that I learned from during my undergrad. Calculus by Spivak is a great book, even to look back on!