The following are my notes on this video by Andrej Karpathy that provide a high level overview on what LLMs are and how they’re trained. It is by no means meant to supplement the content of the video and recommend giving it a watch yourself.
Info Last Updated: 5-21-2025
This is an evergrowing log of ideas, thoughts, and small excerpts that have stood out to me over the years. It used to live as a Note on my phone, but there’s some value in imparting it back out to the world. Many of the these revolve around the theme of doing your most authentic and best work and others align with the values I hold dearly. I’ve often come back to this note in search of some words of encouragement or when I’ve felt a little lost.
A Markov chain is a model that describes a set of transitions which are determined by some probability distribution that satisfy the Markov property.
Latent means hidden. Latent space can also be known as the embedding space
Latent Space refers to an abstract multi-dimensional space containing feature values that we cannot interpret directly, but which encodes a meaningful internal representation of externally observed events.
The motivation to learn a latent space (set of hidden topics/ internal representations) over the observed data (set of events) is that large differences in observed space/events could be due to small variations in latent space (for the same topic). Hence, learning a latent space would help the model make better sense of observed data than from observed data itself, which is a very large space to learn from.