Episode 3 - Vision Transformers

Today, we're discussing the highly influential paper "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," published by Dosovitskiy and colleagues in 2021. It’s a paper that’s amassed nearly 45,000 citations, which speaks volumes about its impact.

At the time, the transformers architecture was already the standard for natural language processing. What this paper did was extend that breakthrough to computer vision. The authors introduced a pure transformer model for images outperforming over traditional Convolutional Neural Networks (CNNs). The model was more efficient in training, and—crucially—brought scaling laws to computer vision. In simple terms, the larger the model, the better the performance.