Transformer architecture
Arquitectura transformer (Attention Is All You Need)
- Date
- 2017 · confidence high
- Place
- Google Brain and Google Research, Mountain View, California, United States
- Credited to
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin (Google)
- People
- 👤 Aidan N. Gomez👤 Ashish Vaswani1985–👤 Illia Polosukhin👤 Jakob Uszkoreit👤 Llion Jones1984–👤 Łukasz Kaiser1981–👤 Niki Parmar👤 Noam Shazeer1976–
- Field
- Knowledge & symbols
What it is
Neural network based exclusively on attention mechanisms, parallelisable and scalable to trillions of parameters. It is the architecture behind every large language, image and protein model.
What it led to
- Large language models (GPT, Claude, Gemini, Llama) and generative AI
- AlphaFold and scientific AI
- The geopolitical AI race and the energy demand of data centres
Built on
- Backpropagation for training neural networks1986
- PageRank and Google1998
- AlexNet and the deep learning revolution2012
- LSTM and sequence-to-sequence models
- attention mechanism (Bahdanau, 2014)
- GPUs and TPUs
Made possible
- AlphaFold protein structure prediction2020
- Protein structure prediction by deep learning (AlphaFold 2)2020-2021
- GPT and large language models2018-2020
- ChatGPT and conversational AI assistants2022
- Diffusion models and generative imagery2020-2022
- Autonomous AI agents and reasoning models2023-2025
- GPT and large language models
- BERT