GPT and large language models
GPT y los grandes modelos de lenguaje
- Date
- 2018-2020 · confidence high
- Place
- OpenAI, San Francisco, California, United States
- Credited to
- OpenAI (Alec Radford, Ilya Sutskever, Dario Amodei and others); GPT-1 (2018), GPT-2 (2019), GPT-3 with 175 billion parameters (2020); Google's BERT (2018)
- Field
- Knowledge & symbols
What it is
Transformer models trained to predict the next word across trillions of words from the internet, acquiring general language and reasoning abilities as they grow. GPT-3 showed that scale produces few-shot learning.
What it led to
- ChatGPT and mass-market generative AI
- Scaling laws and the investment of hundreds of billions in data centres
- Debates over copyright, misinformation, employment and existential risk
Built on
- Markov chains1906
- Turing test and the idea of machine intelligence1950
- Graphics processing unit (GeForce 256)1999
- Cloud computing (AWS S3 and EC2)2006
- CUDA general-purpose GPU computing2007
- Transformer architecture2017
- word2vec (2013)
- GPU and CUDA
- Wikipedia and Common Crawl (corpus)
Made possible
- ChatGPT and conversational AI assistants2022
- Proof assistants (Coq, Lean) and AI mathematics (AlphaProof)1989-2024
- coding assistants (Copilot)
- AI agents
- multimodal models