full-title:: [Understanding Encoder And Decoder LLMs](https://omnivore.app/me/understanding-encoder-and-decoder-ll-ms-18b4225c289)
site:: [Ahead of AI](https://magazine.sebastianraschka.com/p/understanding-encoder-and-decoder)
author:: Sebastian Raschka, PhD
labels:: [[PROJECTS/MOSAICO]] 
date-saved:: [[18-10-2023]]
date-published:: [[17-06-2023]]
date-archived:: [[18-10-2023]]
is-archived:: 10
source:: [[Omnivore]]
state:: [[archived]]

	- collapsed:: true
	  * ### Highlights
	  collapsed:: true
		- > BERT (**B**idirectional **E**ncoder **R**epresentations from **T**ransformers) is an encoder-only architecture based on the Transformer's encoder module. [⤴️](https://omnivore.app/me/understanding-encoder-and-decoder-ll-ms-18b4225c289#b769d9a7-30b4-414f-9fe7-b492112c397b)
		- > The main idea behind masked language modeling is to mask (or replace) random word tokens in the input sequence and then train the model to predict the original masked tokens based on the surrounding context. [⤴️](https://omnivore.app/me/understanding-encoder-and-decoder-ll-ms-18b4225c289#7406f31a-4621-49f5-a442-0369e9b192be)
		- > The \[CLS\] token is a placeholder token for the model, prompting the model to return a _True_ or _False_ label indicating whether the sentences are in the correct order or not. [⤴️](https://omnivore.app/me/understanding-encoder-and-decoder-ll-ms-18b4225c289#8ca8535f-2ba4-49b6-bd33-8b6f31fdea85)
		- > decoder-only models like GPT exploded in popularity thanks to breakthrough in text generation via GPT-3, ChatGPT, and GPT-4. [⤴️](https://omnivore.app/me/understanding-encoder-and-decoder-ll-ms-18b4225c289#c152dfc9-62a9-4551-a4f5-0cee6f36d6b8)