Dictionary AI Basics
Tokens
What it means
The chunks of text a model reads and writes - usually word-pieces. 'Unbelievable' might be split into 'un', 'believ', 'able'.
An everyday example
Tokens are the LEGO bricks of language. The model builds every sentence brick by brick.