Part 9/15:
In language models, context window size is a critical parameter, dictating how much input the model can "remember" at once.
GPT-3 features a 2,048-token limit.
ChatGPT is rumored to support approximately 8,000 tokens, roughly equivalent to 8 pages of text.
Future iterations might push this boundary to 60,000 tokens or more, enabling longer, more coherent text generation—such as writing a full-length book from a single prompt.
Current constraints:
The short-term memory of models is a bottleneck, analogous to a toddler’s working memory—capable of handling only a handful of instructions simultaneously.
Enhancing context size directly correlates with an AI's ability to handle complex tasks and long-term dependencies.