Part 2/15:
One of the clearest indicators in AI development is the rapid, exponential increase in the number of parameters over the past five years. Starting from tens of millions in early models, such as GPT-2 with 1.5 billion parameters, we've seen a consistent jump towards models with hundreds of billions, and now potentially trillions.
A key graph showcases this exponential trend—rising from 100 million parameters to over a trillion. This trend aligns with milestones like Google's Switch Transformer, which hit the trillion-parameter mark, and indicates that the AI community is on a direct trajectory toward models significantly larger.
Rumors: What Might GPT-4 Look Like?
There are two main narratives circulating: