Part 1/9:

The Evolution of AI: Scaling and System Two Thinking

The incredible advancements in artificial intelligence (AI) over the last five years can best be encapsulated by the concept of scale. While there have been notable algorithmic improvements, today's leading models fundamentally rely on the same Transformer architecture that emerged in 2017. The primary difference lies in the amount of data and computational power utilized for training these models. For instance, in 2019, training a model like GPT-2 cost approximately $5,000. Today, however, training models can run into the hundreds of millions of dollars. This escalation raises legitimate concerns about whether the AI field might plateau or "hit a wall".