Part 8/10:
Following various tests with smaller models and an exploration of networking capabilities, the cluster was finally tasked with running the most sophisticated model: the 405 billion parameter Llama 3.1405 B. This represented a monumental challenge and an opportunity to test the true limits of local hardware capabilities.
Despite successful initial loading times into RAM, each test attempted to execute this heavy model resulted in very slow processing speeds of under one token per second—far from the desired efficiency. Swapping out for Thunderbolt connections brought little improvement, which hinted at deeper limitations in the architecture being used for such complex tasks.