Sort:  

If beneficial mutations are exceedingly rare on average, you need a population of several million animals before you'll see one.
To build on that with the next one, which will also presumably require millions of those animals, the entire population needs to be replaced by the offspring of the first one.

So if there are say 26 improvements, call them A-Z, to get us from ape to person, that one lucky ape with improvement A, has to have children with A, and those children have to have children; until there are millions of descendants before one of them is born with B.
Now B needs to expand from one ape to millions through direct inheritance until there are enough Bs to get lucky and mutate a C.
If each step requires the entire population be replaced by a single ancestor, that takes hundreds of generations and puts a cap on how quickly the process can happen.
Even if something like radiation increased the rate of mutation so that 1 in 10,000 apes got a beneficial one, the time required to get to 10,000 offspring isn't that much shorter than the time required to get to 10 million.
This is the main argument used by the gradualist camp when attacking the punctuated equilibrium model.