Whats your Qwen vs Gemma take on local hardware Marky? I find Gemma WAY faster on old CPU based compute where I run open models on an old laptop, so very anecdotal and not scientific but Gemma is actually usable on an old Lenovo in ollama, Qwen had a literal mental breakdown just trying to answer "what model variant are you?" as a first prompt. It went into a mental meltdown about how to respond that was so bizarre I actually saved it as a text file while it looped over its identity crisis about how to respond to me for over 15 minutes and I had to actually break it and stop it. Gemma responded with its model variant in about 11 seconds.
VERY unscientific but using it is what really matters. I wanted to use Qwen but it just won't run on moldy old hardware and Gemma does ok there.
So I am curious what your benches say about them compared to each other on accuracy?
Gemma is MOE model, so not all parameters are activating at once. Qwen 27B is very demanding as all 27 billion parameters are active for every turn. This allows it to perform considerably better than anything in it's size as 99% of models are MOE these days.
CPU inference is going to be slow and you are going to be limited to very small MOE models. Give LM Studio a try, it is considerably better than Ollama.