What do you use those small model for? And how do you perceive the gap with leading closed source LLMs?
I've seen that a lot of you use heavily quantised models with 30-something billions, sometimes even MoE, and it got me wondering: what are the real gains? (excluding privacy and the fact that it probably feels just better to actually own the infras…