Are you quanting your memory?
Title. Curious about how people are generally dealing with the kv cache. BF16? Q8? Q4? Turboquant or some other secret sauce? I run bf16 everything hoping that I'd get less hallucinations and because that's what the g4 and q3.6 are natively tra…