We all know modern "intelligent" Quantization that uses an imatrix to make a Q4_K_XL model to feel like Q6_K. But here is what i notice: While this works well on most English tasks, the effect can be reversed on other languages or niche tasks…
This is getting ridiculous. DDTreee paper: https://liranringel.github.io/ddtree/DDTree.pdf submitted by /u/Thrumpwart [link] [comments]
As part of our ongoing translation quality research at Alconost, we put six models through subtitle translation into six language pairs. At first glance the numbers told a clean story. Then human QA added a chapter. Models: TranslateGemma-12b ge…
This is a KLD eval across community GGUF quants of Qwen3.5-9B, comparing mean KLD to the BF16 baseline. The goal is to give people a data-driven basis for picking a file rather than just grabbing whatever is available. KLD (KL Divergence): "…
I have totally missed the boat on speculative decoding. Today when generating some code again for the frontend i found myself staring down at some quite monotonic javascript code. I decided to give a go at the speculative decoding settings of llama.cpp…
https://huggingface.co/JANGQ-AI/MiniMax-M2.7-JANGTQ Used TQ as quantization method where it matters. Finally mac users under 64 gb – esp base m5 users can get a real cloud SOTA-like level LLM running from home. Second image is from a user on olde…
I've started looking into what speculative decoding is/how it works in the past 30 minutes. I realize this is not a lot of time to try to understand something and hope you will forgive me. I have a cognitive block about this question now that I fee…
submitted by /u/FatheredPuma81 [link] [comments]
https://github.com/jundot/omlx/commit/28fab9fc28f0c0013ffb307f3b21d30658ae1a72 submitted by /u/butterfly_labs [link] [comments]