I’ve figured out how to trigger the reasoning process by adding "/think" to the system prompt. Heads up: the <|channel>thought tags have an unusual pipe (|) placement, which is why many LLM fail to parse the reasoning section corr…
LM Studio is an exceptional tool for running local LLMs, but it has a specific quirk: the "Thinking" (reasoning) toggle often only appears for models downloaded directly through the LM Studio interface. If you use external GGUFs from pr…
Qwen 3.5, Gemma-4, Nemotron Cascade 2 and GLM 4.7 flash. Tested to see how performance (speed) degrades with the context increase. used llama.cpp and some nice quants better fitting for 16GB VRAM in my RTX 4080. Here is a result comparison table….
submitted by /u/Inv1si [link] [comments]
Hi everyone, I'm completely new to running AI models locally and would appreciate some guidance. Here are my specs: CPU: AMD Ryzen 9 5950X RAM: 16GB DDR4 GPU: NVIDIA RTX 4060 (8GB VRAM) I know my specs are pretty poor for running local AI, but I wa…
Finally got the Gemma 4 (E4B) model running on my Raspberry Pi 5 (8GB). Since the model requires about 9.6GB of RAM, I had to get creative with memory management. The Setup: Raspberry Pi OS. Lexar SSD (Essential for fast Swap). Memory Management:…
submitted by /u/Mike_mi [link] [comments]
I found a TQ3-quantized version of Qwen3-Coder-Next here: https://huggingface.co/edwardyoon79/Qwen3-Coder-Next-TQ3_0 According to the page, this model requires a compatible inference engine that supports TurboQuant. It also provides a command, but it d…
gemma-4-26B-A4B-it-Q4_K_M can only get around 50% acc on winogrande-debiased-eval.csv with llama-perplexity. Meanwhile qwen3.5-35B-A3B-IQ4_NL can get about 75%+ acc. However, in real-world tasks, the Gemma 4 model performs very well. Why does this disc…
I've noticed this on Qwen3.5 35B before as well, there is a noticeable speed difference between Unsloth's Q4_K_XL and Bartowski's Q4_K_M on the same model, but Gemma 4 seems particularly harsh in this regard: Bartowski gets 38 tk/s, Unsloth…