Llms

“Should I run a larger parameter model, or a higher quality smaller model of the same family?” TLDR; Larger parameter model [lower quantisation quality] > Smaller parameter model [higher quantisation quality] E.g: Qwen2.5 32B Q3_K_M > Qwen2.5 14B Q8_0 Caveats: Don’t go lower than Q3_K_M, or IQ2_M, especially if the model is under 30B~ parameters. This is in the context of two models of the same family and version (e.g. Qwen2.5 Coder). Longer answer: Check out the Code Chaos and Copilots slide deck. ...