Super lazy.
Am I doing something wrong? The model is basically refusing to do any work.
I pass a largish set of instructions via -sysf SYS_PROMPT.md to llama.cpp/build/bin/llama-cli, pass some text to work on at the interactive prompt, and
it is give me a few thoughts, which are paraphrases of the system prompt's instructions (a tiny fraction) and then stops.
I seem to have the opposite problem, using another variant of llama.cpp/build/bin/llama-server (my modified version with caching and multiple processing slots), which I was using very successfully with Qwen3.8-27B-GGUF, and it works for a while with somewhat poor adherence to the sys prompt, but it never stops, it just continues generating forever.
I am not certain what it is generating,, I have to look.
Can someone shed some light on 1) how to make it WORK, instead of stopping after 3-4 lines of thoughts (never closes this) and 2) how to reliably stop it?
With Qwen I was relying on the last } of the JSON object, but it does not seem to work with this model.
Sorry that the model is not performing for your use case. Can you share the full stacktrace if possible. I think by default, llama cpp sets a smaller context window size which might be an issue. Here are a few recommendations from me
Try fixing the token allocations in llama-cli
llama-cli
-m TwIL-LM3-Pro-Q4_K_M.gguf
-c 16384
-n 4096
--temp 1.0
--top-p 0.95
--conversation
-sysf SYS_PROMPT.mdOne more common issue since it uses a granite backend is that having the correct stop sequence which is different from qwen.
Stop strings: ["<|end_of_text|>", "<|start_of_role|>", "<|end_of_role|>"] , these are for the granite based modelsOr else try the granite chat format
<|start_of_role|>system<|end_of_role|>
{Your system instructions here}<|end_of_text|>
<|start_of_role|>user<|end_of_role|>
{User prompt here}<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>The last option would be just setting thinking off and then trying to create JSON if needed. I personally wont recommend this option but hope that the first three solutions work for you.
I can provide a better solution if I get the stacktrace or example to reproduce the error.
--conversation does not seem to exist in the regular llama.cpp
How can I suppress thinking? It spends multiple times more tokens on this, than the actual output.
Thanks.
For this one thing, I would recommend is try using the 4-bit version of the model. It has been specially RL trained to work similar to the 16-bit GGUF while also having much smaller reasoning traces. And there is a non thinking mode also for the model which can be used.
For this one thing, I would recommend is try using the 4-bit version of the model. It has been specially RL trained to work similar to the 16-bit GGUF while also having much smaller reasoning traces. And there is a non thinking mode also for the model which can be used.
What's the correct way to activate the non-thinking mode?
I am hesitant to downgrade from 16 to 4, as quality is important in my use case.
If you are using Ollama, Set the enable_thinking parameter to false in your model options, or end the generation prompt with <|im_start|>assistant\n to bypass the chain-of-thought block.