It's worth noting that only some of these techniques are configurable in modern LLM API outputs. (usually only temperature/top-p/top-k since other penalties require overhead)
Most other penalties don't require much overhead (min_p is basically free).
Most techniques are not made available by API providers because they enable alignment breaking. It's the only explanation for why we are still stuck with only top_p, top_k, and temp of 0-2.
If you want proper sampler settings to be available, your options are oobabooga, sillytavern (dependent on your backend, so vllm backend for example doesn't have top-n sigma yet), or directly running huggingface code. There might be some marginal options here too but in general, sampling innovation is firmly in the hands of open source coomers right now and not in the hands of academics.
Does min_p help with non-creative-writing tasks as well, such as maths or coding? Is there any way to improve/tune the performance for these with sampling?
Do you have a recommendation/guide on tuning the sampling parameters with what little the frontier model providers expose?
All I have ever seen is the same old very old advice of "t=0 for determinism, t=1 for creativity", with little treatment of top_k/top_p. But even the temp has become quite opaque nowadays. Is this really the softmax temperature that we're controlling, or some proxy for it? Claude 3.7 Sonnet doesn't allow me to use t != 1.0 for thinking mode, while Gemini 2.5 Pro, also a reasoner, will happily accept it. Does it apply only to the non-thinking tokens at the end?
If I get good results with t=0.0 (for e.g. coding), do I lose some "capability" by keeping it at 0.0?
