How to use min-p?

[OliverLeeXZ]

Thanks for your great work! Could you please tell us how to use min-p to infer? Previous works modified transformers/generation/utils.py, but I have no idea about how to use min-p in LLM.

[menhguin]

Good point! I will improve the documentation. We had to remove those due to anonymous peer review. But it's relatively straightforward

1. Please check if Min P is already available. Currently it is already available on Transformers, VLLM and many others. You mentioned Transformers, which already merged Min P a few months back: Generate: add min_p sampling huggingface/transformers#30639

To use it, you only need to add a gen/output hyperparameter like you would with top_p or temperature (I think, it's been a while).

# Generate text using Top-p sampling
output = model.generate(
    input_ids,
    do_sample=True,        # Enable sampling
    top_p=0.9,             # Cumulative probability threshold
    min_p=0.1
    max_length=50          # Maximum length of generated text
)

2. If you'd like to mess around with your own custom samplers, you can check out the changes in the above PR to see what you need to get it working. The actual implementation which we copied into the paper is under logits_process.py, but you'd need to change a lot of other files which reference logits_process.py: https://github.com/huggingface/transformers/blob/80f2b1610fa17ebf582897c8611180cac38652f0/src/transformers/generation/logits_process.py#L4 . What you need to change entirely depends on how the inference engine is set up. For VLLM, changes were much simpler: Support Min P Sampler vllm-project/vllm#1642
3. Do note that our evaluations were conducted on VLLM. This is important because VLLM does temperature scaling before truncation sampling, whereas Hugging Face does the reverse order. This means you will see different behaviour depending on what you use. I recommend VLLM due to its faster speed and because diversity from temperature is higher if you do it before truncation (for creative writing for example). You will probably get better benchmark scores from Hugging Face, but I feel it sort of defeats the purpose of using temperature sampling at all.

Let me know if you have other questions!

[OliverLeeXZ]

Thanks for your repaid response! I have another question. Can min-p be extended to other decoding strategies? Such as greedy, beam search, or nucleus? Or did you compare min-p with these methods?

[menhguin]

Theoretically yes. But We've only tested it in combination with temperature scaling, and the benefits to combining it with another truncation sampler like Top P are unclear.