✨Hi there! ✨

This is the repository for the Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs! Here, you will find the following:

• Min P Code Implementation: The latest implementation of Min P sampling from the Huggingface Transformers library as of June 2024.
• WandB logs of GPQA and GSM8K evals: Logs comparing results between Min P and Top P for both GPQA and GSM8K evaluations, at different truncation sampling parameters and temperature scaling values.

• Colab notebook to replicate GPQA and GSM8K evals: If you’d like to replicate the GPQA and GSM8K COT evaluations in the paper, you may do so at [PUBLIC]_Min_P_Evals_Replication_for_GPQA_and_GSM8K_COT.ipynb
• Logs for AlpacaEval Creative Writing: For logs of the independently run AlpacaEval Creative Writing evals for Min P, see https://github.com/IlyaGusev/quest (not affiliated with authors). The original repo did not included replication instructions, so for that, see: https://github.com/menhguin/quest
• Interactive Demo: For the independently created interactive demo, check out https://artefact2.github.io/llm-sampling/index.xhtml (not affiliated with authors)
• Accuracy-diversity tradeoff charts: For entropy experiment scripts, see: https://github.com/BlackHC/minp_scripts

1. Please check if Min P is already available. Currently it is already available on Transformers, VLLM and many others. Transformers has already merged Min P a few months back: huggingface/transformers#30639

To use it, you only need to add a gen/output hyperparameter like you would with top_p or temperature (I think).

# Generate text using Top-p sampling
output = model.generate(
    input_ids,
    do_sample=True,        # Enable sampling
    top_p=0.9,             # Cumulative probability threshold
    min_p=0.1,
    max_length=50          # Maximum length of generated text
)

2. To integrate your own custom samplers, you can check out the changes in the above PR to see what you need to get it working. The actual implementation which we copied into the paper is under logits_process.py, but you'd need to change a lot of other files which reference logits_process.py: https://github.com/huggingface/transformers/blob/80f2b1610fa17ebf582897c8611180cac38652f0/src/transformers/generation/logits_process.py#L4 . What you need to change entirely depends on how the inference engine is set up. For VLLM, changes were much simpler: vllm-project/vllm#1642
3. Do note that our evaluations were conducted on VLLM (for Min P, Top P, Top K and no truncation sampling/pure sampling). This is important because VLLM does temperature scaling before truncation sampling, whereas Hugging Face does the reverse order. This means you will see different behaviour depending on what you use. I recommend VLLM due to its faster speed and because diversity from temperature is higher if you do it before truncation (for creative writing for example). You will probably get better benchmark scores from Hugging Face, but I feel it sort of defeats the purpose of using temperature sampling at all. Another note: Eta sampling and Epsilon sampling were conducted via Hugging Face. Ideally we would've done it on VLLM, but Hugging Face was the only option available. It didn't work very well and seemed quite difficult to set up on VLLM, so we de-prioritised it.

Let me (@menhguin) know if you have other questions!