Repository navigation
Fine-tuning with fp16 leads to NaN #5
Description
Hi,
Thank you for releasing the checkpoints publicly.
I'm trying to use the mHuBERT-147 in a downstream speech translation task. When I train with fp32 I have no issue but with fp16 I get NaN at some point during training. I'm training using Fairseq.
Do you know if there's a way to make the model compatible with fp16?
Activity
mzboito commentedon Apr 11, 2025
Hi,
Many thanks for experimenting with our model!
The third iteration of mHuBERT-147 is fp32 model, this can explain the instability you are experiencing.
However, I have successfully fine-tuned this model numerous times on Huggingface in fp16 since release. My observation is that it works on both precisions for mainstream languages, but can go quickly to NaN when working on more challenging low-resource settings with fp16.
This could be due the fairseq checkpoint carrying optimization information, different from the huggingface one... Maybe try to flush it?
issam9 commentedon Apr 11, 2025
Hi,
Thank you for the quick reply.
Not sure this is what you meant but I tried deleting last_optimizer_state and optimizer_history then using the checkpoint. I had to comment _upgrade_state_dict in checkpoint_utils.py as well but I still get gradient overflow. I tried training with --amp and the training is okay but BLEU doesn't improve because the loss is scaled very low because of gradient overflow.
Btw I tried this on both English and French.
Do you think there's something else I can try? Otherwise I'll just stick to training with fp32 which goes well :)
Have a nice weekend!
mzboito commentedon Apr 11, 2025
Hi again,
I'm sorry, but I don't know what could be happening. I would suggest you stick to fp32, or shift to HuggingFace codebase (or speechbrain).
Have a nice weekend you too!