AssertionError: can only test a child process #3

Activity

1. cpphoo commented on Oct 7, 2022

   

   cpphoo

   on Oct 7, 2022

   Owner

   More actions

   Hi @fikry102, could you give me the command you used for training? It is a little difficult to tell what is going on without knowing which model you are training.

   For what it is worth, I have not encountered this before and I can only give you a few pieces of advice on how to debug:

   1. Is the model checkpointed at the last iterations? If yes, then you should be fine. If not, proceed to the next item.
   2. Could the problem be related to this issue (https://discuss.pytorch.org/t/error-while-multiprocessing-in-dataloader/46845)? To check, I would first set the number of training epochs to 1 to see whether you can reproduce the error. If yes, then I would try playing around with the number of workers or removing tqdm for training to further diagnose the problem.

   Hope this helps!
2. fikry102 commented on Oct 7, 2022

   

   fikry102

   on Oct 7, 2022

   Last edited by fikry102

   Author

   More actions

   Thanks for your detailed answer.
   The error occured during training student representation using MiniImageNet as the source and ChestX as the target.
   In spite of the error, the whole training process is a success.
   My preliminary speculation is that the error may occur during pseudolabeling the dataset.
   I will try to conduct more detailed debugging to solve my problem. Changing num_workers may be helpful.

   BTW, I think the logic of "import configs" in 【miniImageNet_few_shot.py, EuroSAT_few_shot.py and so on】can be further improved. This is beacuse there is no configs.py in the folder dataset. ( in the folder student_STARTUP, teacher_miniImageNet). A possible solution for this chaos could be (just after a little bit of thought, maybe not so good):
   1.replace import configs with configs=None in 【miniImageNet_few_shot.py, EuroSAT_few_shot.py and so on】
   2.after import from datasets import miniImageNet_few_shot ... , add import configs and miniImageNet_few_shot .configs=configs in【student_STARTUP/STARTUP.py, teacher_miniImageNet/train.py and so on】