Performance of PAN in Total-text

[TianBaoGe]

Hi, thanks for your great job.
I run the script and get the H-mean of PAN in Total-text. I find my H-mean is about 0.73, while it is 0.835 in the original paper, without pre-training.

Are the experiment settings different between them?

[gaotongxiao]

ping @jeffreykuang

[TianBaoGe]

ping @jeffreykuang

Is there a problem with the official generated code of Total-Text? I used the same settings as in the paper, have you conducted relevant experiments?

[gaotongxiao]

By "official generated code", do you mean the conversion script? https://github.com/open-mmlab/mmocr/blob/main/tools/data/textdet/totaltext_converter.py

I think it's a quite straightforward script so it's less likely a data conversion problem. The reason is unclear to us as we haven't conducted experiments on TotalText. It would be great if you could share more information, including

1. Have you run other experiments on PANet? Were the results consistent with the paper?
2. The config file that you used to train PANet on;
3. The log file of training.

[jeffreykuang]

I didn't conduct comparative experiments on total text. I did experiments on icdar15 or icdar17. I suggest you to check the pretrained model used in the paper.

[TianBaoGe]

By "official generated code", do you mean the conversion script? https://github.com/open-mmlab/mmocr/blob/main/tools/data/textdet/totaltext_converter.py

I think it's a quite straightforward script so it's less likely a data conversion problem. The reason is unclear to us as we haven't conducted experiments on TotalText. It would be great if you could share more information, including

1. Have you run other experiments on PANet? Were the results consistent with the paper?
2. The config file that you used to train PANet on;
3. The log file of training.

I followed https://github.com/open-mmlab/mmocr/blob/main/tools/data/textdet/totaltext_converter.py to generate the Total-Text dataset

(1) Here is my config file I used to train PANet:
panet_r18_fpem_ffm_600e_totaltext.txt

(2) Here is my log file of training:
20211020_165607.log

The highest Hmean score in my experiments is 0.784, while is 83.5 in the original paper.

[TianBaoGe]

I didn't conduct comparative experiments on total text. I did experiments on icdar15 or icdar17. I suggest you to check the pretrained model used in the paper.

I just used resnet18 without syntxt to train the model, but the final result was terrible.
You can check my experiment settings in the reply above.
Looking forward to your reply. Thanks a lot!

[jeffreykuang]

As for our experimental results, you can check https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/README.md.

For text detection, we found the following aspects are very important:

• pretrained model
• data augmentation
• if ignore hard text bounding boxes.

[TianBaoGe]

As for our experimental results, you can check https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/README.md.

For text detection, we found the following aspects are very important:

• pretrained model
• data augmentation
• if ignore hard text bounding boxes.

In fact, in the official implementations of PAN, the config files of total-text and CTW1500 are exactly the same, so I just directly used your CTW config file in https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/README.md for training, but the result is really bad.