About low accuracy on valid set, overfitted on train set

[thirteenzzn]

Hello! Thank you for your fine-tuning code firstly. However, I met some problems in performance of the model.

I implemented the code and finetune the model "CLIP_L14" on datasets: Oxford Pets, Caltech101 and ImageNet with the same fine-tuning config in the paper except the batch size (Due to the limitation of the device, I set the batch size as 32). But the model performance bad on the validation set with accuracies around 1-5%, but on the train set, the accuracies are around 90%. It seems a typical overfit problem. I changed the learning rate, regulation config, epochs and other related config but failed to solve the problems.

So, I wonder that do you meet the same problem on similar datasets or if there are some methods to solve this problem.