New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

loss function mutation when training #48

Open

GJemini opened this issue Apr 10, 2023 · 0 comments

GJemini commented Apr 10, 2023

Hello,i'm very grateful for your work.but when i was training the model,there is always a problem that the loss mutated and validation images became irregular or all black，with the iou metrics dropping down to almost 0.i trained the model with a single RTX2080ti and set the batchsize as 4,while the other parameters remained original.later i reduced the max_lr to 1e-3 but still cannot work.here's the wandb report:
https://wandb.ai/jemini/cross_view_transformers_test/runs/0404_144821?workspace=user-jemini
then i found a pretrained checkpoint later.i trained with it again, and still the same problem.
https://wandb.ai/jemini/cross_view_transformers_test/runs/0408_131432?workspace=user-jemini
hope for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment