Why MultiGPU dp seems slower?

[ricardorei]

❓ Questions and Help

Having 2 gpus with DP seems to be slowers than using just 1. Is it normal?

My intuition is that if you are using 2 GPUs and the batch is being splitted into 2 batches, this should be faster. But when I tested the same code using 1 vs >1 my epoch time increased

Code

Minimalist Implementation of a BERT Sentence Classifier

What have you tried?

I also tried to run ddp but my code seems to break with a TypeError: cannot serialize '_io.TextIOWrapper' object error. I searched online but I couldn't find the reason...

What's your environment?

• OS: Linux

[williamFalcon]

you should double your batch size.
dp still has overhead in communication, so it won't be linear scaling.

also try ddp