running on multiple GPU is very slow

[201power]

I am trying to run 50-layer residual network with 4 K40m GPUs and it's very slow (same batch_size 16 as running on single GPU), take 6 hours for 1 epoch. However, If I run it on 1 GPU the speed is normal.

System: CentOS, digits v3, nvcaffe-0.14

BTW, I tried use Googlenet and it was ok on 4 GPUs.

Any suggestion or potential issue?

[lukeyeager]

Hi @201power, there are several things that could be going on:

1. cuDNN v4 is significantly faster than v3. That makes scaling across multiple GPUs less effective because a single GPU is just a lot quicker than it used to be and the GPU communication becomes more of a bottleneck. When NCCL gets integrated into Caffe, that should help some to speed up the cross-GPU communication.
2. Some networks may be more conducive to multi-GPU than others. It's possible that the architecture of your ResNet requires more communication and less computation than other networks like GoogLeNet.
3. Or this could be a bug. How much of a slowdown are we talking about? If you use one GPU how long does it take for 1 epoch?

[201power]

Tried cuDNN v3, still the same speed, 1 epoch on 1 GPU take 4 mins, on 4 GPU take 6 hours.
I used the prototxt here:
https://github.com/201power/ResNet-Generator-for-caffe/blob/master/resnet50_trainval.prototxt

It contains two new layers, batch_norm and scale, which is not used in the default googlenet

[lukeyeager]

When I try to run that network I get an error about Message type "caffe.LayerParameter" has no field named "scale_param".

NVcaffe 0.14 has the batch_norm layer type, but I guess you wrote this scale layer yourself?

[lukeyeager]

Looks like you're getting the scale layer from BVLC/caffe#3591, right?

I'm assuming you hacked up DIGITS to allow BVLC/caffe? This is definitely not a supported configuration. Still happy to help if I can, though!

[201power]

Yes, I merged NVcaffe 0.14 with caffe master branch, which has the scale layer:
https://github.com/201power/caffe

I think it's the issue with my caffe. I'll debug. Thanks!

BTW do nvcaffe plan to integrate scale layer?

[lukeyeager]

I think it's the issue with my caffe. I'll debug. Thanks!

Let me know when you resolve this issue - I hope it will be a simple configuration problem, but I'd like to know for sure that it's not a cuDNN or multi-GPU bug.

BTW do nvcaffe plan to integrate scale layer?

NVcaffe will pick up the new scale layer (among other things) when we start working on our next release (see comment on release methodology here). We don't really pull in new features in-between releases - just major bugfixes.

[201power]

Thanks. Now I am using original caffe-0.14 with googlenet, however, it because very slow as well.
Exact same job took 52 min using 4 GPU last time with exactly same software (digits/caffe/cudnn).

GPU status:

Also, I am experiencing soft lockup everytime I tried to abort a multi-GPU job in digits, do you get the same msg as well?

[lukeyeager]

Hmm, that looks like trouble. How recent is your NVIDIA driver?

What happens when you try 2 GPUs instead of 1 or 4?

[201power]

I tried 2 GPU and it's also very slow. The nvidia driver is 352.79
1 GPU always works fine.

Any other thing I can try/check?

[lukeyeager]

I can go ask some people who would knowm more about this. A few questions before I do:

1. You're using NVIDIA/caffe@caffe-0.14, right?
2. You're using cuDNN v4 (4.0.4), right?
3. Do you see the soft lockup error every time? Or does it only show up intermittently?

Can you try building without cuDNN at all and see if that makes a difference? I'm just spitballing here.

[201power]

1. Yes.
2. Yes.
3. Yes, every time I abort a multi-GPU job I saw a soft lockup (not for single GPU job).

I tried building caffe without cuDNN and it has the same issue, still very slow.
Thanks.

[thatguymike]

In the multigpu path there is sychronization between the GPUs that can cause delays, but I haven't seen those messages before. Do you have enough disk IO to feed all the GPUs? e.g. do you see messages about waiting for data in the log output?

[201power]

I do see one waiting for data in caffe log, during loading the network. However, it seems does not cause delays.

I have 377G available disk space for the user, it should be fine.

[thatguymike]

Not a question of diskspace, but IO throughput. As you add GPUs, the pressure on disk goes up. However, I haven't seen soft lockup errors before. What motherboard chipset is this, e.g. Intel X99

[thatguymike]

What version of Linux and kernel revision? Anything in system logs from NVRM?

[201power]

The motherboard chipset is supermicro x10drg-0t
Linux kernel: 3.10.0-229.el7.x86-64

[thatguymike]

K40s?

[201power]

K40m. I checked /var/log/messages and here are (part of) things I found related to NVRM:

[lukeyeager]

Can you try rolling back to your previously installed driver?

[201power]

It's a fresh computer, so it's the only version of driver we installed.
We can try to use previous version of driver though.

When running digits with 4 GPU, it always has 2 GPU with 0% utilization, is this normal?

[lukeyeager]

Don't worry about it then. So this was working for you earlier this week and you didn't uninstall or reinstall a driver or anything like that?

I'm working on trying to reproduce this bug from my end, but it may take me a while to get Caffe built on CentOS since I haven't done that before...

[201power]

Yup, there is no change on the software/hardware since the last working job.
Yes, install caffe on centos is a little time consuming, make sure you use OpenBLAS.

[lukeyeager]

And when you go back to your original build, GoogLeNet still works?

[lukeyeager]

Quick smoketest on Ubuntu passed. The scaling is bad but I'm not seeing any NVRM lines in dmesg.

OS: Ubuntu 14.04
GPUs: 2 x K40c
Driver: 352.79
Caffe: NVcaffe v0.14.2
cuDNN: 4.0.7

1 GPU: 1min 53sec
2 GPU: 2min 29sec

Time to try it on CentOS...

[201power]

The googlenet does not work anymore when I go back to the original build.

[lukeyeager]

Smoketest on CentOS also passed.

OS: CentOS 7
GPUs: 2 x K40c
Driver: 352.79
Caffe: NVcaffe v0.14.2

1 GPU: 1min, 4sec
2 GPU: 50sec

(It's faster because I chose a different dataset and it's processing fewer images)

[201power]

Is it possible that supermicro chipset causing the issue?

[lukeyeager]

Let's investigate whether you might have a bad GPU. Can you try this:

1. Do single-GPU training on each of your 4 GPUs
2. Do double-GPU training for each pair between your 4 GPUs (6 combinations total)

[lukeyeager]

Is it possible that supermicro chipset causing the issue?

Sure, that's possible too.

[201power]

I actually have 8xK40m, training on single GPU is fine:

However, training on two GPUs become very slow for all GPUs.

[lukeyeager]

And you say that multi-GPU was running at a reasonable speed just a few days ago? Can you think of anything relevant that you might have changed on the machine since then?

[lukeyeager]

FYI, this Dockerfile is enough to get Caffe built on CentOS7 (I had to do a bunch more hackery to build pycaffe and DIGITS, though).

https://gist.github.com/lukeyeager/fc0b21a62fca7b3edb24

[201power]

Nothing changed I can think of...just a reboot..
When I first installed caffe, it's running slow on multiple GPU. There is only 1 time I got it's running fast on 4 GPUs.

Thanks for the docker file.

[lukeyeager]

Let's take this offline for now. I sent you an email.

[201power]

I did not receive the email, did you sent to ?

[lukeyeager]

In the end, this was a duplicate of NVIDIA/caffe#10.

[TimZaman]

@lukeyeager Does the fact that we have to use an nvidia fork of Caffe mean that we cannot use vanilla ResNets with Caffe? Anyone got experience with that? (Resnet work fine with the latest torch from source)