-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Degraded performance after updating from Cuda 8.0 to Cuda 9.0, and cuDNN v5 to cuDNN v7? #429
Comments
Testing Cuda 8.0 (
This makes me wonder if CUDA Toolkit 8.0 GA2 (Feb 2017) added something new that Neural-Style doesn't require, and thus we can disable/remove it in order to lower memory use. |
Another setup for comparison:
Cuda 7.5, and cuDNN v4 Setup memory usage:
Running
It looks like the memory usage significantly increased in the Cuda 8.0 (Feb 2017) update, while previous versions of Cuda were around 23.5% more efficient, unless the issue scales with the image size value. If the issue scales with the image size value, that would significantly lower the largest possible image size. |
An interesting side effect of changing the Cuda, Cudnn, Torch7 versions, and maybe even the Ubuntu versions), is that the seed value effects seems to change. So if you use |
Another way to slightly lower memory usage, seems to be possible by stripping layers from a VGG model: |
have you tried instance normalization with your setups? I encounter problems here .... |
@flaushi Are you refering to instance normalization from fast-neural-style? |
Yes! |
I'm confused @ProGamerGov , the GPU memory usage definitely scales with image output size -- were these CUDA benchmarks taken with the same image output size? One of my ec2 snapshots has a corrupted cuda install, somehow, so I'm deciding which cuda to go with -- would rather use the most memory-efficient. |
@ajhool I believe that I was using |
That's really interesting. I sensed a significant speed increase on the newer versions of Cuda and cudnn, but "sensed" is the operative word because I didn't do a controlled benchmark. I also sensed the memory usage increase. The latest Cuda supports the Volta GPU, too, so I'm excited to see if there are significant rendering gains to be made, there. What I'm actually finding (I think) is that for smaller renders the main bottleneck occurs in some process that luajit is executing on the CPU. So, when I run multiple renders on the same GPU (~60% GPU memory usage), I am seeing significant (2x-3x) rendering slowdowns, and I am also seeing all 4 of the CPU cores get locked up at 100% due to a luajit process. I believe that I'd posted an issue about this in the past but might be misremembering. But concurrent rendering seems to be constrained by the CPU before the GPU, frustratingly. |
After using
th neural_style.lua -gpu 0 -backend cudnn
to compare an older install of Neural-Style, to a newer one, I noticed that the performance seems to have gotten worse. Is this something that's fixable? Or are the new versions of Cuda and cuDNN just not as efficient as the previous versions?cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
,cudnn-8.0-linux-x64-v5.0-ga.tgz
Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-38-generic x86_64)
Compared to:
Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1038-aws x86_64)
cudnn-9.0-linux-x64-v7.tgz
,libcudnn7_7.0.3.11-1+cuda9.0_amd64.deb
Before the network loads the model, more memory is also used:
This is a pretty large change in terms of resource usage, and it's definitely not for the better.
The text was updated successfully, but these errors were encountered: