Singularity runs, but gives semi-random results #42
Comments
|
If it helps, there are some outputs that are perfectly fine and others that are borked: In BallStick output In NODDI output |
|
Hi Celstark, Thanks for reporting in such detail. From what I understand from all your experiments is that MDT works fine as long as you run it on bare-metal, but as soon as you run it within Singularity, results default to 0.5? I am also confused on why this would happen. Possible reasons I can think of are:
The fact that it returns 0.5 is not a coincidence, this is the default starting value and if no computations are performed, this is typically what you would get as a result. I am not very experienced with Singularity and as such I am afraid I can't be much of an help. Perhaps you could run the shell command "clinfo" within the singularity container and list the devices found in the container? Best, Robbert |
|
Reviving this old issue as I've returned to hit the problem ;) Unfortunately, neither of these ideas seem to be it. I've got trivially-tweaked Singularity images based off of the github code provided recipes that build both Intel and NVIDIA based Singularity images. I'm processing the same exact data with the same model in both. For example: I've run this a few times, even using different Singularity images built slightly differently and the results are always consistent. Runs inside Singularity on the GPU - check. Now the Intel build: We run and we get output, but it's not quite right and differs from run to run as if some matrix actually isn't getting initialized. Yes, that's an AMD processor, so here's a 144-core Intel one: The attached screenshot shows 2 runs in the Singularity NVIDIA on the top (identical) and 3 runs (2 on the "titan" AMD machine and 1 on the "wario" Intel machine). The bands in one are at 0.5 exactly.
|
|
Hi celstrark, I am sorry to hear that MDT gave you so much problems. The problem is that OpenCL is so ill-supported by different vendors. At this point I don't know how to help you. Family and work are taking 120% of my time already. Are you interested in developing MDT further? Best, Robbert |
|
I understand the constraints and am under similar ones here. On our end, MDT works wonderfully so long as we run it in a more typical setup. So, for this step in the processing, we will just run things manually. I poked around for a bit to see if I could spot something to no avail, but I'll keep this in the back of my mind. Thanks for all you've done with the package! Craig |
|
Hi Craig, Many thanks for sharing the Singularity recipe. I have implemented your recipe on our HPC and came across the same issue, with ODI values fixed at 0.5. It sounds like you have a working solution to run MDT on your workstation though, would you be happy to share more about your setup so we can continue to implement MDT? That will be much appreciated! Best Wishes, |
|
On the workstation, it's running via CUDA. Here's the definition file I used: As you can see, not much to it -- really just your Dockerfile.nvidia converted over. I never did get a working setup to run on the CPU in a container. |
|
Thanks very much for sharing this, Craig. I have implemented this on our HPC and the script was able to detect the GPUs. However, the script is still getting stuck at the “We will use the optimiser Powell with default settings” (#58). I think it might be related to my OpenCL too. When I ran I will ask the system administrators to update these drivers and see if that helps. Best Wishes, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've used MDT for a number of years with a lot of success, running on workstations with NVIDIA GPUs. Some time back, I'd posted about issues with OpenCL Intel processing not being multi-threaded. While the problem got solved in that we used many threads, the output is never close to what the GPU numbers are. Here's a screenshot showing a GPU-derived image on the left and a CPU / Singularity image on the right:
A bit of decoding -- the arrow is pointing to a section showing the values under the cursor across multiple runs. There are two NVIDIA runs on different machines that give identical results of 0.5777... (good). There's also a run on one of these machines setting the OpenCL device to be device1, which was 'CPU - pthread-Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (Portable Computing Language)'. This comes very, very close (0.578) and I'll attribute to different floating point units. Now, the GPU run took <1 minute and the CPU one took 934 minutes on a 20-core machine, which was a bit insane, but at least the numbers lined up.
Moving on, though, to attempts to bundle this in Singularity. I've used the supplied script, my own script, and many variants on each. In the end, I get images that are either a constant 0.5 everywhere within the mask or images like the one on the right that show something of the brain, but whose values are way, way off -- 0.0256 in the example here. It's almost as if the values are being cast into the wrong format. We run quickly as the threading works well here, but this is clearly quite wrong. I'd love to be able to fix this, but I've thrown at it what I can think of and hit the wall.
The text was updated successfully, but these errors were encountered: