{lib}[GCCcore/12.3.0] PyTorch-bundle v2.1.2, SoX v14.4.2, libmad v0.15.1b, ... w/ CUDA 12.1.1 #19987

Micket · 2024-02-28T00:10:19Z

(created using eb --new-pr)

I have combined what I found from several conflicting PRs here to make what I think we want to have as the PyTorch-bundle.

I made some choices here:

Stole some from other PRs to combine into this one
Using SentencePiece with the existing gperftools instead of introducing a duplicate
Verified torchvision and torchtext was using supported versions for PyTorch 2.1
Added torchaudio
kept torchdata despite it being abandoned (since it was only just abandoned relatively recently)

(I don't really mind how this gets merged, if someone wants to update their PR instead, feel free to close this or just use it as a reference).

…oX-14.4.2-GCCcore-12.3.0.eb, libmad-0.15.1b-GCCcore-12.3.0.eb, SentencePiece-0.2.0-GCC-12.3.0.eb

jfgrimm · 2024-02-28T09:21:19Z

Test report by @jfgrimm
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
gpu22.viking2.yor.alces.network - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA H100 PCIe, 535.86.10, Python 3.6.8
See https://gist.github.com/jfgrimm/0c24f489ee602ccdae65b2561aa04572 for a full test report.

jfgrimm · 2024-02-28T09:29:14Z

@boegelbot: please test @ jsc-zen3

boegelbot · 2024-02-28T09:30:07Z

@jfgrimm: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=19987 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_19987 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3691

Test results coming soon (I hope)...

- notification for comment with ID 1968569676 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-28T09:41:16Z

Test report by @boegelbot
FAILED
Build succeeded for 4 out of 5 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/09cf6f9d7dc502b335ffba08e041ee14 for a full test report.

edit: succeeded apart from a sanity check that requires a GPU

Micket · 2024-02-28T15:28:16Z

RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

huh, jsc-zen3 should have that right?

jfgrimm · 2024-02-28T17:29:18Z

@boegelbot: please test @ jsc-zen3
EB_ARGS=--include-easyblocks-from-pr=3236

boegelbot · 2024-02-28T17:30:08Z

@jfgrimm: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=19987 EB_ARGS="--include-easyblocks-from-pr=3236" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_19987 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3696

Test results coming soon (I hope)...

- notification for comment with ID 1969491560 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-28T17:40:50Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3236
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/edc71ffb6e1d18e07ccea070c5c1191a for a full test report.

SebastianAchilles · 2024-02-28T18:27:01Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 545.23.08, Python 3.9.18
See https://gist.github.com/SebastianAchilles/3d75091724ac38810a1d1d29c8ff4744 for a full test report.

jfgrimm · 2024-02-29T11:01:30Z

@boegelbot: please test @ generoso
EB_ARGS=--include-easyblocks-from-pr=3236

boegelbot · 2024-02-29T11:05:09Z

@jfgrimm: Request for testing this PR well received on login1

PR test command 'EB_PR=19987 EB_ARGS="--include-easyblocks-from-pr=3236" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_19987 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12991

Test results coming soon (I hope)...

- notification for comment with ID 1970894376 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-02-29T11:29:09Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3236
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/63054e19e7841ff7438dbc464e51aea8 for a full test report.

jfgrimm · 2024-02-29T14:21:39Z

@Micket I've made a PR to your fork that makes sure the easybuild provided sox is used for torchaudio: Micket#19

…dle212 patch torchaudio to use external sox; enable CUDA support

jfgrimm · 2024-02-29T14:36:15Z

Test report by @jfgrimm
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
gpu21.viking2.yor.alces.network - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA H100 PCIe, 535.86.10, Python 3.6.8
See https://gist.github.com/jfgrimm/de8c44168aed72593159b5fbc6822460 for a full test report.

jfgrimm · 2024-02-29T14:46:29Z

Test report by @jfgrimm
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3236
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
himem01.viking2.yor.alces.network - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7643 48-Core Processor, Python 3.6.8
See https://gist.github.com/jfgrimm/9887ac847fe7ff2acdcfe08bad3d9822 for a full test report.

easybuild/easyconfigs/p/PyTorch-bundle/PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb

akesandgren · 2024-04-11T05:43:36Z

easybuild/easyconfigs/p/PyTorch-bundle/PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb

+            '" and not kaldi_io_test"'  # requires kaldi_io
+            '" and not test_dup_hw_acel"'  # requires special render device permissions
+            '" and not test_h264_cuvid"'  # requires special render device permissions
+            '" and not test_hecv_cuvid"'  # requires special render device permissions


Typo, should be hevc

VRehnberg · 2024-04-11T06:21:21Z

Test report by @VRehnberg
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-04 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 550.54.14, Python 3.6.8
See https://gist.github.com/VRehnberg/fac06f4e6ebcdcd4edb03ed44fcd5372 for a full test report.

VRehnberg · 2024-04-11T06:21:54Z

Test report by @VRehnberg
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-04 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 550.54.14, Python 3.6.8
See https://gist.github.com/VRehnberg/d42b620745b746b59e5a7f49c403dda7 for a full test report.

VRehnberg · 2024-04-11T06:21:57Z

Test report by @VRehnberg
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-05 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 550.54.14, Python 3.6.8
See https://gist.github.com/VRehnberg/6b7113318ae567cef3a5ff97d39fdeeb for a full test report.

VRehnberg · 2024-04-11T07:58:16Z

Test report by @VRehnberg
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-04 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 550.54.14, Python 3.6.8
See https://gist.github.com/VRehnberg/e1a80da7762acaa5fa8467499586c470 for a full test report.

akesandgren · 2024-04-11T13:22:35Z

Most annnoying...
On my Ubuntu 20.04, broadwell, V100 node tests/ignite/contrib/handlers/test_tqdm_logger.py passes
but on my Ubuntu 22.04 zen3, A100 node it fails.

I think I've rebuilt everything recently enough on both systems...

I get this:


capsys = <_pytest.capture.CaptureFixture object at 0x7ff16d613790>

    def test_pbar(capsys):
        n_epochs = 2
        loader = [1, 2]
        engine = Engine(update_fn)
    
        pbar = ProgressBar()
        pbar.attach(engine, ["a"])
    
        engine.run(loader, max_epochs=n_epochs)
    
        captured = capsys.readouterr()
        err = captured.err.split("\r")
        err = list(map(lambda x: x.strip(), err))
        err = list(filter(None, err))
        if get_tqdm_version() < Version("4.49.0"):
            expected = "Epoch [2/2]: [1/2]  50%|     , a=1 [00:00<00:00]"
        else:
            expected = "Epoch [2/2]: [1/2]  50%|     , a=1 [00:00<?]"
>       assert err[-1] == expected
E       AssertionError: assert 'Epoch [2/2]:...a=1 [00:00<?]' == 'Epoch [2/2]:...a=1 [00:00<?]'
E         - Epoch [2/2]: [1/2]  50%|     , a=1 [00:00<?]
E         + Epoch [2/2]: [1/2]  50%|                                                   , a=1 [00:00<?]

tests/ignite/contrib/handlers/test_tqdm_logger.py:61: AssertionError

and the same in a bunch of other tqdm tests

VRehnberg · 2024-04-11T14:34:15Z

So the width padding whitespace of the progress bar from tqdm is different than expected. As far as failures go it's about as benign as I can imagine. But, I agree that it is annoying.

I suppose that we can drop that with all the other loggers, probably is an issue with tqdm rather than ignite. But, it was nice to have some test that not all integration with third party loggers had failed.

Micket · 2024-04-11T16:38:22Z

I don't care anymore. I'm just ignoring til this is merged.

…ove some more tests

akesandgren · 2024-04-12T07:27:09Z

Gah.... inject-checksums messed things up...
refixing...

…ernet access

akesandgren · 2024-04-12T09:51:07Z

Test report by @akesandgren
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
b-cn1603.hpc2n.umu.se - Linux Ubuntu 22.04, x86_64, AMD EPYC 7313 16-Core Processor, 1 x NVIDIA NVIDIA A100 80GB PCIe, 545.29.06, Python 3.10.12
See https://gist.github.com/akesandgren/6e4c65774270b117665773d9e099462b for a full test report.

Micket · 2024-04-12T09:56:28Z

You also removed the --ignore=tests/ignite/contrib/handlers/test_tqdm_logger.py???

akesandgren · 2024-04-12T10:15:05Z

Test report by @akesandgren
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
b-cn1502.hpc2n.umu.se - Linux Ubuntu 20.04, x86_64, Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz, 2 x NVIDIA Tesla V100-PCIE-16GB, 545.29.06, Python 3.8.10
See https://gist.github.com/akesandgren/aefea610bba329c00498ceef17ae8e9d for a full test report.

Running out of CUDAmemory.... not much to do then I guess.

akesandgren · 2024-04-12T10:43:01Z

You also removed the --ignore=tests/ignite/contrib/handlers/test_tqdm_logger.py???

Yeah, I forgot to reload before adding my stuff...
readded it now.

akesandgren · 2024-04-12T12:30:12Z

Test report by @akesandgren
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
b-cn1603.hpc2n.umu.se - Linux Ubuntu 22.04, x86_64, AMD EPYC 7313 16-Core Processor, 1 x NVIDIA NVIDIA A100 80GB PCIe, 545.29.06, Python 3.10.12
See https://gist.github.com/akesandgren/6c4753a4d668255c5a6ae5c2e53c6a48 for a full test report.

akesandgren

LGTM

…asyconfigs into 20240228001016_new_pr_PyTorch-bundle212

akesandgren · 2024-04-12T13:39:19Z

Going in, thanks @Micket!

adding easyconfigs: PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb, S…

9f5f0b7

…oX-14.4.2-GCCcore-12.3.0.eb, libmad-0.15.1b-GCCcore-12.3.0.eb, SentencePiece-0.2.0-GCC-12.3.0.eb

Micket added the update label Feb 28, 2024

Micket added 2 commits February 28, 2024 01:25

Add torchvision patch

569fac8

Fix patch name

438f8ed

easybuilders deleted a comment from boegelbot Feb 28, 2024

jfgrimm added this to the 4.x milestone Feb 28, 2024

patch torchaudio to use external sox; enable CUDA support

576c988

Merge pull request #19 from jfgrimm/20240228001016_new_pr_PyTorch-bun…

b2087ca

…dle212 patch torchaudio to use external sox; enable CUDA support

jfgrimm reviewed Feb 29, 2024

View reviewed changes

easybuild/easyconfigs/p/PyTorch-bundle/PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb Outdated Show resolved Hide resolved

fix long line

9d1901e

akesandgren requested changes Apr 11, 2024

View reviewed changes

Fix typo

aad5993

VRehnberg mentioned this pull request Apr 11, 2024

{ai}[foss/2023a] tinygrad v0.8.0 w/ CUDA 12.1.1 #20339

Draft

Drop another test

e58d908

PyTorch-bundle: add patch for ignite and re-enable launcher test, rem…

1397cb8

…ove some more tests

akesandgren added 2 commits April 12, 2024 09:37

PyTorch-bundle: restore layout after inject-checksums messing with it

9e737f4

PyTorch-bundle: remove one more test from torchtext that requires int…

77f29fb

…ernet access

PyTorch-bundle: restore removal of test_tqdm_logger

969579d

Shorten comment to fix long line

959eef6

akesandgren approved these changes Apr 12, 2024

View reviewed changes

akesandgren modified the milestones: 4.x, release after 4.9.1 Apr 12, 2024

Merge branch 'develop' of https://github.com/easybuilders/easybuild-e…

e36468a

…asyconfigs into 20240228001016_new_pr_PyTorch-bundle212

akesandgren merged commit c04dd25 into easybuilders:develop Apr 12, 2024
9 checks passed

Micket deleted the 20240228001016_new_pr_PyTorch-bundle212 branch April 12, 2024 14:29

casparvl mentioned this pull request May 7, 2024

disable test that fetches from the web for torchtext extension in PyTorch-bundle v2.1.2 #20484

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lib}[GCCcore/12.3.0] PyTorch-bundle v2.1.2, SoX v14.4.2, libmad v0.15.1b, ... w/ CUDA 12.1.1 #19987

{lib}[GCCcore/12.3.0] PyTorch-bundle v2.1.2, SoX v14.4.2, libmad v0.15.1b, ... w/ CUDA 12.1.1 #19987

Micket commented Feb 28, 2024 •

edited

Loading

jfgrimm commented Feb 28, 2024

jfgrimm commented Feb 28, 2024

boegelbot commented Feb 28, 2024

boegelbot commented Feb 28, 2024 •

edited by jfgrimm

Loading

Micket commented Feb 28, 2024

jfgrimm commented Feb 28, 2024

boegelbot commented Feb 28, 2024

boegelbot commented Feb 28, 2024

SebastianAchilles commented Feb 28, 2024

jfgrimm commented Feb 29, 2024

boegelbot commented Feb 29, 2024

boegelbot commented Feb 29, 2024

jfgrimm commented Feb 29, 2024

jfgrimm commented Feb 29, 2024

jfgrimm commented Feb 29, 2024

akesandgren Apr 11, 2024

VRehnberg commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

akesandgren commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

Micket commented Apr 11, 2024

akesandgren commented Apr 12, 2024

akesandgren commented Apr 12, 2024

Micket commented Apr 12, 2024

akesandgren commented Apr 12, 2024 •

edited

Loading

akesandgren commented Apr 12, 2024

akesandgren commented Apr 12, 2024

akesandgren left a comment

akesandgren commented Apr 12, 2024

{lib}[GCCcore/12.3.0] PyTorch-bundle v2.1.2, SoX v14.4.2, libmad v0.15.1b, ... w/ CUDA 12.1.1 #19987

{lib}[GCCcore/12.3.0] PyTorch-bundle v2.1.2, SoX v14.4.2, libmad v0.15.1b, ... w/ CUDA 12.1.1 #19987

Conversation

Micket commented Feb 28, 2024 • edited Loading

jfgrimm commented Feb 28, 2024

jfgrimm commented Feb 28, 2024

boegelbot commented Feb 28, 2024

boegelbot commented Feb 28, 2024 • edited by jfgrimm Loading

Micket commented Feb 28, 2024

jfgrimm commented Feb 28, 2024

boegelbot commented Feb 28, 2024

boegelbot commented Feb 28, 2024

SebastianAchilles commented Feb 28, 2024

jfgrimm commented Feb 29, 2024

boegelbot commented Feb 29, 2024

boegelbot commented Feb 29, 2024

jfgrimm commented Feb 29, 2024

jfgrimm commented Feb 29, 2024

jfgrimm commented Feb 29, 2024

akesandgren Apr 11, 2024

Choose a reason for hiding this comment

VRehnberg commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

akesandgren commented Apr 11, 2024

VRehnberg commented Apr 11, 2024

Micket commented Apr 11, 2024

akesandgren commented Apr 12, 2024

akesandgren commented Apr 12, 2024

Micket commented Apr 12, 2024

akesandgren commented Apr 12, 2024 • edited Loading

akesandgren commented Apr 12, 2024

akesandgren commented Apr 12, 2024

akesandgren left a comment

Choose a reason for hiding this comment

akesandgren commented Apr 12, 2024

Micket commented Feb 28, 2024 •

edited

Loading

boegelbot commented Feb 28, 2024 •

edited by jfgrimm

Loading

akesandgren commented Apr 12, 2024 •

edited

Loading