Potential Bug in Calculating Dataloader num_workers in Multi-node environment

[sitecao]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

From the line below, it looks like the num_workers will be a function of the cluster size and decrease as cluster size increases. (in my use case os.cpu_count() // WORLD_SIZE is always taken to be value of nw)

yolov5/utils/datasets.py

Line 113 in e7bf382

nw = min([os.cpu_count() // WORLD_SIZE, batch_size if batch_size > 1 else 0, workers]) # number of workers

For example, a p4d instance has 96 vCPUs and 8 GPUs. If we were to use a cluster of size 2, WORLD_SIZE will be 16 and we will have nw = 96 // 16 = 6. And if we have 8 nodes, WORLD_SIZE will be 64 and nw = 96 // 64 = 1.

From the reply from the author on below (unrelated) issue, it seems like this model has not been officially tested on a multi-node environment and I think this may be a potential bug. Can you please confirm on this? I think the intention of the author is os.cpu_count() // torch.cuda.device_count()?
#1445 (comment)

Additional

No response

[github-actions]

👋 Hello @sitecao, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@sitecao yes that's correct. We've used single P4d instances for training but not scaled to multi-node training, so it's possible there are multi-node bugs latent in the code.

The intention of os.cpu_count() // WORLD_SIZE is to use 96/8 = 12 workers on a P4d instance per RANK, i.e. to use all the available resources on a node whether it's Multi-GPU training or just 8 single-GPU trainings.

It seems like we should divide WORLD_SIZE by a node count. Is that possible?

If you can confirm a fix and submit a PR (making sure it doesn't alter the single-node behavior) I'll get that fast tracked. Thanks!

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!