config NUM_THREADS for caching data or make .cache file compatible

[iumyx2612]

Search before asking

• I have searched the YOLOv5 issues and found no similar feature requests.

Description

So when I work with new dataset I have to cache data. But parameter NUM_THREAD is set too high cause program to crash
NUM_THREADS = min(8, os.cpu_count())
my os.cpu_count() is actually 8, it use up all of my cpus and causing crash.
I usually train yolov5 using Google Colab and interact with it there. But I want to debug something, so I downloaded my .cache file from colab to my local computer. I thought I won't go through a caching stage, but hash from the .cache file I downloaded from colab cache['hash'] appeared to have different hash from get_hash(self.label_files + self.img_files) because of the the path on google colab is different from my local computer

Use case

Low-end computers

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@iumyx2612 thanks for the feedback! What value of NUM_THREADS works on your system then? I've never seen any issues here even in resource constrained environments like Colab free.

[iumyx2612]

@iumyx2612 thanks for the feedback! What value of NUM_THREADS works on your system then? I've never seen any issues here even in resource constrained environments like Colab free.

I have to use NUM_THREADS = 4 so it doesn't crash.
For more info, my CPU is Intel Core i7-7700HQ @ 2.8Ghz (8CPUs). And I was using Pycharm + Google Chrome while caching

[glenn-jocher]

@iumyx2612 what happens if you do NUM_THREADS=6 or NUM_THREADS=7?

[iumyx2612]

@iumyx2612 what happens if you do NUM_THREADS=6 or NUM_THREADS=7?

7 is okay. I tested while still opening Pycharm and tons of Google Chrome tabs

[glenn-jocher]

@iumyx2612 ok so it seems like a good first step is simply to update the code so that we leave at least 1 cpu free.

NUM_THREADS = min(8, max(1, os.cpu_count() - 1))

[glenn-jocher]

@iumyx2612 good news 😃! Your original issue may now be fixed ✅ in PR #5706. To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[iumyx2612]

@iumyx2612 good news 😃! Your original issue may now be fixed ✅ in PR #5706. To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

The problem still persists, before I called LoadImagesAndLabels seperately on a script. Today, I ran train.py and CPU reached 100%, causing crash. I had to manually set NUM_THREADS to 4 and closed all other programs