Default use one thread in fluid #7208

reyoung · 2018-01-04T09:37:44Z

No description provided.

tonyyang-svail

Excellent!

luotao1 · 2018-01-04T10:54:47Z

python/paddle/v2/fluid/__init__.py

+            'speed will not be optimized if you use data parallel. It will '
+            'fail if this PaddlePaddle binary is compiled with OpenBlas since'
+            ' OpenBlas does not support multi-threads.'.format(num_threads),
+            file=sys.stderr)


这段注释没太看明白。openblas的时候可以用OPENBLAS_NUM_THREADS来设置，不知道和这个PR有没有关系。

这个PR非常简单，就是要把OMP_NUM_THREADS默认设置成1。因为确实在数据并行下，这个环境变量设置成其他的值并不合适。

当cpu核数大于trainer_count时，如果设成1，会导致cpu核数没用满。
OMP_NUM_THREADS = cpu核数/trainer_count

Paddle/python/paddle/v2/__init__.py

Lines 112 to 115 in 7508d52

threads = num_processors / trainer_count

threads = '1' if threads < 1 else str(threads)

set_env("OMP_NUM_THREADS", threads)

set_env("MKL_NUM_THREADS", threads)

qingqing01 · 2018-01-04T10:54:57Z

Could you add an issue to describe the problem?

tensor-tang · 2018-01-05T07:06:31Z

Two concern I should point out :

OMP_NUM_THREADS will also impact MKL, not only OpenBLAS mentioned in OpenBLAS doesn't support multi thread application #7234.
Force setting it to 1 will limit MKLML and MKLDNN multi-processing, which means use only 1 core to compute. I suggest it could be calculated by settings like threads_pre_trainer=available_threads/trainer_count.

tonyyang-svail · 2018-01-05T08:40:57Z

I see. Change OPENBLAS_NUM_THREADS is a better approach.

Default use one thread in fluid

b585c93

reyoung requested review from qingqing01 and tonyyang-svail January 4, 2018 09:37

tonyyang-svail approved these changes Jan 4, 2018

View reviewed changes

luotao1 reviewed Jan 4, 2018

View reviewed changes

reyoung merged commit 2b259bf into PaddlePaddle:develop Jan 4, 2018

tonyyang-svail mentioned this pull request Jan 5, 2018

OpenBLAS doesn't support multi thread application #7234

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default use one thread in fluid #7208

Default use one thread in fluid #7208

reyoung commented Jan 4, 2018

tonyyang-svail left a comment

luotao1 Jan 4, 2018

reyoung Jan 4, 2018

luotao1 Jan 5, 2018

qingqing01 commented Jan 4, 2018

tensor-tang commented Jan 5, 2018 •

edited

Loading

tonyyang-svail commented Jan 5, 2018

	threads = num_processors / trainer_count
	threads = '1' if threads < 1 else str(threads)
	set_env("OMP_NUM_THREADS", threads)
	set_env("MKL_NUM_THREADS", threads)

Default use one thread in fluid #7208

Default use one thread in fluid #7208

Conversation

reyoung commented Jan 4, 2018

tonyyang-svail left a comment

Choose a reason for hiding this comment

luotao1 Jan 4, 2018

Choose a reason for hiding this comment

reyoung Jan 4, 2018

Choose a reason for hiding this comment

luotao1 Jan 5, 2018

Choose a reason for hiding this comment

qingqing01 commented Jan 4, 2018

tensor-tang commented Jan 5, 2018 • edited Loading

tonyyang-svail commented Jan 5, 2018

tensor-tang commented Jan 5, 2018 •

edited

Loading