Add min_pool_size, Add default value of should_shuffle #70

reyoung · 2016-09-13T14:40:52Z

min_pool_size would be infinite by default.
- add unittest for min_pool_size
Fix bug in can_over_batch_size
- add unittest for can_over_batch_size
Add DEFINE_PROVIDER_EX
Add default value of should_shuffle
- When training, the default value of should_shuffle is True.
- When testing, the default value of should_shuffle is False.
- User a set a provider should_shuffle or not by pass it to @provider
- should_shuffle can handle a list of value, not just boolean
Add input order mapping by using name
- Add unittest
Add check to check input format.
- Default is close for speed reason.
- User could stop train when check error, or continue train without
  this train sample.
use deque instead of vector in generators pool, make erase
generator faster.
Add chinese/english documentation
Make should shuffle = false in unittest
Add python files to depends.

reyoung · 2016-09-13T14:41:52Z

paddle/gserver/dataproviders/DataProvider.cpp

+DataProvider* DataProvider::create(const DataConfig& config,
+                                   const ModelConfig& modelConfig,
+                                   bool useGpu) {
+  return registrar_.createByType(config.type(), config, modelConfig, useGpu);


Add ModelConfig in DataProvider::create to get input layer order

emailweixu · 2016-09-13T17:56:23Z

paddle/gserver/dataproviders/DataProvider.h

+    return dp;\
+  });\
+})
+


add more comment

emailweixu · 2016-09-13T22:50:10Z

Also please update the data provider documentation

reyoung · 2016-09-14T11:50:08Z

@emailweixu Update codes. Add chinese docs. Engligh document will be added asap.

emailweixu · 2016-09-14T23:18:20Z

Need to fix test

reyoung · 2016-09-18T06:01:17Z

@emailweixu The unittest error before, is because we didn't disable shuffle when unittest and this patch set the min_pool_size to unlimited. It makes data shuffle correctly, and influences the unittest before.

Also add english documentation.

reyoung · 2016-09-18T07:27:21Z

doc/ui/data_provider/pydataprovider2.rst

-* cache is a data cache strategy, see `cache`_.
-* Init_hook function is invoked once the data provider is initialized,
-  see `init_hook`_.
+.. autofunction:: paddle.trainer.PyDataProvider2.provider


Here, we use paddle.trainer.PyDataProvider2.provider's comments as documentation.

* min_pool_size would be infinite by default. * add unittest for min_pool_size * Fix bug in can_over_batch_size * add unittest for can_over_batch_size * Add DEFINE_PROVIDER_EX * Add default value of should_shuffle * When training, the default value of should_shuffle is True. * When testing, the default value of should_shuffle is False. * User a set a provider should_shuffle or not by pass it to `@provider` * should_shuffle can handle a list of value, not just boolean * Add input order mapping by using name * Add unittest * Add check to check input format. * Default is close for speed reason. * User could stop train when check error, or continue train without this train sample. * use deque instead of vector in generators pool, make erase generator faster. * Add chinese/english documentation * Make should shuffle = false in unittest * Add python files to depends.

* refactor lower function * refine LoweredFunc code gen * add const support

* add paddleIArray * use final inherit, rm data_

* update paddlenlp usage * update paddlelsim * update readme Co-authored-by: ceci3 <592712189@qq.com>

[DOC] Add C/GO API and R demo

Made several changes: - an -> a - Realse->Release - Traning ->Training - Unify application with noun.

This reverts commit 01a4c47.

add group_pattern_util.ShardableAxesProvider

reyoung reviewed Sep 13, 2016
View reviewed changes

reyoung force-pushed the fix_can_over_batch_size branch 4 times, most recently from d20681d to 78170c3 Compare September 13, 2016 15:15

emailweixu reviewed Sep 13, 2016
View reviewed changes

paddle/gserver/dataproviders/DataProvider.h

return dp;\

});\

})

Copy link

Collaborator

emailweixu Sep 13, 2016 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add more comment

reyoung force-pushed the fix_can_over_batch_size branch 2 times, most recently from aab1c00 to 0210938 Compare September 14, 2016 11:48

reyoung force-pushed the fix_can_over_batch_size branch from 0210938 to 079cc3f Compare September 14, 2016 11:51

reyoung force-pushed the fix_can_over_batch_size branch from fee49a9 to 5b62bb1 Compare September 18, 2016 05:43

reyoung force-pushed the fix_can_over_batch_size branch from 5b62bb1 to e32e474 Compare September 18, 2016 07:09

reyoung commented Sep 18, 2016

View reviewed changes

reyoung force-pushed the fix_can_over_batch_size branch from e32e474 to 18101a9 Compare September 18, 2016 07:58

reyoung force-pushed the fix_can_over_batch_size branch from 18101a9 to 981d733 Compare September 19, 2016 09:31

emailweixu merged commit 90b9cba into PaddlePaddle:master Sep 19, 2016

reyoung deleted the fix_can_over_batch_size branch September 22, 2016 04:48

DemoMoon mentioned this pull request Mar 24, 2021

oneDNN 如何能提升DeepSpeech的语音处理性能 #31838

Closed

thisjiang pushed a commit to thisjiang/Paddle that referenced this pull request Oct 28, 2021

Refactor Load/Store IR Nodes (PaddlePaddle#70)

3fdd2b6

* refactor lower function * refine LoweredFunc code gen * add const support

gglin001 added a commit to graphcore/Paddle-fork that referenced this pull request Dec 8, 2021

add paddleIArray (PaddlePaddle#70)

1ae448d

* add paddleIArray * use final inherit, rm data_

wangxicoding pushed a commit to wangxicoding/Paddle that referenced this pull request Dec 9, 2021

Update ofa bert (PaddlePaddle#70)

5949b25

* update paddlenlp usage * update paddlelsim * update readme Co-authored-by: ceci3 <592712189@qq.com>

zhoutianzi666 pushed a commit to zhoutianzi666/Paddle that referenced this pull request May 23, 2022

Merge pull request PaddlePaddle#70 from qili93/doc_add_c_api

2d87a6f

[DOC] Add C/GO API and R demo

danleifeng added a commit to danleifeng/Paddle that referenced this pull request Jul 22, 2022

edit memory_sparse_table save thread_pool_size (PaddlePaddle#70)

933d14f

AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this pull request Sep 19, 2022

Fix typographical error (PaddlePaddle#70)

f02d52a

Made several changes: - an -> a - Realse->Release - Traning ->Training - Unify application with noun.

zmxdream pushed a commit to zmxdream/Paddle that referenced this pull request Feb 10, 2023

[add slots shuffle] (PaddlePaddle#70)

f023bee

qizhaoaoe pushed a commit to qizhaoaoe/Paddle that referenced this pull request Mar 3, 2023

bug fix (PaddlePaddle#70)

01a4c47

qizhaoaoe pushed a commit to qizhaoaoe/Paddle that referenced this pull request Mar 3, 2023

Revert "bug fix (PaddlePaddle#70)" (PaddlePaddle#71)

ce760c9

This reverts commit 01a4c47.

lizexu123 pushed a commit to lizexu123/Paddle that referenced this pull request Feb 23, 2024

fix distill demo (PaddlePaddle#70)

30fe5c4

hanhaowen-mt pushed a commit to hanhaowen-mt/Paddle that referenced this pull request Feb 29, 2024

fix(op): conv transpose backward input grad error (PaddlePaddle#70)

70dd030

Fridge003 pushed a commit to Fridge003/Paddle that referenced this pull request Mar 15, 2024

Merge pull request PaddlePaddle#70 from tc20042008/xk-cinn-trivalop-fuse

08b45af

add group_pattern_util.ShardableAxesProvider

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add min_pool_size, Add default value of should_shuffle #70

Add min_pool_size, Add default value of should_shuffle #70

reyoung commented Sep 13, 2016 •

edited

Loading

reyoung Sep 13, 2016

emailweixu Sep 13, 2016 •

edited

Loading

emailweixu commented Sep 13, 2016

reyoung commented Sep 14, 2016

emailweixu commented Sep 14, 2016

reyoung commented Sep 18, 2016 •

edited

Loading

reyoung Sep 18, 2016

Add min_pool_size, Add default value of should_shuffle #70

Add min_pool_size, Add default value of should_shuffle #70

Conversation

reyoung commented Sep 13, 2016 • edited Loading

reyoung Sep 13, 2016

Choose a reason for hiding this comment

emailweixu Sep 13, 2016 • edited Loading

Choose a reason for hiding this comment

emailweixu commented Sep 13, 2016

reyoung commented Sep 14, 2016

emailweixu commented Sep 14, 2016

reyoung commented Sep 18, 2016 • edited Loading

reyoung Sep 18, 2016

Choose a reason for hiding this comment

reyoung commented Sep 13, 2016 •

edited

Loading

emailweixu Sep 13, 2016 •

edited

Loading

reyoung commented Sep 18, 2016 •

edited

Loading