Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cherry Pick] Fix distributed batch sampler #8691

Merged
merged 1 commit into from
Jul 1, 2024

Conversation

JunnYu
Copy link
Member

@JunnYu JunnYu commented Jul 1, 2024

PR types

Bug fixes

PR changes

APIs

Description

num_samples 向下去整, 防止prefrech预取时候超过数据集最大长度...

# 该情况下,计算存在问题,当向上去整的时候2848会超过数据集的最大长度2844
len(self.dataset) = 2844
self.nranks = 8
int( len(self.dataset)* 1.0 / self.nranks) * self.nranks = 2840
int(ceil(len(self.dataset)* 1.0 / self.nranks)) * self.nranks = 2848
# 该情况下计算不会有问题,因为整除了
len(self.dataset) = 2844
self.nranks = 4
int( len(self.dataset)* 1.0 / self.nranks) * self.nranks = 2844
int(ceil(len(self.dataset)* 1.0 / self.nranks)) * self.nranks = 2844

Copy link

paddle-bot bot commented Jul 1, 2024

Thanks for your contribution!

@JunnYu JunnYu merged commit b88c709 into release/2.8 Jul 1, 2024
5 of 7 checks passed
DesmonDay pushed a commit to DesmonDay/PaddleNLP that referenced this pull request Sep 5, 2024
DesmonDay added a commit that referenced this pull request Sep 5, 2024
* quick fix from pretrained. (#8487)

* quick fix os.path.split (#8508)

* Cp/fix (#8569)

* [Safetensors] Fix fast safe open slice. (#8512)
* [FIX DDP] fix ddp (#8549)

* [BUG] Fix build train valid test datasets (#8823)

* Update causal_dataset.py

* Add twenty redundant data in post pretrain (#8777)

* 给dataset再添加20条数据,防止blend dataset出现错误

* num_samples向下去整,防止数据集的溢出 (#8691)

* update release_grads (#8834)

* update release_grads (#8834)

* [Trainer] Fix release_grads (#9085)

* fix pp release_grads

* add dataloader_drop_last to evaldataloader (#8773)

* bugfix

* Fix eval hang (#9052)

* fix pipeline eval

* fix eval dataloader_num_workers

---------

Co-authored-by: Zhong Hui <zhonghui.net@gmail.com>
Co-authored-by: yujun <50394665+JunnYu@users.noreply.github.com>
Co-authored-by: gongel <ainlp88@qq.com>
@ZHUI ZHUI deleted the fix_DistributedBatchSampler_ branch September 26, 2024 08:13
Copy link

codecov bot commented Sep 29, 2024

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 55.20%. Comparing base (0f428bb) to head (1926e7b).
Report is 55 commits behind head on release/2.8.

Files with missing lines Patch % Lines
paddlenlp/utils/batch_sampler.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           release/2.8    #8691      +/-   ##
===============================================
+ Coverage        55.14%   55.20%   +0.06%     
===============================================
  Files              608      611       +3     
  Lines            94580    95055     +475     
===============================================
+ Hits             52158    52478     +320     
- Misses           42422    42577     +155     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants