Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PSLIB] Add Metrics Module, Support User-defined Add Metric #38789

Merged
merged 18 commits into from
Jan 28, 2022

Conversation

WorgenZhang
Copy link
Contributor

PR types

New features

PR changes

Others

Describe

  • 新增可自定义扩展评估指标模块(metrics.h/metrics.cc),对于新增评估指标,用户只需添加新的MetricMsg类,并在AucCalculator中完成数据添加、指标计算逻辑的实现;
  • 目前评估指标模块已支持AucCalculator、MultiTaskAucCalculator、CmatchRankAucCalculator、MaskAucCalculator、CmatchRankMaskAucCalculator、WuAucCalculator
  • 新增解析uid、并根据uid进行global_shuffle功能,dataset.py新增set_shuffle_by_uid和set_uid_slot函数

@paddle-bot-old
Copy link

paddle-bot-old bot commented Jan 7, 2022

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

if (!this->merge_by_insid_) {
return fleet_ptr->LocalRandomEngine()() % this->trainer_num_;
} else {
if (this->merge_by_insid_) {
return XXH64(data.ins_id_.data(), data.ins_id_.length(), 0) %
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. 此处 globalshuffle 在二次开发时需要新增 if/else,建议按照 neg sampling #36564 方法重构;
  2. 类、函数、变量命名按统一ps规范来 https://ku.baidu-int.com/knowledge/HFVrC7hq1Q/pKzJfZczuc/UQuex_mf2m/sTOYzTS6so3HUG
  3. 文件是否可以放入统一ps的目录中,在统一ps未开发完之前也能正常调用?
  4. 此处的 BasicAucCalculator 和 paddlebox 中的 BasicAucCalculator 是什么关系?
  5. 需要新增 metrics 基类,方便后续开发

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

先进行功能合入,后续再考虑放入统一ps

@paddle-bot-old
Copy link

Sorry to inform you that 4382e31's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

Copy link
Contributor

@yaoxuefeng6 yaoxuefeng6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thunderbrook
Thunderbrook previously approved these changes Jan 27, 2022
fuyinno4
fuyinno4 previously approved these changes Jan 27, 2022
@WorgenZhang WorgenZhang reopened this Jan 27, 2022
Copy link
Contributor

@XiaoguangHu01 XiaoguangHu01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fuyinno4 fuyinno4 merged commit 2e6be88 into PaddlePaddle:develop Jan 28, 2022
WorgenZhang added a commit to WorgenZhang/Paddle that referenced this pull request Jan 28, 2022
…ddle#38789)

* [PSLIB] Add Metrics Module, Support User-defined Add Metric

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI Coverage

* modify role_maker

* update CMakeLists.txt
fuyinno4 pushed a commit that referenced this pull request Jan 28, 2022
* 12.3 first add metrics module

* add Mask/MultiTask

* add WuAUC

* [PSLIB] Update WuAUC Compute

* [PSLIB] Change WuAUC Compute Mehod

* [PSLIB] Clean WuAUC Compute

* [PSLIB] Clean Metric Module Unused Code

* mv metric instance

* [PSLIB] Add Metrics Module, Support User-defined Add Metric (#38789)

* [PSLIB] Add Metrics Module, Support User-defined Add Metric

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI Coverage

* [PSLIB] Modify According to CI Coverage

* modify role_maker

* update CMakeLists.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants