-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Trainer support simultaneously parse JSON files and cmd arguments. #7768
Conversation
Thanks for your contribution! |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #7768 +/- ##
===========================================
- Coverage 57.25% 57.22% -0.04%
===========================================
Files 585 587 +2
Lines 87977 88215 +238
===========================================
+ Hits 50372 50477 +105
- Misses 37605 37738 +133 ☔ View full report in Codecov by Sentry. |
需要加到CI脚本中去 |
单测文件路径是tests/trainer/test_argparser.py |
paddlenlp/trainer/argparser.py
Outdated
output_dir_arg = next( | ||
(arg for arg in sys.argv if arg == "--output_dir" or arg.startswith("--output_dir=")), None | ||
) | ||
if output_dir_arg is None: | ||
if "output_dir" in json_args.keys(): | ||
sys.argv.extend(["--output_dir", json_args["output_dir"]]) | ||
else: | ||
raise ValueError("The following arguments are required: --output_dir") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
为什么这里要 特判 output_dir ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
如果不特判output_dir,出现json文件里有output_dir参数,但是命令行里没有的情况,执行281行vars(self.parse_args())的时候就会报错,但是我们现在不希望让它报错,所以进行了output_dir的特判
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
你参考一下这个函数,这个函数也是一样的支持本地文件。
看看这个是怎么处理的。
这个作为通用的parser,在这里做output_dir
之类的特判是不太合理的。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…7768) * add parse_json_file_and_cmd_lines * change unit test file path * Change the way the JSON file is determined * Merge parameter parsing judgment branches and add comments. * remove the special handling of output_dir * Add remaining_args warning
PR types
New features
PR changes
LLM
Description
Add functionality to the Trainer to simultaneously parse JSON files and command line arguments.