support other ext tasks except aso task and fix sentiment analysis based on SKEP #4357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
Codecov Report
@@ Coverage Diff @@
## develop #4357 +/- ##
========================================
Coverage 39.65% 39.65%
========================================
Files 433 433
Lines 60936 60936
========================================
Hits 24167 24167
Misses 36769 36769 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
wawltor
reviewed
Jan 5, 2023
| @@ -146,6 +147,11 @@ def convert_example_to_feature_cls(example, tokenizer, label2id, max_seq_len=512 | |||
| return encoded_inputs | |||
|
|
|||
|
|
|||
| def remove_blanks(example): | |||
| example["text"] = re.sub(" +", "", example["text"]) | |||
There was a problem hiding this comment.
去除原文中的空格，当前的tokenizer 在encode时会忽略空格，导致input_ids长度!=原始文本的长度，会有匹配上的一些问题。
| def remove_blanks(example): | ||
| example["text"] = re.sub(" +", "", example["text"]) | ||
| return example | ||
|
|
| - ``negative_ratio``: 最大负例比例，该参数只对抽取类型任务有效，适当构造负例可提升模型效果。负例数量和实际的标签数量有关，最大负例数量 = negative_ratio * 正例数量。该参数只对训练集有效，默认为5。为了保证评估指标的准确性，验证集和测试集默认构造全负例。 | ||
| - ``is_shuffle``: 是否对数据集进行随机打散，默认为True。 | ||
| - ``seed``: 随机种子，默认为1000. | ||
| 其中，参数``negative_ratio``表示对于一个样本，为每个子任务（属性级的观点抽取，属性级的情感分类）最多生成``negative_ratio``个负样本。如果额外提供了属性同义词标或隐性观点抽取词表，将结合两者信息生成更多的负样本，以增强属性聚合和隐性观点抽取能力。 |
There was a problem hiding this comment.
这里的参数介绍，是否要保留 label_studio_file 、 task_type 相关的
| "sentiment_prompt_prefix": "情感倾向", | ||
| "separator": "##", | ||
| "not_mentioned_option": "未提及", | ||
| "options": "正向,负向,未提及", |
There was a problem hiding this comment.
这里的options被写死了，如果用户想自定义 "正向,负向,中性" 类似的options，这块的定制能力是不是没有了？
There was a problem hiding this comment.
如沟通，当前只对外暴露的options，保持情感分类定制能力
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Function optimization & Bug fixes
PR changes
APIs & Docs
Description