[train] u2++-lite training support #2202
Conversation
|
What is the functionality of 'apply_non_blank_embedding'? Are there any reference materials available for learning? |
whiteshirt0429
force-pushed
the
diwu-u2++-lite
branch
2 times, most recently
from
a2f0674
to
0a5ee16
Compare
it is a new feature |
whiteshirt0429
force-pushed
the
diwu-u2++-lite
branch
from
db2be51
to
84f9221
Compare
|
u2++ lite is used for reducing rescoring latency，runtime and latency result will be check in soon |
wenet/transformer/asr_model.py
Outdated
| @@ -133,6 +143,34 @@ def _forward_ctc(self, encoder_out: torch.Tensor, | |||
| loss_ctc = self.ctc(encoder_out, encoder_out_lens, text, text_lengths) | |||
| return loss_ctc | |||
There was a problem hiding this comment.
看起来 _forward_ctc 不再需要了，asr_model.py 中没有其他地方会调用
There was a problem hiding this comment.
k2 和paraformer里面都有重定义的 _forward_ctc, 比如k2里面是要在里面算lfmmi。所以我建议，修改asr_model._forward_ctc,让他返回loss和logits，而不是弃用 asr_model._forward_ctc, 同时修改k2和paraformer对应的重定义函数
| @@ -63,7 +67,8 @@ def forward(self, hs_pad: torch.Tensor, hlens: torch.Tensor, | |||
| loss = self.ctc_loss(ys_hat, ys_pad, hlens, ys_lens) | |||
| # Batch-size average | |||
| loss = loss / ys_hat.size(1) | |||
| return loss | |||
| ys_hat = ys_hat.transpose(0, 1) | |||
| return loss, ys_hat | |||
There was a problem hiding this comment.
这里ctc的返回值已经变成俩了，所以k2和paraformer里的ctc调用的返回值也得改一下，不然会报错
wenet/utils/executor.py
Outdated
| if info_dict["model_conf"]["apply_non_blank_embedding"]: | ||
| logging.warn( | ||
| 'Had better load a well trained model if' | ||
| 'apply_non_blank_embedding is true !!!' | ||
| ) |
There was a problem hiding this comment.
这个可以挪到 train_utils.py::check_modify_and_save_config函数吗？原因是：
- 放在
executor::train里，每个epcoh都要打印 check_modify_and_save_config这个函数就是专门用来检查配置的，符合这里的log含义
| for module_name in args.freeze_modules: | ||
| if module_name in name: | ||
| param.requires_grad = False | ||
| logging.debug("{} module is freezed".format(name)) |
There was a problem hiding this comment.
不freeze 多卡训练会有问题，对齐也会发生变化
There was a problem hiding this comment.
不freeze 多卡训练会有问题，对齐也会发生变化
get，多卡训练报啥错
wenet/transformer/asr_model.py
Outdated
| maxlen = encoder_out.size(1) | ||
| top1_index = torch.argmax(ctc_probs, dim=2) | ||
| indices = [] | ||
| for j in range(topk_prob.size(0)): |
whiteshirt0429
force-pushed
the
diwu-u2++-lite
branch
3 times, most recently
from
78915c3
to
f82a4c9
Compare
[train] add instructions for use
whiteshirt0429
force-pushed
the
diwu-u2++-lite
branch
from
f82a4c9
to
bfaa8a3
Compare
|
设置 pre-commit 了吗？ |
|
这是google用在RNNT中的frame reduce / blank skip 用在AED架构中吗？ |
当时做的时候没有了解这些，刚才搜了一下 k2 团队也有类似的工作。我理解思想上都差不多，都是为了降低计算量，减小延迟。这里主要是为了减少推理时的延迟。 |
No description provided.