Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ASR] change default initializer to kaiming_uniform #1577

Merged
merged 6 commits into from
Mar 22, 2022

Conversation

Jackwaterveg
Copy link
Contributor

@Jackwaterveg Jackwaterveg commented Mar 17, 2022

conformer 有 4 种 带参数的 API。

  1. conv 类型。包含 conv1D,conv2D。 torch 使用 kaming_uniform 初始化 (weight 和 bias 都是)
  2. linear 类型 torch 使用 kaming_uniform 初始化 (weight 和 bias 都是)
  3. layernorm 和 batchnorm torch 使用 constant 初始化 (weight 1, bias 0)
  4. nn.Embedding torch 使用 normal 初始化

类型 1 和 2 没有默认对齐:

  1. conv paddle使用的是:weight 用 kaimingNormal, bias 使用 0
  2. linear paddle使用的是:weight 用 XaiverUniform , bias 使用 0

本 PR 对齐了 wenet 的 conformer 初始化。 另外,paddle 的 kaiming_uniform 和 torch的 kaiming_uniform 没有对齐,所以对API 进行了 hack,从而实现对齐。

目前使用aishell数据集,在 dist_sampler模式训练下,不使用speed_perturb, conformer 可以达到 4.88%。比原先 5.6 % 的结果提升了 0.72%的绝对 CER。和 wenet 不用 speed_perturb 得到的 4.81% 的结果只相差 0.07% (wenet 使用 speed_perturb 时结果为 4.61% )。

Fix #1575

@Jackwaterveg Jackwaterveg added this to the r0.2.0 milestone Mar 17, 2022
@mergify mergify bot added S2T asr/st Example labels Mar 17, 2022
paddlespeech/s2t/modules/initializer.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/initializer.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/initializer.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/initializer.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/initializer.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/initializer.py Outdated Show resolved Hide resolved
paddlespeech/s2t/models/u2/u2.py Show resolved Hide resolved
paddlespeech/s2t/modules/conformer_convolution.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/decoder.py Outdated Show resolved Hide resolved
paddlespeech/s2t/modules/decoder.py Outdated Show resolved Hide resolved
paddlespeech/s2t/exps/u2/model.py Outdated Show resolved Hide resolved
paddlespeech/s2t/models/u2/u2.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@zh794390558 zh794390558 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jackwaterveg Jackwaterveg merged commit fcc1762 into PaddlePaddle:develop Mar 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

提升 conformer 的精度
2 participants