[WIP] Pruned-transducer-stateless5-for-WenetSpeech (offline and streaming)

[luomingshuang]

A pruned-transducer-stateless5 recipe for WenetSpeech.

[luomingshuang]

Note: The training is running, I will update the results on time.

Results trained with the M subset:
Offline:

decoding-method	epoch	avg	use-averaged-model	dev-CER(%)	test-net-CER(%)	test-meeting-CER(%)
greedy_search	15	9	True	10.43	11.45	19.37
modified_beam_search	15	9	True	10.03	11.13	18.36
fast_beam_search	15	9	True	10.38	11.28	19.5

Streaming:

decoding-method	epoch	avg	use-averaged-model	dev-CER(%)	test-net-CER(%)	test-meeting-CER(%)
greedy_search	15	6	True	11.23	13.12	21.34
fast_beam_search	15	6	True	11.12	12.85	21.18

Results trained with the L subset:
Offline:

decoding-method	epoch	avg	use-averaged-model	dev-CER(%)	test-net-CER(%)	test-meeting-CER(%)
greedy_search	1	1	False	12.18	14.31	25.31
greedy_search	2	1	False	10.6	12.33	20.77
greedy_search	2	1	True	8.79	10.6	15.89
greedy_search	3	1	True	8.89	9.56	16.42
greedy_search	4	1	True	8.22	9.03	14.54
modified_beam_search	4	1	True	8.17	9.04	14.44
fast_beam_search	4	1	True	8.29	9.00	14.93
greedy_search	4	2	True	8.40	9.37	14.89
greedy_search	5	1	True	8.30	8.85	14.57
greedy_search	5	2	True	8.48	9.19	14.81
greedy_search	6	1	True	9.70	9.36	17.24

Streaming (decode.py):

decoding-method	epoch	avg	use-averaged-model	dev-CER(%)	test-net-CER(%)	test-meeting-CER(%)
greedy_search	1	1	False	14.27	17.45	30.80
greedy_search	2	1	False	12.77	15.03	25.83
greedy_search	3	1	False	12.28	13.98	25.07
greedy_search	4	1	False	11.30	12.81	21.83
greedy_search	5	1	False	11.22	12.52	21.83
greedy_search	5	1	True	8.88	10.60	16.68
greedy_search	6	1	True	8.93	10.22	16.26
greedy_search	7	1	True	8.86	9.89	15.93
greedy_search	8	1	True	9.76	11.73	17.66
greedy_search	8	2	True	12.33	14.53	21.25
greedy_search	7	2	True	17.61	19.48	30.03
greedy_search	5	2	True	12.02	14.93	22.46

Streaming (streaming_decode.py):

decoding-method	epoch	avg	use-averaged-model	dev-CER(%)	test-net-CER(%)	test-meeting-CER(%)
greedy_search	5	1	True	8.79	10.80	16.82
greedy_search	5	2	True	11.9	14.93	22.36
greedy_search	6	1	True	8.79	10.47	16.34
greedy_search	7	1	True	8.78	10.12	16.16
greedy_search	8	1	True	9.64	11.77	17.57
fast_beam_search	6	1	True	8.85	10.69	16.34
wenet (ctc greedy search - full)	/	/	/	8.85	9.78	17.77
wenet (ctc greedy search - 16)	/	/	/	9.32	11.02	18.79
wenet (ctc prefix beam search - full)	/	/	/	8.80	9.73	17.57
wenet (ctc prefix beam search - 16)	/	/	/	9.25	10.96	18.62
wenet (attention rescoring - full)	/	/	/	8.60	9.26	17.34
wenet (attention rescoring - 16)	/	/	/	8.87	10.22	18.11

[danpovey]

You have a result there with --use-averaged-model True and --avg 1. I believe that option only supports --avg value greater than one?? Or perhaps we changed this at some point or I misremembered?

[csukuangfj]

You have a result there with --use-averaged-model True and --avg 1. I believe that option only supports --avg value greater than one?? Or perhaps we changed this at some point or I misremembered?

It must be a typo in the table, I believe.

---

Edited: The code does not support --epoch 1 --avg 1 --use-averaged-model 1,
but it supports --epoch X --avg 1 --use-averaged-model 1 where X > 1.

[luomingshuang]

Training commands:

Offline:

export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

python ./pruned_transducer_stateless5/train.py \
  --lang-dir data/lang_char \
  --exp-dir pruned_transducer_stateless5/exp_L_offline \
  --world-size 8 \
  --num-epochs 15 \
  --start-epoch 2 \
  --max-duration 120 \
  --valid-interval 3000 \
  --model-warm-step 3000 \
  --save-every-n 8000 \
  --average-period 1000 \
  --training-subset L

Streaming:

export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

python ./pruned_transducer_stateless5/train.py \
  --lang-dir data/lang_char \
  --exp-dir pruned_transducer_stateless5/exp_L_streaming \
  --world-size 8 \
  --num-epochs 15 \
  --start-epoch 1 \
  --max-duration 140 \
  --valid-interval 3000 \
  --model-warm-step 3000 \
  --save-every-n 8000 \
  --average-period 1000 \
  --training-subset L \
  --dynamic-chunk-training True \
  --causal-convolution True \
  --short-chunk-size 25 \
  --num-left-chunks 4

Decoding commands:

Offline:

export CUDA_VISIBLE_DEVICES='0'

python ./pruned_transducer_stateless5/decode.py \
  --lang-dir data/lang_char \
  --exp-dir pruned_transducer_stateless5/exp_L_offline \
  --use-averaged-model True \
  --max-duration 600 \
  --epoch 4 \
  --avg 1 \
  --decoding-method greedy_search \
  --simulate-streaming 0 \
  --causal-convolution 0 \
  --decode-chunk-size 16 \
  --left-context 64

Streaming:

export CUDA_VISIBLE_DEVICES='0'

python pruned_transducer_stateless5/streaming_decode.py \
        --epoch 6 \
        --avg 1 \
        --decode-chunk-size 16 \
        --left-context 64 \
        --right-context 0 \
        --exp-dir ./pruned_transducer_stateless5/exp_L_streaming \
        --use-averaged-model True \
        --decoding-method greedy_search \
        --num-decode-streams 200

[luomingshuang]

Best Results Conclusion:

Offline:

decoding-method	epoch	avg	use-averaged-model	DEV	TEST-NET	TEST-MEETING
greedy_search	4	1	True	8.22	9.03	14.54
modified_beam_search	4	1	True	8.17	9.04	14.44
fast_beam_search	4	1	True	8.29	9.00	14.93
pruned-rnnt2_greedy search	10	2	/	7.80	8.75	13.49
pruned-rnnt2_modified beam search (beam size 4)	10	2	/	7.76	8.71	13.41
pruned-rnnt2_fast beam search (set as default)	10	2	/	7.94	8.74	13.80
wenet (ctc greedy search - full)	/	/	/	8.85	9.78	17.77
wenet (ctc prefix beam search - full)	/	/	/	8.80	9.73	17.57
wenet (attention rescoring - full)	/	/	/	8.60	9.26	17.34

Streaming:

decoding-method	epoch	avg	use-averaged-model	dev-CER(%)	test-net-CER(%)	test-meeting-CER(%)
greedy_search	7	1	True	8.78	10.12	16.16
modified_beam_search	7	1	True	8.53	9.95	15.81
fast_beam_search	7	1	True	9.01	10.47	16.28
wenet (ctc greedy search - 16)	/	/	/	9.32	11.02	18.79
wenet (ctc prefix beam search - 16)	/	/	/	9.25	10.96	18.62
wenet (attention rescoring - 16)	/	/	/	8.87	10.22	18.11

[luomingshuang]

It seems that there are no specially many deletions in the start of the recognized results based on pruned rnnt5 comparing to pruned rnnt2.

[danpovey]

It seems that there are no specially many deletions in the start of the recognized results based on pruned rnnt5 comparing to pruned rnnt2.

Yes, we realized that the issue only affects my setup because the "padding_idx=0" arg to nn.Embedding makes the initial parameters and grads zero for the 0th row of the embedding tensor. So it turns out, other optimizers are unaffected.

[wgb14]

There are many results, can you or someone else please point to the specific comparison where it is worse? We should investigate if so. I'd like to see whether the training data contains any instances of repeated words that are not spoken, this might explain the repeated words in the transcript.

Is there any insight why this recipe performed worse than pruned_transducer_stateless2 in batch mode?

Sure, I meant the WER numbers. Copied from https://github.com/k2-fsa/icefall/blob/master/egs/wenetspeech/ASR/RESULTS.md:

Pruned Transducer 5:

Using the codes from this PR #447.

When training with the L subset, the CERs are

Offline:

decoding-method	epoch	avg	use-averaged-model	DEV	TEST-NET	TEST-MEETING
greedy_search	4	1	True	8.22	9.03	14.54
modified_beam_search	4	1	True	8.17	9.04	14.44
fast_beam_search	4	1	True	8.29	9.00	14.93

Pruned Transducer 2:

Using the codes from this PR #349.

When training with the L subset, the CERs are

	dev	test-net	test-meeting	comment
greedy search	7.80	8.75	13.49	--epoch 10, --avg 2, --max-duration 100
modified beam search (beam size 4)	7.76	8.71	13.41	--epoch 10, --avg 2, --max-duration 100
fast beam search (1best)	7.94	8.74	13.80	--epoch 10, --avg 2, --max-duration 1500
fast beam search (nbest)	9.82	10.98	16.37	--epoch 10, --avg 2, --max-duration 600
fast beam search (nbest oracle)	6.88	7.18	11.77	--epoch 10, --avg 2, --max-duration 600
fast beam search (nbest LG, ngram_lm_scale=0.35)	8.83	9.88	15.47	--epoch 10, --avg 2, --max-duration 600

You can see all 3 decoding method perform worse. Here I assume offline is batch mode and this is a fair comparison

[wgb14]

I asked because we want to train a streaming model on gigaspeech, and I'm trying to find a proper recipe to start with.

[danpovey]

Oh, that is not good. @luomingshuang is there any chance you could show me a tensorboard log that compares the 2 runs?
One possibility is that we have moved to a new sampler that does not globally randomize the samples, and they are in some very nonrandom order that interacts badly with training. For librispeech this seemed to help slightly, but it may not be the same here necessarily. It could be other things thouhgh, we need to investigate.

One thing I notice is that with the old model, the best results are obtained at a much later epoch. It could be that something is going wrong with the model averaging. @luomingshuang can you include some results with --avg 1 and --use-averaged-model False, for all the epochs that we have? And please remind me of the locations of the new and old systems and run ~dpovey/bin/analyze.py for all the epochs that you have, so I can have a look and see if anything weird is going on with the norms of the parameter values. It would also be great if you could run with --print-diagnostics=True and save the output, for all the epochs you have, for the new model at least, so I can see if anything weird is visible in those diagnostics.

[danpovey]

Also @luomingshuang, for the "Offline" results reported in this PR, were those obtained with the same model as the streaming results? If so, could the fact that this is a streaming model explain the WER increase vs. pruned_transducer_stateless2? (However, it could not explain the degradation with increasing epoch).

[pzelasko]

Oh, that is not good. @luomingshuang is there any chance you could show me a tensorboard log that compares the 2 runs? One possibility is that we have moved to a new sampler that does not globally randomize the samples, and they are in some very nonrandom order that interacts badly with training. For librispeech this seemed to help slightly, but it may not be the same here necessarily. It could be other things thouhgh, we need to investigate.

In case of large datasets like Gigaspeech I recommend shuffling the manifests with shuf or sth before starting the training.

[csukuangfj]

You can see all 3 decoding method perform worse. Here I assume offline is batch mode and this is a fair comparison

Please pay attention to the number of epochs that has been trained.

The worse one is trained only for 4 epochs while the better one is 10 epochs.

[wgb14]

Please pay attention to the number of epochs that has been trained.

The worse one is trained only for 4 epochs while the better one is 10 epochs.

Make sense. Do you have numbers from the last epoch?

[danpovey]

Oh, that is not good. @luomingshuang is there any chance you could show me a tensorboard log that compares the 2 runs? One possibility is that we have moved to a new sampler that does not globally randomize the samples, and they are in some very nonrandom order that interacts badly with training. For librispeech this seemed to help slightly, but it may not be the same here necessarily. It could be other things thouhgh, we need to investigate.

In case of large datasets like Gigaspeech I recommend shuffling the manifests with shuf or sth before starting the training.

Guys, we should try this at some point. E.g. on full Librispeech or WeNetSpeech. I don't want to change the recipes without testing it though.

[csukuangfj]

Yes, I agree.

[luomingshuang]

The comparison of tensorboard logs (pruned rnnt2 (part1 and part2) and pruned rnnt5) https://tensorboard.dev/experiment/4tBm3f1aSvuzDxyuWeYuLQ/#scalars&_smoothingWeight=0.97:

It seems that the training loss in pruned rnnt5 maybe can converge forward if continue to train.

I compare the performance of pruned rnnt2 and pruned rnnt5 for each epoch as follows (the max-durations are different between pruned rnnt2 and pruned rnnt5 training processes):

model	decoding-method	epoch	avg	use-averaged-model	dev (CER%)	test-net(CER%)	test-meeting(CER%)
pruned_rnnt2	greedy_search	0	1	False	12.24	14.52	24.24
pruned_rnnt2	greedy_search	1	1	False	11.07	12.29	23.52
pruned_rnnt2	greedy_search	2	1	False	10.28	11.17	19.79
pruned_rnnt2	greedy_search	3	1	False	10.16	10.82	19.31
pruned_rnnt2	greedy_search	4	1	False	9.69	10.40	18.81
pruned_rnnt2	greedy_search	5	1	False	9.22	9.98	17.98
pruned_rnnt2	greedy_search	6	1	False	9.34	9.84	18.35
pruned_rnnt2	greedy_search	7	1	False	9.50	9.76	17.79
pruned_rnnt2	greedy_search	8	1	False	9.63	9.82	19.84
pruned_rnnt2	greedy_search	9	1	False	9.10	9.32	16.63
pruned_rnnt2	greedy_search	10	1	False	9.07	9.35	17.43
pruned_rnnt5	greedy_search	1	1	False	12.16	14.30	25.30
pruned_rnnt5	greedy_search	2	1	False	10.59	12.33	20.73
pruned_rnnt5	greedy_search	3	1	False	10.01	11.28	19.82
pruned_rnnt5	greedy_search	4	1	False	9.86	10.76	19.12
pruned_rnnt5	greedy_search	5	1	False	9.98	10.62	19.94
pruned_rnnt5	greedy_search	6	1	False	9.30	9.96	17.71
pruned_rnnt5	greedy_search	7	1	False	9.68	10.11	18.53
pruned_rnnt5	greedy_search	8	1	False	10.28	10.00	19.94
pruned_rnnt5	greedy_search	2	1	True	8.79	10.60	15.89
pruned_rnnt5	greedy_search	3	1	True	8.89	9.56	16.42
pruned_rnnt5	greedy_search	4	1	True	8.22	9.03	14.54
pruned_rnnt5	greedy_search	5	1	True	8.30	8.85	14.57
pruned_rnnt5	greedy_search	6	1	True	9.70	9.36	17.24
pruned_rnnt5	greedy_search	7	1	True	8.07	8.79	13.73
pruned_rnnt5	greedy_search	8	1	True	7.97	8.35	13.69
pruned_rnnt5	greedy_search	9	1	True	7.91	8.31	13.56
pruned_rnnt5	modified_beam_search	9	1	True	7.88	8.39	13.66
pruned_rnnt5	fast_beam_search	9	1	True	8.03	8.31	14.07
pruned_rnnt5	greedy_search	10	1	True	7.95	8.43	13.37
pruned_rnnt5	modified_beam_search	10	1	True	7.98	8.56	13.58
pruned_rnnt5	fast_beam_search	10	1	True	8.07	8.40	13.91
pruned_rnnt5	greedy_search	10	2	True	9.52	11.05	14.77
pruned_rnnt5	greedy_search	11	1	True	8.40	8.10	14.54
pruned_rnnt5	greedy_search	12	1	True	8.89	9.85	14.56
pruned_rnnt5	greedy_search	13	1	True	9.45	11.08	15.25

It seems the results between based on use-averaged-model=True and use-averaged-model=False are very different, such as |pruned_rnnt5| greedy_search | 8 | 1| False| 10.28|10.00 |19.94 | and |pruned_rnnt5| greedy_search | 8 | 1| True| 7.79| 8.35 | 13.69 |. Or it is necessary to continue training for some epochs. Sorry, I look for the best results based on use-averaged-model=False, so I thought we can get the best results when epoch=4 and avg=1. Now, it seems we should look for the best results based on use-averaged-model=True

[csukuangfj]

Please remove this else branch. it does not make sense.

[danpovey]

OK, fine. I suppose you can try more epochs, e.g. at least train until 10 epochs to match the baseline?

[luomingshuang]

OK, I will try to train it based on the pretrained epoch-x.pt until 10 epochs or more.

OK, fine. I suppose you can try more epochs, e.g. at least train until 10 epochs to match the baseline?

[luomingshuang]

The new results for wenetspeech based on pruned rnnt5 and the comparison of results between pruned rnnt5 and pruned rnnt2 are as follows ( the best results of pruned rnnt5 are more closed to pruned rnnt2's at present ):

model	decoding-method	epoch	avg	use-averaged-model	dev (CER%)	test-net(CER%)	test-meeting(CER%)
pruned_rnnt5	greedy_search	9	1	True	7.91	8.31	13.56
pruned_rnnt5	modified_beam_search	9	1	True	7.88	8.39	13.66
pruned_rnnt5	fast_beam_search	9	1	True	8.03	8.31	14.07
pruned_rnnt2	greedy search	10	2	False	7.80	8.75	13.49
pruned_rnnt2	modified beam search (beam size 4)	10	2	False	7.76	8.71	13.41
pruned_rnnt2	fast beam search (1best)	10	2	False	7.94	8.74	13.80

[danpovey]

OK; can you report the results with --avg 2 --use-averaged-model False, for epoch 10 when you get them? Those should in principle be similar to the pruned_rnn2 baseline. The number of parameters is the same?

[luomingshuang]

model	decoding-method	epoch	avg	use-averaged-model	dev (CER%)	test-net(CER%)	test-meeting(CER%)
pruned_rnnt5	greedy_search	10	2	True	9.52	11.05	14.77
pruned_rnnt5	greedy_search	10	2	False	10.25	12.34	15.50

According the above table, it seems that the results based on avg=2 get worse than avg=1.

About the number of model parameters:

model	the number of model parameters
pruned rnnt2	88978927
pruned rnnt5	97487351

Is a relatively small and easy neural network better for wenetspeech?

[lianrzh]

@danpovey @luomingshuang Is this model pretrained_epoch_7_avg_1.pt the average of epoch6 and epoch7?
I used this model to decode the test datasets and found there was no way to reproduce the results.The model is in ./pruned_transducer_stateless5/exp_L_streaming.
ln -s pretrained_epoch_7_avg_1.pt epoch-7.pt
python ./pruned_transducer_stateless5/streaming_decode.py \ --exp-dir ./pruned_transducer_stateless5/exp_L_streaming \ --use-averaged-model False \ --epoch 7 \ --avg 1 \ --decode-chunk-size 16 \ --left-context 64 \ --right-context 0 \ --decoding-method greedy_search \ --num-decode-streams 2000
The results of my run are shown below.
greedy_search 11.12 best for DEV
greedy_search 12.92 best for TEST_NET
greedy_search 21.86 best for TEST_MEETING

Please help me to find out which process may be wrong. Thank You.

[luomingshuang]

pretrained_epoch_7_avg_1.pt means only epoch 7. Maybe you can have a try again with the following command:

export CUDA_VISIBLE_DEVICES='0'

python pruned_transducer_stateless5/streaming_decode.py \
        --epoch 7 \
        --avg 1 \
        --decode-chunk-size 16 \
        --left-context 64 \
        --right-context 0 \
        --exp-dir ./pruned_transducer_stateless5/exp_L_streaming \
        --use-averaged-model True \
        --decoding-method greedy_search \
        --num-decode-streams 200

Note: set use-averaged-model as True.

@danpovey @luomingshuang Is this model pretrained_epoch_7_avg_1.pt the average of epoch6 and epoch7? I used this model to decode the test datasets and found there was no way to reproduce the results.The model is in ./pruned_transducer_stateless5/exp_L_streaming. ln -s pretrained_epoch_7_avg_1.pt epoch-7.pt python ./pruned_transducer_stateless5/streaming_decode.py \ --exp-dir ./pruned_transducer_stateless5/exp_L_streaming \ --use-averaged-model False \ --epoch 7 \ --avg 1 \ --decode-chunk-size 16 \ --left-context 64 \ --right-context 0 \ --decoding-method greedy_search \ --num-decode-streams 2000 The results of my run are shown below. greedy_search 11.12 best for DEV greedy_search 12.92 best for TEST_NET greedy_search 21.86 best for TEST_MEETING

Please help me to find out which process may be wrong. Thank You.

[danpovey]

thanks @luomingshuang !
I assume that requires to also make a link to epoch-6.pt?

[csukuangfj]

I will try to find epoch-6.pt

[luomingshuang]

Em....I am not sure now. If can have a test for this, it will be OK.

[csukuangfj]

Here is the exp path on our server:

/ceph-ms/luomingshuang/codes/icefall-wenetspeech-pruned-rnnt5/egs/wenetspeech/ASR/pruned_transducer_stateless5/exp_L_streaming

Post it here as a reminder.

---

I am creating a PR at

https://huggingface.co/luomingshuang/icefall_asr_wenetspeech_pruned_transducer_stateless5_streaming/upload/main/exp

to upload epoch-6.pt.

[csukuangfj]

We have uploaded epoch-6.pt.

Please retry.

[lianrzh]

@csukuangfj The size of epoch-6.pt is 1.56G and epoch-7.pt is 390M.
The two models cannot be averaged, and such errors will be reported：
"icefall/checkpoint.py", line 424, in average_checkpoints_with_averaged_model batch_idx_train_end = state_dict_end["batch_idx_train"] KeyError: 'batch_idx_train'

[csukuangfj]

Please wait a moment, I will upload epoch-7.pt

[luomingshuang]

390M means that the epoch-7.pt checkpoint file just contains model parameters.

[csukuangfj]

The size of epoch-6.pt is 1.56G and epoch-7.pt is 390M.

The difference in size is due to that epoch-6.pt is saved by train.py, which also contains optimizer.state_dict().
While pretrained_epoch_7_avg_1.pt is exported by export.py, which contains only model.state_dict().

[csukuangfj]

@lianrzh
I just uploaded epoch-7.pt.

Please retry.

I am uploading epoch-{3,4,5}.pt

[csukuangfj]

@lianrzh
We have uploaded epoch-3.pt, epoch-4.pt, epoch-5.pt, epoch-6.pt, and epoch-7.pt.

Please try it again.