-
Notifications
You must be signed in to change notification settings - Fork 221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
error of cut_set.compute_and_store_features #320
Comments
Could you show the output of
? |
Yes,
|
Yes,
|
According to the error message:
Lines 153 to 164 in ef7a037
Could you use the above code to print num_samples and compare the value with |
@csukuangfj , Thanks a lot, Fanjun.
I found that it is the same as
it is converted seconds is 00:19:59:33==19*60+59.33 ==1199.33 |
I believe soxi's output uses rounding.
Can you show the value of Also, the
does not match the error message. The error message says
Perhaps the code chooses |
summary it is as follows:
I still don't find the error message says
|
I found that the error is from Line 301 in ef7a037
|
In that case, I think you can trace the program to find the place where |
Cool, good debugging! BTW you might want to validate (a subset) of your manifests with „validate(recordings, read_data=True)” that checks if the manifest correctly describes the data. It could help locate the step that corrupted the metadata.
… Wiadomość napisana przez Fangjun Kuang ***@***.***> w dniu 6/19/21, o godz. 05:59:
I found that the error is from
In that case, I think you can trace the program to find the place where duration is changed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
@pzelasko ,I have validated my data is correct using
I believe the above command is correct.
I found that the above error is caused by rounding the duration.
I look for where is round the duration(e.g:1199.3274375 -> 1199.328 ). |
Can you check reco2dur in Kaldi data dir? Lhotse simply reads it when importing and computes the num samples from that. If Kaldi does rounding in reco2dur then num samples would be incorrect.
See https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/kaldi.py#L50
… Wiadomość napisana przez shanguanma ***@***.***> w dniu 6/19/21, o godz. 06:41:
@pzelasko ,I have validated my data is correct using utils/validate_data_dir.sh --no-feats folder, then I use lhotse command
covered Kaldi format to lhotse format, the command is as follows:
for name in train_trn train_cv dev_sge dev_man; do
mkdir -p lhotse_data/$name
lhotse kaldi import kaldi_format/$name 16000 lhotse_data/$name
done
I believe the above command is correct.
Cool, good debugging! BTW you might want to validate (a subset) of your manifests with „validate(recordings, read_data=True)” that checks if the manifest correctly describes the data. It could help locate the step that corrupted the metadata.
I found that the above error is caused by rounding the duration.
https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/audio.py#L247
I look for where is round the duration(e.g:1199.3274375 -> 1199.328 ).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
In Kaldi, I don't think reco2dur is ever used for anything very exact, so
it's definitely possible that there could be excessive rounding going on.
…On Sat, Jun 19, 2021 at 7:29 PM Piotr Żelasko ***@***.***> wrote:
Can you check reco2dur in Kaldi data dir? Lhotse simply reads it when
importing and computes the num samples from that. If Kaldi does rounding in
reco2dur then num samples would be incorrect.
See
https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/kaldi.py#L50
> Wiadomość napisana przez shanguanma ***@***.***> w dniu 6/19/21, o godz.
06:41:
>
>
> @pzelasko ,I have validated my data is correct using
utils/validate_data_dir.sh --no-feats folder, then I use lhotse command
> covered Kaldi format to lhotse format, the command is as follows:
>
> for name in train_trn train_cv dev_sge dev_man; do
> mkdir -p lhotse_data/$name
> lhotse kaldi import kaldi_format/$name 16000 lhotse_data/$name
> done
> I believe the above command is correct.
>
> Cool, good debugging! BTW you might want to validate (a subset) of your
manifests with „validate(recordings, read_data=True)” that checks if the
manifest correctly describes the data. It could help locate the step that
corrupted the metadata.
>
> I found that the above error is caused by rounding the duration.
>
>
https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/audio.py#L247
>
> I look for where is round the duration(e.g:1199.3274375 -> 1199.328 ).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#320 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLO2SWIFOUGS4MG2I4K3TTR5QZANCNFSM466TRGQQ>
.
|
In that case we might need to change the import data dir function to read the audio samples directly and compute the duration and num samples. Might be good to allow parallel processing as its potentially slow (especially with pipes in wav scp). I’m back in July, if anybody wants to pick it up before then, feel free to do it.
… Wiadomość napisana przez Daniel Povey ***@***.***> w dniu 6/19/21, o godz. 07:59:
In Kaldi, I don't think reco2dur is ever used for anything very exact, so
it's definitely possible that there could be excessive rounding going on.
On Sat, Jun 19, 2021 at 7:29 PM Piotr Żelasko ***@***.***>
wrote:
> Can you check reco2dur in Kaldi data dir? Lhotse simply reads it when
> importing and computes the num samples from that. If Kaldi does rounding in
> reco2dur then num samples would be incorrect.
>
> See
> https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/kaldi.py#L50
>
> > Wiadomość napisana przez shanguanma ***@***.***> w dniu 6/19/21, o godz.
> 06:41:
> >
> >
> > @pzelasko ,I have validated my data is correct using
> utils/validate_data_dir.sh --no-feats folder, then I use lhotse command
> > covered Kaldi format to lhotse format, the command is as follows:
> >
> > for name in train_trn train_cv dev_sge dev_man; do
> > mkdir -p lhotse_data/$name
> > lhotse kaldi import kaldi_format/$name 16000 lhotse_data/$name
> > done
> > I believe the above command is correct.
> >
> > Cool, good debugging! BTW you might want to validate (a subset) of your
> manifests with „validate(recordings, read_data=True)” that checks if the
> manifest correctly describes the data. It could help locate the step that
> corrupted the metadata.
> >
> > I found that the above error is caused by rounding the duration.
> >
> >
> https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/audio.py#L247
> >
> > I look for where is round the duration(e.g:1199.3274375 -> 1199.328 ).
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub, or unsubscribe.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#320 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAZFLO2SWIFOUGS4MG2I4K3TTR5QZANCNFSM466TRGQQ>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Possibly the issue only occurs past a certain duration.
On Sat, Jun 19, 2021 at 8:05 PM Piotr Żelasko ***@***.***>
wrote:
… In that case we might need to change the import data dir function to read
the audio samples directly and compute the duration and num samples. Might
be good to allow parallel processing as its potentially slow (especially
with pipes in wav scp). I’m back in July, if anybody wants to pick it up
before then, feel free to do it.
> Wiadomość napisana przez Daniel Povey ***@***.***> w dniu 6/19/21, o
godz. 07:59:
>
>
> In Kaldi, I don't think reco2dur is ever used for anything very exact, so
> it's definitely possible that there could be excessive rounding going on.
>
>
> On Sat, Jun 19, 2021 at 7:29 PM Piotr Żelasko ***@***.***>
> wrote:
>
> > Can you check reco2dur in Kaldi data dir? Lhotse simply reads it when
> > importing and computes the num samples from that. If Kaldi does
rounding in
> > reco2dur then num samples would be incorrect.
> >
> > See
> >
https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/kaldi.py#L50
> >
> > > Wiadomość napisana przez shanguanma ***@***.***> w dniu 6/19/21, o
godz.
> > 06:41:
> > >
> > >
> > > @pzelasko ,I have validated my data is correct using
> > utils/validate_data_dir.sh --no-feats folder, then I use lhotse command
> > > covered Kaldi format to lhotse format, the command is as follows:
> > >
> > > for name in train_trn train_cv dev_sge dev_man; do
> > > mkdir -p lhotse_data/$name
> > > lhotse kaldi import kaldi_format/$name 16000 lhotse_data/$name
> > > done
> > > I believe the above command is correct.
> > >
> > > Cool, good debugging! BTW you might want to validate (a subset) of
your
> > manifests with „validate(recordings, read_data=True)” that checks if
the
> > manifest correctly describes the data. It could help locate the step
that
> > corrupted the metadata.
> > >
> > > I found that the above error is caused by rounding the duration.
> > >
> > >
> >
https://github.com/lhotse-speech/lhotse/blob/ef7a037426f1b602a54f4d9ea43e711007e85719/lhotse/audio.py#L247
> > >
> > > I look for where is round the duration(e.g:1199.3274375 -> 1199.328
).
> > >
> > > —
> > > You are receiving this because you were mentioned.
> > > Reply to this email directly, view it on GitHub, or unsubscribe.
> >
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub
> > <
#320 (comment)
>,
> > or unsubscribe
> > <
https://github.com/notifications/unsubscribe-auth/AAZFLO2SWIFOUGS4MG2I4K3TTR5QZANCNFSM466TRGQQ
>
> > .
> >
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#320 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLO5IASHNUUI37EAAEFLTTSBYRANCNFSM466TRGQQ>
.
|
See Line 50 in ef7a037
You are right. we should read the audio samples directly and compute the duration and num samples.
Yes, Only part sentences have such errors. |
my point is that we can probably rely on reco2dur for shorter wav files.
…On Tue, Jun 22, 2021 at 6:17 AM Piotr Żelasko ***@***.***> wrote:
Closed #320 <#320> via #321
<#321>.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#320 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLOZJRDPOTRMTU7SLORTTT63BDANCNFSM466TRGQQ>
.
|
When I use snowfall to run an asr model, at the prepare Lhotse formate data,I encountered the following problems.
Error detail log is as follows:
Could you help me to solve it? @pzelasko
Thanks a lot.
The text was updated successfully, but these errors were encountered: