"Check failed" after decoding + shortest_path

[pzelasko]

I've been playing around with the recipe to get to know k2 better. I trained the model for 10 epochs and then tried to perform decoding with the following code (running everything on the CPU):

def create_G_decoding_graph(G, L):
    G = k2.arc_sort(G)
    decoding_graph = k2.intersect(G, L).invert_()
    decoding_graph = k2.add_epsilon_self_loops(decoding_graph)
    return decoding_graph

LG = create_G_decoding_graph(G, L)
LG.scores.requires_grad_(False)

acoustic_fsa = k2.DenseFsaVec(nnet_output, supervision_segments)

lattices = k2.intersect_dense_pruned(LG, acoustic_fsa, search_beam=5.0, output_beam=5.0, min_active_states=10, max_active_states=200)

best_paths = k2.fsa_algo.shortest_path(lattices, True)

# Trying to get the recognized words as strings for the first supervision segment in the batch
hyp_words = [symbol_table._id2sym[x.item()] for x in best_paths[0].aux_labels if x > 0]

This crashes with the following message + stack trace:

[F] /home/pzelasko/k2-repo/k2/csrc/array.h:Range:107 Check failed: size <= Dim() - start (32525 vs. 0)


[ Stack-Trace: ]
/home/pzelasko/k2-repo/build/lib/libk2_log.so(k2::internal::GetStackTrace()+0x38) [0x7f0cdcef8508]
/home/pzelasko/k2-repo/build/lib/libk2context.so(+0x6de0a) [0x7f0cdd437e0a]
/home/pzelasko/k2-repo/build/lib/libk2context.so(k2::Array1<int>::Range(int, int) const+0x2c7) [0x7f0cdd44ac27]
/home/pzelasko/k2-repo/build/lib/libk2context.so(k2::RaggedShape::Index(int, int, int*)+0x322) [0x7f0cdd50c5f2]
/home/pzelasko/miniconda3/envs/k2env/lib/python3.7/site-packages/_k2.cpython-37m-x86_64-linux-gnu.so(+0x673fc) [0x7f0ce24fc3fc]
/home/pzelasko/miniconda3/envs/k2env/lib/python3.7/site-packages/_k2.cpython-37m-x86_64-linux-gnu.so(+0x17c76) [0x7f0ce24acc76]
/home/pzelasko/miniconda3/envs/k2env/bin/python(_PyMethodDef_RawFastCallKeywords+0x274) [0x56379f2da914]

Some questions:

1. Did the decoding fail somehow, or did I construct the LG incorrectly? I was using the G.fsa.txt created by run.sh.
2. The decoding on the CPU consumes a lot of memory (9GB with the settings I put in here, if search_beam = 20 or even 10 it's going >20GB). Is this because all the utterances in the batch (N = 42) are being decoded simultaneously? I believe I'm using a release build.

[qindazhu]

As G contains disambiguation symbols bug L does not, we should use L_disambig.fst.txt in data/lang_nosp for L. We also need to determinize the intersection result on phones and then remove disambiguation symbols. I just wrap determinize and rm-epsilon to python this week early, will write decoding example script next week after I finish work at hand.
RE memory usage, I never call intersect_dense_pruned on CPU before, has no idea for now, maybe @danpovey can give some insights.

[pzelasko]

Wait, does G really contain disambiguation symbols? I thought they only exist in the L alphabet (as "phones") and not in the G alphabet (as "words").

[qindazhu]

You can check the end of words.txt, it should contain #0 etc.

[xiaohui-zhang]

@pzelasko here is the reason why we need #0 in G http://kaldi-asr.org/doc/graph.html#graph_disambig. @qindazhu thanks for your update, looking forward to your decoding example!

[danpovey]

Thanks for your comment! I don't think we committed the fix yet (@qindazhu please do!) but we discovered, I think, a problem in the script that creates G.fst in snowfall, that it was including the end of sentence marker `</s>`.

…

On Fri, Dec 4, 2020 at 12:42 PM Xiaohui Zhang ***@***.***> wrote: @pzelasko <https://github.com/pzelasko> here is the reason why we need #0 in G http://kaldi-asr.org/doc/graph.html#graph_disambig. @qindazhu <https://github.com/qindazhu> thanks for your update, looking forward to your decoding example! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLOZG6HI6FJDIIL5KJBLSTBSDDANCNFSM4UFEQKZA> .

[qindazhu]

Yeah, I have removed </s> in G and I can now get determinized result of LG, but the problem I have now is: I need to get aux_labels (i.e. word) from the determinized LG (on labels, i.e. phones), we (@danpovey and I) suppose for arcs in Det(LG), those corresponding arcs in LG should have at most one non-epsilon aux_labels. However, I found that there may be multiple non-epsilon aux_labels for an arc in Det(L*G) (even if I have removed disambig symbols in Det(L*G) before selecting aux_labels). There must be something wrong, I need to step into it to see what happend.

Anyway, will make a PR with my lastest updates which removing </s> in G.

[qindazhu]

Fixed </s> stuff in https://github.com/k2-fsa/snowfall/pull/33/files

[pzelasko]

Thanks! Never too late to learn.