Missing complete Segment output

[kbramhendra]

hi,
While I am decoding with HLG, There is some random ness to the outputs, sometimes completes segment in the batch is missing and returning empty output, but when I am decoding alone the segment its coming out properly, can you please help to decode the error ?

[csukuangfj]

Is a segment an utterance?

How long is your segment?

Are you able to compare the nnet_output of the segment between batch decoding and decoding alone?

[pkufool]

I think you can first try using a larger beam and max-active-states. And do you use the latest k2? There are some discussion here k2-fsa/icefall#292 not sure whether they are the same issue (i.e about the prunning).

[kbramhendra]

Thank you for replying. The segment is an utterance, its happening with different types of durations...the nnet outputs are same in both the case...this is problem is happening intermittently, beam size 20 and max-active-states is 7000 , yes my issue is similar to 292 but i am using release post that issue. I am using 1.22 dev2022nov19 release.

[pkufool]

I suspect this issue also relates to this #1087, @glynpu @danpovey I think we should revisit that PR (i.e. intersect_dense_pruned with allow parital) and merge it.

[kbramhendra]

For your reference the issue is happening at k2.intersect_dense_pruned. Here i am attaching the lattice (generated by using torch.save(lattice.as_dict(), 'lattice.pt') ). Thanks

lattices.pt.zip

[kbramhendra]

Hi...any suggestion on how to solve this ?

[pkufool]

lattice_2.pdf

You can see that there is no final state in the graph (be pruned ?), so the lattice will be empty after connectting.

I think #1087 can solve this, can you try cherring pick the changes and see if it fix your problem.

[csukuangfj]

Could you post the output of

python3 -m k2.version

?

If you are using an old version of k2, please consider updating it.

[kbramhendra]

Hi for illegal memory output

_collecting environment information...

k2 version: 1.23.4
Build type: Release
Git SHA1:
Git date:
Cuda used to build k2: 10.2
cuDNN used to build k2: 8.7.0
Python version used to build k2: 3.9
OS used to build k2: Ubuntu 18.04.6 LTS
CMake version: 3.22.1
GCC version: 12.1.0
CMAKE_CUDA_FLAGS: -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_70,code=sm_70 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas
CMAKE_CXX_FLAGS: -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/include -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable -Wno-strict-overflow
PyTorch version used to build k2: 1.12.0+cu102
PyTorch is using Cuda: 10.2
NVTX enabled: True
With CUDA: True
Disable debug: True
Sync kernels : True
Disable checks: False
Max cpu memory allocate: 214748364800 bytes (or 200.0 GB)
k2 abort: False
file: /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/lib/python3.9/site-packages/k2-1.23.4.dev20230705+cuda10.2.torch1.12.0-py3.9-linux-x86_64.egg/k2/version/version.py
_k2.file: /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/lib/python3.9/site-packages/k2-1.23.4.dev20230705+cuda10.2.torch1.12.0-py3.9-linux-x86_64.egg/k2.cpython-39-x86_64-linux-gnu.so

[kbramhendra]

With the latest version I ran into some problems...so I tried to install this 1.23.4 which just previous to that. with this and changes in the #1087 am getting this problem. Earlier also i was using 1.23.4 only just with the changes #1087 i am getting this " Check failed: e == cudaSuccess (700 vs. 0) Error: an illegal memory access was encountered " this error

[pkufool]

@kbramhendra OK, I will try to merge #1087 as soon as possible.

[kbramhendra]

Thank you for the help. It would be great if you could do that.

[kbramhendra]

For you reference this is the script i have post integration of #1087. Thank you

intersect_dense_pruned.cu.zip

[pkufool]

@kbramhendra Can you try #1218 , I made some fixes to #1087 , see if it can solve your problem. It still has some problems for chunk by chunk decoding, so I can't merge it now, sorry!

[kbramhendra]

Thanks a lot of your effort and time. I really appreciate it. I have tested it on few samples its seems to be working. I will get back to if i have face any problem. But its great help. Thank you. @pkufool