v3.1.0

❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤

What's Changed

• [ctc] Update search.py by @pengzhendong in #2398
• fix mask to bias by @Mddct in #2401
• [ssl/w2vbert] weight copy from meta w2vbert-2.0 by @Mddct in #2392
• [lint] fix linter version by @xingchensong in #2405
• [search] Update search.py by @xingchensong in #2406
• fix mask bias dtype in sdpa by @Mddct in #2407
• Fix ckpt conversion bug by @zhr1201 in #2399
• [dataset] restrict batch type by @Mddct in #2410
• [wenet/bin/recognize.py] modify args to be consistent with train by @Mddct in #2411
• [transformer] remove pe to device by @Mddct in #2413
• add timer for steps by @Mddct in #2416
• [dataset] support repeat by @Mddct in #2415
  • (!! breaking changes, we recommand step_save instead of epoch_save !!) 🚀🚀🚀
• [transformer] fix sdpa u2pp training nan by @Mddct in #2419
  • (!! important bug fix, enjoy flash attention without pain !!) 🚀🚀🚀
• [transformer] fix sdpa mask for ShowRelAttention by @xingchensong in #2420
• [runtime/libtorch] fix jit issue by @xingchensong in #2421
• [dataset] add shuffle at shards tar/raw file level by @kakashidan in #2424
• [dataset] fix cycle in recognize.py by @Mddct in #2426
• [dataset] unify shuf conf by @Mddct in #2427
• fix order by @Mddct in #2428
• [runtime] upgrade libtorch version to 2.1.0 by @xingchensong in #2418
• [torchaudio] Fix torchaudio interface error (#2352) by @lsrami in #2429
• [paraformer] fsdp fix submodule call by @Mddct in #2431
• fix modify by @Mddct in #2436
• [deprecated dataset] small fix by @kakashidan in #2440
• [dataset] add singal channel conf & processor by @kakashidan in #2439
• fix list shuffle in recognize.py by @Mddct in #2446
• fix list_shuffle in cv_conf by @Mddct in #2447
• [runtime] Fixed failed compilation without ITN. Now, compiling ITN is mandatory. by @roney123 in #2444
• [runtime] add blank_sacle in ctc_endpoint by @jia-jidong in #2374
• fix step in continue training in steps mode by @Mddct in #2453
• fix export_jit.py by @Mddct in #2455
• [fix] fix copyright by @robin1001 in #2456
• [fix] fix copyright by @xingchensong in #2457
• fix llama rope by @Mddct in #2459
• [train_engine] support fsdp by @Mddct in #2412
  • (!! breaking changes, enjoy both fsdp & deepspeed !!) 🚀🚀🚀
• [env] update python version and deepspeed version by @xingchensong in #2462
  • (!! breaking changes, you may need to update your env !!) ❤❤❤
• fix rope pos embdining by @Mddct in #2463
• [transformer] add multi warmup and learning rate for different modules by @Mddct in #2449
  • (!! Significant improvement on results of whisper !!) 💯💯💯
• [whisper] limit language to Chinese by @xingchensong in #2470
• [train] convert tensor to scalar by @xingchensong in #2471
• [workflow] upgrad python version to 3.10 by @xingchensong in #2472
  • (!! breaking changes, you may need to update your env !!) ❤❤❤
• refactor cache behaviour in training mode (reduce compute cost and me… by @Mddct in #2473
• fix ut by @Mddct in #2477
• [transformer] Make MoE runnable by @xingchensong in #2474
• [transformer] fix mqa by @Mddct in #2478
• enable mmap in torch.load by @Mddct in #2479
• [example] Add deespeed configs of different stages for illustrative purposes by @xingchensong in #2485
• [example] Fix prefetch and step_save by @xingchensong in #2486
  • (!! Significant decrease on cpu ram !!) 💯💯💯
• [ctl] simplified ctl by @Mddct in #2483
• [branchformer] simplified branchformer by @Mddct in #2482
• [e_branchformer] simplified e_branchformer by @Mddct in #2484
• [transformer] refactor cache by @Mddct in #2481
• fix gradient ckpt in branchformer/ebranformer by @Mddct in #2488
• [transformer] fix search after refactor cache by @Mddct in #2490
• [transformer] set use_reentrant=False for gradient ckpt by @xingchensong in #2491
• [transformer] fix warning: ignore(True) has been deprecated by @xingchensong in #2492
• [log] avoid reduntant logging by @xingchensong in #2493
• [transformer] refactor mqa repeat by @Mddct in #2497
• [transformer] fix mqa in cross att by @Mddct in #2498
• [deepspeed] update json config by @xingchensong in #2499
• [onnx] clone weight for whisper by @xingchensong in #2501
• [wenet/utils/train_utils.py] fix log by @Mddct in #2504
• [transformer] keep high precisioin in softmax by @Mddct in #2508
• [websocket] 8k and 16k support by @Sang-Hoon-Pakr in #2505
• [Fix #2506] Specify multiprocessing context in DataLoader by @MengqingCao in #2507
• [mask] set max_chunk_size according to subsample rate by @xingchensong in #2520
• Revert "[Fix #2506] Specify multiprocessing context in DataLoader" by @xingchensong in #2521
• [transformer] try to fix mga in onnxruntime by @Mddct in #2519
• [utils] update precision of speed metric by @xingchensong in #2524
• fix segmentfault in (#2506) by @MengqingCao in #2530

New modules and methods (from LLM community) by @Mddct & @fclearner 🤩🤩🤩

• [transformer] support multi query attention && multi goruped by @Mddct in #2403
• [transformer] add rope for transformer/conformer by @Mddct in #2458
• LoRA support by @fclearner in #2049

New Contributors

• @lsrami made their first contribution in #2429
• @jia-jidong made their first contribution in #2374
• @MengqingCao made their first contribution in #2507

Full Changelog: v3.0.1...v3.1.0

❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤