[Performance]: Deepseek-v2 support

[ZixinxinWang]

Proposal to improve performance

I implement the newest Deepseek-v2 model(which is a MOE with 236B parameter, 21B activated) and it's relatively slow, what else should be optimized

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

[simon-mo]

#4650 (comment)