WIP: [MPT] Support MPT-7b-instruct model

[vvchernov]

There is implementation of the original mpt-7b-instruct model from hugging face on Relax and some updates on mlc-llm pipelines side to launch it by mlc_chat_cli.

Current state:

1. It supports pipeline with and without kv cache. The original model uses the latter case
2. I launch it in mlc-llm-chat in both cases and it answers some reasonable words close to question topic, but it is not expected answer. Looks like it still has some problem with accuracy inside topology
3. It is needed help to rebase this branch to current mlc due to issues with transform passes, param manager and quantization schemes infrastructure

Note: need to merge PR and use new version of TVM to correct work of MPT model

cc @yzh119 @masahi

[yzh119]

Hi @vvchernov , please remove the WIP in the title when you feel the PR is ready.
btw, I'll upstream the TVM native flash attention implementation soon so you don't need to rely on external modules.

[masahi]

Could you give me know when flash attention implementation will be finished and how it can be used on mlc_llm API?

I'm also interested in the TVM-native attention! I want to fuse all of split -> rotary -> attention in llama. Split is needed after combining matmuls in QKV projections. I managed to fuse rotary into split https://github.com/masahi/mlc-llm/blob/cutlass-int8/mlc_llm/transform/fuse_split_rotary_embedding.py and the next step is to fuse them into attention. For now I'm using the cutlass kernel, which is pretty much impossible to modify to support rotary fusion.

I wonder if such fusion is possible in the presence of KV cache update, though.

[masahi]

Just found that the decoder attention kernel in FasterTransformer supports rotary fusion. It also takes KV cache as input.

https://github.com/NVIDIA/FasterTransformer/blob/f8e42aac45815c5be92c0915b12b9a6652386e8c/src/fastertransformer/kernels/decoder_masked_multihead_attention/decoder_masked_multihead_attention_template.hpp#L1363-L1364

[casper-hansen]

@vvchernov How far away is this PR from being ready such that it works with the MPT model family?

[vvchernov]

Hello guys, sorry for the late response. I was transferred to another task (accuracy benchmarking of LLM) and could not finish this one. Now I upstream my last changing in MPT model and mlc-llm pipeline with/without kv cache. It is still required debugging due to low model accuracy and transferring this branch to the top of mlc. It will be great if you help me