Support Mixtral quantization using INC

[dudilester]

No description provided.

[dudilester]

jenkins vllm-perf run of this branch:
https://tf-jenkins-ctrl01.habana-labs.com/job/static-benchmarks/job/vllm-benchmarks/185/
functionality passed, acc is at 95% (will test combinations of enabling quantization/high-precision of different layers to achieve 99% acc, however this does not require any code change only run-time configuration change so should not impact code merging)

[szutenberg]

Hi @dudilester ,

Your changes look fine but the testing is insufficient.

• your branch does not contain the latest flat_pa changes Port flat PA from habana_next to habana_main #169 . Please rebase your branch and rerun tests.
• gerrit change used in your tests installs custom inc binary. Please make sure that all required changes are merged and do not install any custom binaries.

Thanks!

[dudilester]

Hi @szutenberg,

Thank you for the review.
I've rebased the branch to include latest habana_main commits and removed the INC installation from the patch to use the regular INC of build 1.18.0.395.

https://tf-jenkins-ctrl01.habana-labs.com/job/static-benchmarks/job/vllm-benchmarks/195/

{"mlperf-moe/mixtral-8x7b-tp1-fp8": {"rouge1": 45.9205, "rouge2": 23.7295, "rougeL": 30.8274, "rougeLsum": 42.9159, "gsm8k": 72.66, "mbxp": 57.58, "gen_len": 4163594, "gen_num": 15000, "gen_tok_len": 2132769, "tokens_per_sample": 142.2, "accuracy": 95.71, "valid": true, "warmup_duration": 512.0, "test_duration": 4143.0, "eval_duration": 243.0}}

[szutenberg]

LGTM now. Thanks @dudilester 👍