New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Upstream codebase diff #470

Draft

kzawora-intel wants to merge 985 commits into main from habana_main

kzawora-intel commented Nov 6, 2024 •

edited

Loading

Scope of changes:

Contiguous PA
Multi-step scheduling
Automatic prefix caching
Padding-aware scheduling/max_num_prefill_seqs
Guided decoding fixes
FP8 support (INC/w8a8/weights_load_device)
ApplyToppTopkScalar sampler optimization
LoRA/MultiLoRA support
FusedMoE support
Model changes (adding mark_steps)
Tests
FakeHPU mode
CI stuff (.jenkins, .github)
Lots of minor stuff (RNG, FSDPA flag, reduced block fragmentation)

github-advanced-security bot found potential problems

View reviewed changes

.github/workflows/cpu-test.yml

		@@ -0,0 +1,35 @@
		name: cpu-test

Check failure

Code scanning / Scorecard

Token-Permissions High

score is 0: no topLevel permission defined
Remediation tip: Visit https://app.stepsecurity.io/secureworkflow.
Tick the 'Restrict permissions for GITHUB_TOKEN'
Untick other options
NOTE: If you want to resolve multiple issues at once, you can visit https://app.stepsecurity.io/securerepo instead.
Click Remediation section below for further remediation help

kzawora-intel marked this pull request as draft

November 6, 2024 13:49

kzawora-intel added the habana label

github-advanced-security bot found potential problems

View reviewed changes

.github/workflows/codespell.yml

		@@ -0,0 +1,45 @@
		name: codespell

Check failure

Code scanning / Scorecard

Token-Permissions High

score is 0: no topLevel permission defined
Remediation tip: Visit https://app.stepsecurity.io/secureworkflow.
Tick the 'Restrict permissions for GITHUB_TOKEN'
Untick other options
NOTE: If you want to resolve multiple issues at once, you can visit https://app.stepsecurity.io/securerepo instead.
Click Remediation section below for further remediation help

github-advanced-security bot found potential problems

View reviewed changes

tests/distributed/test_utils.py

+              def test_stateless_process_group(worker):
+                  port1 = get_open_port()
+                  with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+                      s.bind(("", port1))

Check warning

Code scanning / CodeQL

Binding a socket to all network interfaces Medium test

'' binds a socket to all interfaces.

Copilot Autofix AI 7 days ago

To fix the problem, we need to bind the socket to a specific interface instead of all interfaces. In this case, we can bind it to the loopback interface 127.0.0.1, which is commonly used for local testing and development. This change will limit the socket to accept connections only from the local machine, reducing the security risks.

Suggested changeset 1

tests/distributed/test_utils.py

@@ -124,3 +124,3 @@
                 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
-                    s.bind(("", port1))
+                    s.bind(("127.0.0.1", port1))
                     port2 = get_open_port()

Copilot is powered by AI and may make mistakes. Always verify output.

github-advanced-security bot found potential problems

View reviewed changes

vllm/entrypoints/openai/api_server.py

    
                  sock = socket.socket(family=family, type=socket.SOCK_STREAM)

                  sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

                  sock.bind(addr)

Check warning

Code scanning / CodeQL

Binding a socket to all network interfaces Medium

'' binds a socket to all interfaces.

Copilot Autofix AI 4 days ago

To fix the problem, we need to ensure that the socket binds to a specific interface rather than all interfaces. This can be achieved by modifying the create_server_socket function to check if the provided address is empty or 0.0.0.0 and raise an error or use a default specific interface instead.

Modify the create_server_socket function to validate the address.
If the address is empty or 0.0.0.0, raise an error or use a default specific interface.
Update the run_server function to handle the potential error raised by create_server_socket.

Suggested changeset 1

vllm/entrypoints/openai/api_server.py

@@ -612,2 +612,5 @@
             def create_server_socket(addr: Tuple[str, int]) -> socket.socket:
+                if addr[0] in ("", "0.0.0.0"):
+                    raise ValueError("Binding to all interfaces is not allowed. Please specify a valid IP address.")
                 family = socket.AF_INET
@@ -640,3 +643,7 @@
                 sock_addr = (args.host or "", args.port)
-                sock = create_server_socket(sock_addr)
+                try:
+                    sock = create_server_socket(sock_addr)
+                except ValueError as e:
+                    logger.error(e)
+                    return

Copilot is powered by AI and may make mistakes. Always verify output.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[' and containing many repetitions of 'AA(),'.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(' and containing many repetitions of 'AA=,'.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(A=' and containing many repetitions of ',A='.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(' and containing many repetitions of 'AA= ),A('.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[' and containing many repetitions of 'AA()'.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(' and containing many repetitions of 'AA=,'.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(A=' and containing many repetitions of ',A='.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(' and containing many repetitions of 'AA=)A('.

vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

    
                  # Llama3.2 models more reliable.

                  TOOL_CALL_REGEX = re.compile(

                      r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",

Check failure

Code scanning / CodeQL

Inefficient regular expression High

This part of the regular expression may cause exponential backtracking on strings starting with '[A(A=' and containing many repetitions of ')A(A='.

CNTRYROA and others added 24 commits

November 22, 2024 21:13


          [Model] Fix Baichuan BNB online quantization (vllm-project#10572)

cfea9c0

Signed-off-by: Chen Wu <cntryroa@gmail.com>


          Update default max_num_batch_tokens for chunked prefill to 2048 (vllm…

02a43f8

…-project#10544)


          [AMD] Add support for GGUF quantization on ROCm (vllm-project#10254)

7c25fe4


          Prefix Cache Aware Scheduling [1/n] (vllm-project#10128)

4634a89

Signed-off-by: rickyx <rickyx@anyscale.com>


          [2/N] handling placeholders in merged multi-modal processor (vllm-pro…

c8acd80

…ject#10485)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          [Bugfix] multi_modal_kwargs broadcast for CPU tensor parallel (vllm…

4cfe5d2

…-project#10541)

Signed-off-by: Isotr0py <2037008807@qq.com>


          [Platforms] Refactor openvino code (vllm-project#10573)

86a44fb

Signed-off-by: statelesshz <hzji210@gmail.com>


          For ppc64le, disabled tests for now and addressed space issues (vllm-…

651f6c3

…project#10538)


          [Bugfix] Avoid import AttentionMetadata explicitly in Mllama (vllm-pr…

04668eb

…oject#10593)

Signed-off-by: Isotr0py <2037008807@qq.com>


          [bugfix] Fix example/tensorize_vllm_model tests (vllm-project#10595)

17d8fc1

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>


          [Bugfix] Fix LoRA weight sharding (vllm-project#10450)

1700c54

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>


          [CI/Build] Print running script to enhance CI log readability (vllm-p…

1c445dc

…roject#10594)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>


          Revert "Print running script to enhance CI log readability" (vllm-pro…

eda2b35

…ject#10601)


          [model][utils] add extract_layer_index utility function (vllm-project…

c055747

…#10599)

Signed-off-by: youkaichao <youkaichao@gmail.com>


          [doc] update the code to add models (vllm-project#10603)

e4fbb14

Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>


          [Doc] Update README.md with Ray Summit talk links (vllm-project#10610)

49628fe


          Support Cross encoder models (vllm-project#10400)

214efc2

Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>


          [Refactor][MISC] del redundant code in ParallelConfig.postinit (vllm-…

7ea3cd7

…project#10614)

Signed-off-by: MengqingCao <cmq0113@163.com>


          [torch.compile] support encoder based models (vllm-project#10613)

571841b

Signed-off-by: youkaichao <youkaichao@gmail.com>


          [Doc] Add encoder-based models to Supported Models page (vllm-project…

a30a605

…#10616)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          [torch.compile] force inductor threads (vllm-project#10620)

7c2134b

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>


          [torch.compile] add warning for unsupported models (vllm-project#10622)

Signed-off-by: youkaichao <youkaichao@gmail.com>


          [misc] add torch.compile compatibility check (vllm-project#10618)

25d806e

Signed-off-by: youkaichao <youkaichao@gmail.com>


          Limit decode block size (#532)

39c6b6c

Limit decode bucket size to num_hpu_blocks

DarkLight1337 and others added 20 commits

December 10, 2024 13:45


          [Bugfix] Fix usage of deprecated decorator (vllm-project#11025)

391d7b2

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          [Frontend] Use request id from header (vllm-project#10968)

980ad39

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>


          [Pixtral] Improve loading (vllm-project#11040)

bc192a2


          [V1] Multiprocessing Tensor Parallel Support for v1 (vllm-project#9856)

28b3a1c

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>


          monitor metrics of tokens per step using cudagraph batchsizes (vllm-p…

ebf7780

…roject#11031)

Signed-off-by: youkaichao <youkaichao@gmail.com>


          [Bugfix] Fix xgrammar failing to read a vocab_size from LlavaConfig o…

e35879c

…n PixtralHF. (vllm-project#11043)


          Update README.md (vllm-project#11034)

bfd6104


          [Bugfix] cuda error running llama 3.2 (vllm-project#11047)

82c73fd


          Add example of helm chart for vllm deployment on k8s (vllm-project#9199)

fe2e10c

Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>


          Support mllama (llama 3.2) model for HPU (#491)

239739c


          Merge remote-tracking branch 'upstream/main' into HEAD

2126fd2


          Merge remote-tracking branch 'origin/habana_main' into private/kzawor…

89266bc

…a/dec_10_rebase


          Update ray_hpu_executor.py

5a166da


          Enable padding aware scheduling by default on HPU (#606)

0ad9b59

the title says it all really


          Update CODEOWNERS

17e6be7


          Update CODEOWNERS (#608)

15774c4

We want to start enforcing codeowners on our repo, but without calling
vllm upstream owners for review each time.


          Fix TP>1 in encoder-decoder models (#607)

def7ac2


          Add PunicaWrapperHPU to handle LoRA computations

b8fff21


          Align LoRA handling in HPU with PunicaWrapper class (#614)

381453c

This PR adds `PunicaWrapperHPU` class to handle LoRA computations in
HPU. These changes are to align LoRA flow refactoring done in the
upstream branch.

it's that time of the week again

---------

Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: Sam Stoelinga <sammiestoel@gmail.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: xffxff <1247714429@qq.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Signed-off-by: kevin <kevin@anyscale.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Signed-off-by: Jerzy Zagorski <jzagorsk@amazon.com>
Signed-off-by: Richard Liu <ricliu@google.com>
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: Sam Stoelinga <sammiestoel@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: zhou fan <1247714429@qq.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Kevin H. Luu <kevin@anyscale.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: xendo <xendoo@gmail.com>
Co-authored-by: Jerzy Zagorski <jzagorsk@amazon.com>
Co-authored-by: Richard Liu <39319471+richardsliu@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Jeff Cook <jeff@jeffcook.io>
Co-authored-by: Diego Marinho <dmztheone@gmail.com>
Co-authored-by: Gene Der Su <e870252314@gmail.com>
Co-authored-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
Co-authored-by: Michał Kuligowski <mkuligowski@habana.ai>
Co-authored-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>

github-advanced-security bot found potential problems

View reviewed changes

.github/workflows/lint-and-deploy.yaml Fixed Show fixed Hide fixed

michalkuligowski and others added 7 commits

December 11, 2024 12:51


          Revert "Dec 10 rebase"

641367b


          Revert "Dec 10 rebase" (#618)

55f99ea

Reverts #605


          Revert "Revert "Dec 10 rebase""

ad10b73


          Revert "Revert "Dec 10 rebase"" (#619)

df7dd05

i think inception was a decent movie overall


          fix graceful shutdown

07dbd34

Signed-off-by: Konrad Zawora <kzawora@habana.ai>


          Fix multiprocessing executor shutdown (#621)

d312c92

With this patch, mp executor does not hang at the end of application out
of the box, and exits gracefully.


          Update GitHub Actions targets (#622)

7ef6b2c

New useful checks were added, and we're not running them on habana_main
per PR. This PR fixes that.

github-advanced-security bot found potential problems

View reviewed changes

.github/workflows/sphinx-lint.yml

		@@ -0,0 +1,32 @@
		name: Lint documentation

Check failure

Code scanning / Scorecard

Token-Permissions High

score is 0: no topLevel permission defined
Remediation tip: Visit https://app.stepsecurity.io/secureworkflow.
Tick the 'Restrict permissions for GITHUB_TOKEN'
Untick other options
NOTE: If you want to resolve multiple issues at once, you can visit https://app.stepsecurity.io/securerepo instead.
Click Remediation section below for further remediation help


          Add padding to encoder_seq_lens (#610)

449a89d

Without this change we can observe below error:
```
[rank0]:   File "/software/users/kdamaszke/repos/vllm-fork/vllm/model_executor/models/mllama.py", line 959, in forward
[rank0]:     full_text_row_masked_out_mask = full_text_row_masked_out_mask.view(
[rank0]: RuntimeError: shape '[4, -1, 1]' is invalid for input of size 3
```
It occurs when one of the requests is removed from the batch earlier. In
that case, language model is still working on the shapes padded to the
bucketed batch size, while encoder input doesn't. This change is
aligning the batch size on `encoder_seq_lens` to the expected one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

habana

85 participants