Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Support loading GGUF model #5191

Merged
merged 76 commits into from
Aug 5, 2024
Merged

[Core] Support loading GGUF model #5191

merged 76 commits into from
Aug 5, 2024

Commits on Jun 2, 2024

  1. init gguf loading support

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    1ffda2e View commit details
    Browse the repository at this point in the history

  2. add gguf running support

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    f3058b1 View commit details
    Browse the repository at this point in the history

  3. Fix numpy warning

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    259d5b5 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/main' into gguf

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    0035bdf View commit details
    Browse the repository at this point in the history

  5. fix gguf load format

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    995f98e View commit details
    Browse the repository at this point in the history

  6. add more example prompts

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    d116f2e View commit details
    Browse the repository at this point in the history

  7. update requirements.txt

    @Isotr0py
    Isotr0py committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    f387f9e View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. add dequant runtime

    @Isotr0py
    Isotr0py committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    516552a View commit details
    Browse the repository at this point in the history

  2. remove debug code

    @Isotr0py
    Isotr0py committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    de5950d View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. format code

    @Isotr0py
    Isotr0py committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    5bda5f0 View commit details
    Browse the repository at this point in the history

  2. update gguf example

    @Isotr0py
    Isotr0py committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    980c018 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into gguf

    @Isotr0py
    Isotr0py authored Jun 4, 2024
    Configuration menu
    Copy the full SHA
    f969b36 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Merge branch 'vllm-project:main' into gguf

    @Isotr0py
    Isotr0py authored Jun 5, 2024
    Configuration menu
    Copy the full SHA
    e99f521 View commit details
    Browse the repository at this point in the history

  2. Fix requirements.txt

    @Isotr0py
    Isotr0py committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    9d36996 View commit details
    Browse the repository at this point in the history

  3. rename ggml -> gguf

    @Isotr0py
    Isotr0py committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    3a18502 View commit details
    Browse the repository at this point in the history

  4. auto detect gguf quant and format

    @Isotr0py
    Isotr0py committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    e194e28 View commit details
    Browse the repository at this point in the history

  5. use autotokenizer to load gguf tokenizer

    @Isotr0py
    Isotr0py committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    164b643 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Add runtime dequantization for all layers

    @Isotr0py
    Isotr0py committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    b055fb3 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge branch 'main' into gguf

    @Isotr0py
    Isotr0py authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    c93c44e View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. port gguf cuda kernel

    @Isotr0py
    Isotr0py committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    8960270 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. add qwen2 support and gguf mmq for linear

    @Isotr0py
    Isotr0py committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    1d0c6a4 View commit details
    Browse the repository at this point in the history

  2. remove transformers load_dequant_gguf_tensor

    @Isotr0py
    Isotr0py committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    957faec View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. reorder gguf weight iterator

    @Isotr0py
    Isotr0py committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    4555cf5 View commit details
    Browse the repository at this point in the history

  2. fix imatrix

    @Isotr0py
    Isotr0py committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    7f7af2b View commit details
    Browse the repository at this point in the history

  3. fix imatrix

    @Isotr0py
    Isotr0py committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    87078be View commit details
    Browse the repository at this point in the history

  4. refactor, fix column parallel

    @Isotr0py
    Isotr0py committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    ca39edf View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. refactor gguf_kernel and remove dmmv

    @Isotr0py
    Isotr0py committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    cf03757 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. refactor to unmerge weights for gguf

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    c2524a8 View commit details
    Browse the repository at this point in the history

  2. revert get_quantization_config

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    446c64a View commit details
    Browse the repository at this point in the history

  3. revert get_quantization_config

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    dc43654 View commit details
    Browse the repository at this point in the history

  4. revert qwen2

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    2861670 View commit details
    Browse the repository at this point in the history

  5. add quant vocal embeddings

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    1622966 View commit details
    Browse the repository at this point in the history

  6. support quantized parallelhead

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    c4d4f96 View commit details
    Browse the repository at this point in the history

  7. revert qwen2

    @Isotr0py
    Isotr0py committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    9a99252 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Merge remote-tracking branch 'upstream/main' into gguf

    @Isotr0py
    Isotr0py committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    bc1ab48 View commit details
    Browse the repository at this point in the history

  2. rebase gguf support

    @Isotr0py
    Isotr0py committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    3fad5bd View commit details
    Browse the repository at this point in the history

  3. format code

    @Isotr0py
    Isotr0py committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    409bed3 View commit details
    Browse the repository at this point in the history

  4. format code

    @Isotr0py
    Isotr0py committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    b38bd1d View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. support qwen2 gguf

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    3586f12 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into gguf

    @Isotr0py
    Isotr0py authored Jul 4, 2024
    Configuration menu
    Copy the full SHA
    8a56d55 View commit details
    Browse the repository at this point in the history

  3. fix gguf loader

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    defe23f View commit details
    Browse the repository at this point in the history

  4. add gguf test

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    6c4300e View commit details
    Browse the repository at this point in the history

  5. format code

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    266447b View commit details
    Browse the repository at this point in the history

  6. format code

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    d5a7e2f View commit details
    Browse the repository at this point in the history

  7. remove archs<7.0 in cmakelists

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    6026e02 View commit details
    Browse the repository at this point in the history

  8. fix a typo

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    9dc8794 View commit details
    Browse the repository at this point in the history

  9. format code

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    ef9b8a3 View commit details
    Browse the repository at this point in the history

  10. format code

    @Isotr0py
    Isotr0py committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    b708ce6 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. fix failed model test

    @Isotr0py
    Isotr0py committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    be51a27 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2024

  1. Merge branch 'vllm-project:main' into gguf

    @Isotr0py
    Isotr0py authored Jul 7, 2024
    Configuration menu
    Copy the full SHA
    1bd7d16 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Merge branch 'main' into gguf

    @Isotr0py
    Isotr0py authored Jul 10, 2024
    Configuration menu
    Copy the full SHA
    c155f74 View commit details
    Browse the repository at this point in the history

  2. add imatrix and qwen2 test

    @Isotr0py
    Isotr0py committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    e49f96e View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. reorganize gguf kernel

    @Isotr0py
    Isotr0py committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    af0c051 View commit details
    Browse the repository at this point in the history

  2. exclude gguf copied code

    @Isotr0py
    Isotr0py committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    0ce3961 View commit details
    Browse the repository at this point in the history

  3. refactor to merge weights

    @Isotr0py
    Isotr0py committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e599b07 View commit details
    Browse the repository at this point in the history

  4. forma code

    @Isotr0py
    Isotr0py committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    25dcc08 View commit details
    Browse the repository at this point in the history

  5. format code

    @Isotr0py
    Isotr0py committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    eed9a23 View commit details
    Browse the repository at this point in the history

  6. import gguf

    @Isotr0py
    Isotr0py committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    6e5330d View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2024

  1. import gguf

    @Isotr0py
    Isotr0py committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    e5a61be View commit details
    Browse the repository at this point in the history

  2. refactor quantized vocal embedding

    @Isotr0py
    Isotr0py committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    64c5375 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2024

  1. optimize docs

    @Isotr0py
    Isotr0py committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    86ef2b5 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. add docs

    @Isotr0py
    Isotr0py committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    7ccfacb View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into gguf

    @Isotr0py
    Isotr0py committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    28dc7b6 View commit details
    Browse the repository at this point in the history

  3. fix llama embed quant

    @Isotr0py
    Isotr0py committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    1b39fbc View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Fix CUDA graph with gguf

    @Isotr0py
    Isotr0py committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    d413f60 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2024

  1. Merge remote-tracking branch 'upstream/main' into gguf

    @Isotr0py
    Isotr0py committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    1868a94 View commit details
    Browse the repository at this point in the history

  2. fix quant embeddings

    @Isotr0py
    Isotr0py committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    b4e2f29 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Merge branch 'main' into gguf

    @mgoin
    mgoin authored Jul 31, 2024
    Configuration menu
    Copy the full SHA
    2cc6753 View commit details
    Browse the repository at this point in the history

  2. Fix embedding method and format

    @mgoin
    mgoin committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    db54a19 View commit details
    Browse the repository at this point in the history

  3. Cleanup linear comments

    @mgoin
    mgoin committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    2549c3e View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. move gguf to cuda requirements

    @Isotr0py
    Isotr0py committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    0890fa9 View commit details
    Browse the repository at this point in the history

  2. raise error for gguf when tp>1

    @Isotr0py
    Isotr0py committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    5166ac9 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge branch 'main' into gguf

    @mgoin
    mgoin authored Aug 5, 2024
    Configuration menu
    Copy the full SHA
    26349db View commit details
    Browse the repository at this point in the history

  2. Last round of cleanup

    @mgoin
    mgoin committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    73da240 View commit details
    Browse the repository at this point in the history

  3. Improve qweight_type size calc

    @mgoin
    mgoin committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    1c83d63 View commit details
    Browse the repository at this point in the history

  4. Fix lm head tests

    @mgoin
    mgoin committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    1139e7b View commit details
    Browse the repository at this point in the history