Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: tiktoken integration #60

Merged
merged 38 commits into from
Jul 24, 2024
Merged

feat: tiktoken integration #60

merged 38 commits into from
Jul 24, 2024

Commits on Jul 17, 2024

  1. cpp-tiktoken integration init

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    26847a9 View commit details
    Browse the repository at this point in the history

  2. remove cpp-tiktoken folder

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    96dd32f View commit details
    Browse the repository at this point in the history

  3. tiktoken integration init

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    8e9983c View commit details
    Browse the repository at this point in the history

  4. move fmt to third-party dependency

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    4f0fc5d View commit details
    Browse the repository at this point in the history

  5. remove unnecessary option in cortex CMakeLists

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    06effaf View commit details
    Browse the repository at this point in the history

  6. remove unnecessary submodule

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    f4a6703 View commit details
    Browse the repository at this point in the history

  7. make pcre2 as third-party lib

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    c7c6030 View commit details
    Browse the repository at this point in the history

  8. fix CI bug build in window

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    06f323d View commit details
    Browse the repository at this point in the history

  9. fix CI bug in linux

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    3077181 View commit details
    Browse the repository at this point in the history

  10. link directory pcre2

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    5a4a79c View commit details
    Browse the repository at this point in the history

  11. Fix CI bug in linux 'pcre2.h' not found

    @sangjanai
    sangjanai committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    2e059b0 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Fix bug build static third party lib

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    9c106fa View commit details
    Browse the repository at this point in the history

  2. Integate Llama3 successfully

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    1044ea2 View commit details
    Browse the repository at this point in the history

  3. refactor with model type

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    7c4eb94 View commit details
    Browse the repository at this point in the history

  4. remove unnecessary comment

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    9bb8fca View commit details
    Browse the repository at this point in the history

  5. format and code convention

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    406fb57 View commit details
    Browse the repository at this point in the history

  6. fix logic of mistral when preparing input

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    4985a45 View commit details
    Browse the repository at this point in the history

  7. Add remove end_of_text and end_of_turn token feature for llama3

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    3ab27be View commit details
    Browse the repository at this point in the history

  8. fix: patch pcre2 CMakeLists.txt file

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    26f0557 View commit details
    Browse the repository at this point in the history

  9. fix: do not add fPIC to cpp-tiktoken cmake file

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    d7a6dea View commit details
    Browse the repository at this point in the history

  10. fix: suppress warnings

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    6cec922 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. fix: add CMAKE_C_FLAGS

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    58afe8f View commit details
    Browse the repository at this point in the history

  2. fix: CI bug build for window in cpp-tiktoken

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    63b7484 View commit details
    Browse the repository at this point in the history

  3. fix: CI bug for window fmt build fail without utf-8 flag

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    7d807d2 View commit details
    Browse the repository at this point in the history

  4. fix: CI bug for window stop using fmt

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    2f48d96 View commit details
    Browse the repository at this point in the history

  5. fix: build bugs for window - using latest pcre2 third party

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    6772650 View commit details
    Browse the repository at this point in the history

  6. fix: build bugs for window - pcre2 static runtime on

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    654c9e9 View commit details
    Browse the repository at this point in the history

  7. fix CI bug for window build pcre2

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    95679f7 View commit details
    Browse the repository at this point in the history

  8. rename link lib to pcre2-8-static in windows

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    4521928 View commit details
    Browse the repository at this point in the history

  9. fix: use pcre2 dynamic lib for windows

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    3bb3aa0 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. fix: pack tensorrt_llm_nvrtc_wrapper.dll

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    a3f66a7 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. test: only windows

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    27ab0be View commit details
    Browse the repository at this point in the history

  2. fix: disable setLoggerFinder for msvc

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    d1b8392 View commit details
    Browse the repository at this point in the history

  3. fix: cleanup cmake files

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    11e2925 View commit details
    Browse the repository at this point in the history

  4. fix: remove fmt

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    d634357 View commit details
    Browse the repository at this point in the history

  5. fix: pack libtensorrt_llm_nvrtc_wrapper

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    f82af99 View commit details
    Browse the repository at this point in the history

  6. fix: patches windows

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    358955c View commit details
    Browse the repository at this point in the history

  7. fix: rm fmt

    @nguyenhoangthuan99
    nguyenhoangthuan99 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    b4ddb21 View commit details
    Browse the repository at this point in the history