Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TensorRT EP] Weightless API integration #20412

Merged
merged 124 commits into from
May 26, 2024
Merged
Changes from 1 commit
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
89f6d75
Init new yml & dockerfile to update TRT CI
yf711 Mar 11, 2024
6006682
update
yf711 Mar 14, 2024
71c817d
update
yf711 Mar 14, 2024
9d755df
add cuda 12.4 support
yf711 Mar 15, 2024
f42505f
Update win/linux trt yml to cu123 and latest trt
yf711 Mar 18, 2024
9dc2990
test trt CIs with 10.0.0.2
yf711 Mar 18, 2024
aca88a9
Update win trt ver for EA
yf711 Mar 18, 2024
7212de7
fix
yf711 Mar 19, 2024
f830fed
fix
yf711 Mar 20, 2024
8fc5dd2
fix
yf711 Mar 20, 2024
040d27f
update
yf711 Mar 20, 2024
decbb47
fix
yf711 Mar 20, 2024
5815bd6
Make TRT EP supports INT64 for TRT 10
chilo-ms Mar 22, 2024
4166b83
Fix compile warning
chilo-ms Mar 25, 2024
f373f67
merge main
yf711 Mar 27, 2024
494c970
update
yf711 Mar 27, 2024
adb4d3c
update
yf711 Mar 27, 2024
0959856
clean
yf711 Mar 27, 2024
b188108
update ep perf ci dockerfile
yf711 Mar 27, 2024
295dd33
update
yf711 Mar 27, 2024
dfdc36a
update linux trt ci dockerfile for new trt10
yf711 Mar 29, 2024
eef015d
update ep perf ci dockerfile with latest trt10
yf711 Mar 29, 2024
8cb808d
Merge
yf711 Apr 1, 2024
3d3a604
switch condition of linux trt ci dockerfiles
yf711 Apr 1, 2024
9ab0f41
temp fix
yf711 Apr 1, 2024
b59aa8a
fix on ep perf ci dockerfiles
yf711 Apr 1, 2024
7234573
fix
yf711 Apr 1, 2024
5157df7
update on ep perf trt bin dockerfile
yf711 Apr 1, 2024
fdde93a
debug
yf711 Apr 2, 2024
33f36cc
test
yf711 Apr 2, 2024
652b27d
fix
yf711 Apr 2, 2024
a233e86
disable trtexec
yf711 Apr 2, 2024
15054fe
update onnx-tensorrt to 10.0-EA
yf711 Apr 2, 2024
a420e73
Merge branch 'yifanl/trtep_update_ci_dockerfile' into yifanl/chi_trt1…
yf711 Apr 2, 2024
6988df8
revert
yf711 Apr 2, 2024
3ef6fa8
revert
yf711 Apr 2, 2024
d2d7e90
Merge branch 'main' into yifanl/trtep_update_ci_dockerfile
yf711 Apr 2, 2024
bf70e3e
Fix py package pipeline (#20065)
wangyems Mar 27, 2024
40cdbfa
fix
yf711 Apr 2, 2024
d2edf5b
fix
yf711 Apr 2, 2024
cb8ece1
fix
yf711 Apr 2, 2024
d1f2af9
fix
yf711 Apr 2, 2024
133e05c
fix
yf711 Apr 2, 2024
91b7091
slim test
yf711 Apr 2, 2024
75531b6
fix
yf711 Apr 2, 2024
1714675
test win trt ci with trt10-cu118
yf711 Apr 3, 2024
9084e85
Merge branch 'main' into yifanl/chi_trt10
yf711 Apr 3, 2024
d7dd1e3
Merge branch 'main' into yifanl/trtep_update_ci_dockerfile
yf711 Apr 3, 2024
5622cba
set default, revert extra changes
yf711 Apr 3, 2024
7dcac7e
update setup_env_trt.bat
yf711 Apr 3, 2024
6cc9068
Merge branch 'yifanl/trtep_update_ci_dockerfile' into yifanl/chi_trt1…
yf711 Apr 3, 2024
1e6efba
test skipping failed tests
yf711 Apr 4, 2024
5957f14
update onnx-tensorrt to 10.0-EA
yf711 Apr 2, 2024
fb4443f
fix
yf711 Apr 4, 2024
2420716
update skipped tests
yf711 Apr 8, 2024
af5851a
update skipped test
yf711 Apr 8, 2024
912c14c
test
yf711 Apr 8, 2024
52eba2a
test
yf711 Apr 8, 2024
d243fb1
test
yf711 Apr 9, 2024
fb4d491
test
yf711 Apr 10, 2024
ea62afa
test ubi8-cuda12.4
yf711 Apr 10, 2024
4077b29
test ubi8-cuda12.4
yf711 Apr 10, 2024
f863251
fix
yf711 Apr 10, 2024
e75de20
fix
yf711 Apr 10, 2024
ed2df37
test
yf711 Apr 10, 2024
d532715
test cuda11.8-trt10
yf711 Apr 10, 2024
4ba8b55
version correction
yf711 Apr 10, 2024
385c155
test
yf711 Apr 10, 2024
2a3dad2
test ep perf with cuda 12.4 dockerenv
yf711 Apr 10, 2024
f0392bf
fix
yf711 Apr 10, 2024
eca26ec
fix in yml
yf711 Apr 10, 2024
1536468
test filter
yf711 Apr 11, 2024
75eb7aa
nit
yf711 Apr 11, 2024
918cbd4
revert
yf711 Apr 12, 2024
7a3f44e
test
yf711 Apr 12, 2024
c20d295
Merge main into "yifanl/chi_trt10_cuda12"
yf711 Apr 12, 2024
9ecafcf
test
yf711 Apr 15, 2024
55e28e2
update
yf711 Apr 17, 2024
8187f5e
Merge branch 'yifanl/debug_resnet50' into yifanl/chi_trt10_cuda12
yf711 Apr 17, 2024
d8aa5c3
Merge branch 'main' into yifanl/chi_trt10_cuda12
yf711 Apr 17, 2024
e37ccf5
Merge branch 'main' into yifanl/chi_trt10_cuda12
yf711 Apr 18, 2024
7f3f16e
Fix to EP Perf when choosing OSS parser
yf711 Apr 18, 2024
639889b
enable dds ops with trt10ea
yf711 Apr 19, 2024
d51c9d6
fix on trtexec
yf711 Apr 19, 2024
100c9f9
Compile trtexec only if not installed
yf711 Apr 19, 2024
ada40d4
merge main
chilo-ms Apr 19, 2024
1638e94
TensorRT EP: Weightless API integration in ONNX Runtime (#20214)
moraxu Apr 22, 2024
1d5648a
Merge branch 'main' into yifanl/chi_trt10+dockerfile
chilo-ms Apr 22, 2024
01d5835
update onnx
chilo-ms Apr 22, 2024
c5980eb
win-trt10ga
yf711 Apr 24, 2024
5ec8203
onnx-tensorrt 10.0-GA
yf711 Apr 26, 2024
00b5e35
Merge branch 'yifanl/chi_trt10_cuda12' into yifanl/chi_trt10+dockerfile
yf711 Apr 26, 2024
2e92a7a
Revert "onnx-tensorrt 10.0-GA"
yf711 Apr 26, 2024
e0a122b
revert
yf711 Apr 26, 2024
1567d86
update
yf711 Apr 26, 2024
f145d79
10.0-GA
yf711 Apr 26, 2024
af2694f
revert ubi8 dockerfile
yf711 Apr 26, 2024
077e98a
fix compile error
chilo-ms Apr 29, 2024
9968c84
TRT10 GA
yf711 Apr 27, 2024
eec1942
filter tests
yf711 Apr 27, 2024
33c4136
update for GetTensorrtLogger
chilo-ms Apr 29, 2024
6841dce
Merge branch 'main' into yifanl/chi_trt10+dockerfile
yf711 May 1, 2024
bab40ac
Merge main
yf711 May 1, 2024
2bc5b87
revert
yf711 May 1, 2024
9d88daf
lintrunner -a
chilo-ms May 1, 2024
bf8e6bc
change naming from weightless to weight-stripped
chilo-ms May 23, 2024
b711ed7
Merge branch 'main' into yifanl/chi_trt10+dockerfile
chilo-ms May 23, 2024
cd5eba2
rename cache for weight-stripped engine
chilo-ms May 23, 2024
89c8b0f
serialize refitted engine
chilo-ms May 23, 2024
fba65ae
engine refit for non quick load path as well
chilo-ms May 24, 2024
102addf
remove commented code
chilo-ms May 24, 2024
421e59c
lintrunner -a
chilo-ms May 24, 2024
d666ce7
minor update
chilo-ms May 24, 2024
70fc577
add more comments
chilo-ms May 24, 2024
a5d2085
update and modify per reviewer's comment
chilo-ms May 24, 2024
b6c275d
update contrib op doc
chilo-ms May 24, 2024
38dafc1
refactor
chilo-ms May 24, 2024
ddc2ec8
modify contrib op doc
chilo-ms May 25, 2024
39526f1
code refactor
chilo-ms May 25, 2024
db2fce6
fix format
jywu-msft May 25, 2024
a8b9662
Check weight-stripped engine cache automatically in the case EPContex…
chilo-ms May 25, 2024
14765fa
add some verbose logging
jywu-msft May 26, 2024
aad9f86
Add comments and change function name per reviewer's comment
chilo-ms May 26, 2024
8284c8c
fix compiler error
chilo-ms May 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update win trt ver for EA
  • Loading branch information
yf711 committed Mar 18, 2024
commit aca88a97b6dfdbcf0984645d12f560fa03bbe94b
Original file line number Diff line number Diff line change
@@ -233,7 +233,7 @@ stages:
buildArch: x64
msbuildPlatform: x64
packageName: x64-tensorrt
buildparameter: --use_tensorrt --tensorrt_home="$(Agent.TempDirectory)\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --enable_onnx_tests --enable_wcos --build_java --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=60;61;70;75;80"
buildparameter: --use_tensorrt --tensorrt_home="$(Agent.TempDirectory)\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --enable_onnx_tests --enable_wcos --build_java --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=60;61;70;75;80"
runTests: ${{ parameters.RunOnnxRuntimeTests }}
buildJava: true
java_artifact_id: onnxruntime_gpu
Original file line number Diff line number Diff line change
@@ -83,7 +83,7 @@ variables:
${{ if eq(parameters.CudaVersion, '12.2') }}:
value: $(Agent.TempDirectory)\TensorRT-8.6.1.6.Windows10.x86_64.cuda-12.0
${{ if eq(parameters.CudaVersion, '12.3') }}:
value: $(Agent.TempDirectory)\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4
value: $(Agent.TempDirectory)\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4
- name: win_cuda_home
${{ if eq(parameters.CudaVersion, '11.8') }}:
value: $(Agent.TempDirectory)\v11.8
2 changes: 1 addition & 1 deletion tools/ci_build/github/azure-pipelines/post-merge-jobs.yml
Original file line number Diff line number Diff line change
@@ -271,7 +271,7 @@ stages:
BuildConfig: 'RelWithDebInfo'
EnvSetupScript: setup_env_trt.bat
buildArch: x64
additionalBuildFlags: --enable_pybind --build_java --build_nodejs --use_cuda --cuda_home="$(Agent.TempDirectory)\v12.3" --enable_cuda_profiling --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=86
additionalBuildFlags: --enable_pybind --build_java --build_nodejs --use_cuda --cuda_home="$(Agent.TempDirectory)\v12.3" --enable_cuda_profiling --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=86
msbuildPlatform: x64
isX86: false
job_name_suffix: x64_RelWithDebInfo
Original file line number Diff line number Diff line change
@@ -59,7 +59,7 @@ stages:
${{ if eq(parameters.cuda_version, '12.2') }}:
value: $(Agent.TempDirectory)\TensorRT-8.6.1.6.Windows10.x86_64.cuda-12.0
${{ if eq(parameters.cuda_version, '12.3') }}:
value: $(Agent.TempDirectory)\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4
value: $(Agent.TempDirectory)\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4
- name: win_cuda_home
${{ if eq(parameters.cuda_version, '11.8') }}:
value: $(Agent.TempDirectory)\v11.8
Original file line number Diff line number Diff line change
@@ -392,7 +392,7 @@ stages:
variables:
CUDA_VERSION: '12.3'
buildArch: x64
EpBuildFlags: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_version=$(CUDA_VERSION) --cuda_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v$(CUDA_VERSION)" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=37;50;52;60;61;70;75;80"
EpBuildFlags: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_version=$(CUDA_VERSION) --cuda_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v$(CUDA_VERSION)" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=37;50;52;60;61;70;75;80"
EnvSetupScript: setup_env_gpu.bat
EP_NAME: gpu
VSGenerator: 'Visual Studio 17 2022'
Original file line number Diff line number Diff line change
@@ -280,39 +280,39 @@ stages:
parameters:
MACHINE_POOL: 'onnxruntime-Win2022-GPU-T4'
PYTHON_VERSION: '3.8'
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
ENV_SETUP_SCRIPT: setup_env_gpu.bat
EP_NAME: gpu

- template: py-win-gpu.yml
parameters:
MACHINE_POOL: 'onnxruntime-Win2022-GPU-T4'
PYTHON_VERSION: '3.9'
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
ENV_SETUP_SCRIPT: setup_env_gpu.bat
EP_NAME: gpu

- template: py-win-gpu.yml
parameters:
MACHINE_POOL: 'onnxruntime-Win2022-GPU-T4'
PYTHON_VERSION: '3.10'
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
ENV_SETUP_SCRIPT: setup_env_gpu.bat
EP_NAME: gpu

- template: py-win-gpu.yml
parameters:
MACHINE_POOL: 'onnxruntime-Win2022-GPU-T4'
PYTHON_VERSION: '3.11'
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
ENV_SETUP_SCRIPT: setup_env_gpu.bat
EP_NAME: gpu

- template: py-win-gpu.yml
parameters:
MACHINE_POOL: 'onnxruntime-Win2022-GPU-T4'
PYTHON_VERSION: '3.12'
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
EP_BUILD_FLAGS: --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
ENV_SETUP_SCRIPT: setup_env_gpu.bat
EP_NAME: gpu

Original file line number Diff line number Diff line change
@@ -54,7 +54,7 @@ jobs:
WithCache: True
Today: $(TODAY)
AdditionalKey: "gpu-tensorrt | RelWithDebInfo"
BuildPyArguments: '--config RelWithDebInfo --parallel --use_binskim_compliant_compile_flags --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=75'
BuildPyArguments: '--config RelWithDebInfo --parallel --use_binskim_compliant_compile_flags --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=75'
MsbuildArguments: $(MsbuildArguments)
BuildArch: 'x64'
Platform: 'x64'
@@ -74,7 +74,7 @@ jobs:
del wheel_filename_file
python.exe -m pip install -q --upgrade %WHEEL_FILENAME%
set PATH=$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo;%PATH%
python $(Build.SourcesDirectory)\tools\ci_build\build.py --config RelWithDebInfo --use_binskim_compliant_compile_flags --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --test --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=75
python $(Build.SourcesDirectory)\tools\ci_build\build.py --config RelWithDebInfo --use_binskim_compliant_compile_flags --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --test --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4" --cuda_home="$(Agent.TempDirectory)\v12.3" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=75

workingDirectory: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo'
displayName: 'Run tests'
2 changes: 1 addition & 1 deletion tools/ci_build/github/windows/setup_env_gpu.bat
Original file line number Diff line number Diff line change
@@ -17,7 +17,7 @@ if exist PATH=%AGENT_TEMPDIRECTORY%\v12.2\ (
)

@REM Add cuda v12.3 to PATH after setting cuda 11.8/12.2
set PATH=%PATH%;C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4\lib
set PATH=%PATH%;C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4\lib
if exist PATH=%AGENT_TEMPDIRECTORY%\v12.3\ (
set PATH=%PATH%;%AGENT_TEMPDIRECTORY%\v12.3\bin;%AGENT_TEMPDIRECTORY%\v12.3\extras\CUPTI\lib64
) else (
2 changes: 1 addition & 1 deletion tools/ci_build/github/windows/setup_env_trt.bat
Original file line number Diff line number Diff line change
@@ -6,6 +6,6 @@ if exist PATH=%AGENT_TEMPDIRECTORY%\v12.3\ {
} else {
set PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\extras\CUPTI\lib64;%PATH%
}
set PATH=C:\local\TensorRT-10.0.0.2.Windows10.x86_64.cuda-12.4\lib;%PATH%
set PATH=C:\local\TensorRT-10.0.0.6.Windows10.x86_64.cuda-12.4\lib;%PATH%
set GRADLE_OPTS=-Dorg.gradle.daemon=false
set CUDA_MODULE_LOADING=LAZY