Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for INT64 types in TensorRT constant layer calibration #21041

Closed
wants to merge 3 commits into from
Closed

Add support for INT64 types in TensorRT constant layer calibration #21041

wants to merge 3 commits into from

Conversation

kevinch-nv
Copy link
Contributor

Description

  • TensorRT supports INT64 types since TRT 10, so add support for it when calibrating constant layers.
  • Add more logging statements for helpful debug messages to users when there is an issue reading from a calibration cache.

Motivation and Context

  • This change is necessary for functional parity with TRT 10

@kevinch-nv
Add support for INT64 types in TensorRT constant layer calibration
e6fff47 
Signed-off-by: Kevin Chen <kevinch@nvidia.com>
@chilo-ms
Copy link
Contributor

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Linux QNN CI Pipeline

@chilo-ms
Copy link
Contributor

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows ARM64 QNN CI Pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, ONNX Runtime React Native CI Pipeline, Windows x64 QNN CI Pipeline

@chilo-ms
Copy link
Contributor

/azp run Linux MIGraphX CI Pipeline, orttraining-amd-gpu-ci-pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

1 similar comment
@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@chilo-ms
Copy link
Contributor

/azp run Big Models, Linux CPU Minimal Build E2E CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@chilo-ms
Copy link
Contributor

Please help fix Lint / Python format

@kevinch-nv
Fix comment whitespace
ed1a45e 
Signed-off-by: Kevin Chen <kevinch@nvidia.com>
@chilo-ms
Copy link
Contributor

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Linux QNN CI Pipeline

@chilo-ms
Copy link
Contributor

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows ARM64 QNN CI Pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, ONNX Runtime React Native CI Pipeline, Windows x64 QNN CI Pipeline

@chilo-ms
Copy link
Contributor

/azp run Big Models, Linux CPU Minimal Build E2E CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@chilo-ms
Copy link
Contributor

/azp run Linux Android Emulator QNN CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

chilo-ms
chilo-ms previously approved these changes Jun 14, 2024
View reviewed changes
chilo-ms
chilo-ms reviewed Jun 14, 2024
View reviewed changes
onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc Outdated
@@ -108,13 +111,19 @@ bool SetDynamicRange(nvinfer1::INetworkDefinition& network, std::unordered_map<s
case nvinfer1::DataType::kINT32:
weight = static_cast<const int32_t*>(trt_weights.values)[k];
break;
#if NV_TENSORRT_MAJOR >= 10
case nvinfer1::DataType::kINT64:
weight = static_cast<const int64_t*>(trt_weights.values)[k];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converting "const int64_t" to "double" may lost data.

warning C4244: '=': conversion from 'const int64_t' to 'double', possible loss of data

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevinch-nv, fyi this is causing the build to fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@chilo-ms
Copy link
Contributor

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Linux QNN CI Pipeline

@chilo-ms
Copy link
Contributor

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows ARM64 QNN CI Pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, ONNX Runtime React Native CI Pipeline, Windows x64 QNN CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

1 similar comment
@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@chilo-ms
Copy link
Contributor

/azp run Linux MIGraphX CI Pipeline, orttraining-amd-gpu-ci-pipeline Big Models, Linux CPU Minimal Build E2E CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@chilo-ms
Copy link
Contributor

/azp run Big Models, Linux Android Emulator QNN CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@kevinch-nv
Fix warning
521fffa 
Signed-off-by: Kevin Chen <kevinch@nvidia.com>
@chilo-ms chilo-ms mentioned this pull request Jun 19, 2024
Merged
@chilo-ms
Copy link
Contributor

Please help fix the format.

BTW, i create a duplicated PR in case this PR can't be updated for patch release timeline.

@jywu-msft
Copy link
Member

Please help fix the format.

BTW, i create a duplicated PR in case this PR can't be updated for patch release timeline.

#21101

jywu-msft pushed a commit that referenced this pull request Jun 20, 2024
@chilo-ms
Add support for INT64 types in TensorRT constant layer calibration (#…
e737547 
…21101)

This PR is a duplicate of the
#21041
Create this PR in case the original one can't be updated for patch
release timeline.
yf711 pushed a commit that referenced this pull request Jun 21, 2024
@chilo-ms @yf711
Add support for INT64 types in TensorRT constant layer calibration (#…
57475ff 
…21101)

This PR is a duplicate of the
#21041
Create this PR in case the original one can't be updated for patch
release timeline.
@jywu-msft
Copy link
Member

superseded by #21101

@jywu-msft jywu-msft closed this Jun 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chilo-ms chilo-ms chilo-ms left review comments

@jywu-msft jywu-msft jywu-msft left review comments

Assignees
No one assigned
Labels
release:1.18.1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kevinch-nv @chilo-ms @jywu-msft @yf711