-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak after running onnx model numerous times #22271
Comments
If you are running on CPU with languages other than python you can disable MemoryArena: |
### Description Expose enable_mem_arena property for SessionOptions ### Motivation and Context #22271
@yuslepukhin Thank you for your attention on this problem. I have several questions about
|
Memory arena was created to pool GPU allocations as they are much more expensive. However, historically, it was left on by default for CPU scenarios. This is a slub allocator. Memory arena serves only Tensor allocations and nothing else. Typically, OS heap is much more intelligent than arena locking (CUDA workloads usually do not run from multiple threads) and thus may benefit users. The exact perf impact may depend on the model you are running and the environment. In the past we have fixed a few kernels that exhibited heap contention due to inefficiencies, but not many. It does reduce memory footprint. Python environment due to its GC makes matters more complicated. Said that, I will look into your scenario. |
As far as I understand, there are no memory leaks in the main branch version because of #22323 and one can disable memory arena in python. Will this fix be added to the next release ? Which version should I use in my scenario ? |
We are not aware of any memory leaks on Onnxruntime side. However, one issue here, if memory arena is enabled and at least one OrtValue leaks and/or not garbage collected, then that arena is held allocated. I cannot recommend you a specific version, we usually recommend the latest. I think this PR would be in the upcoming release. Depending on the scenario, some customers reported a slight perf improvement and some slight degradation when memory arena is disabled, but nothing major. However, everyone likes the reduced memory consumption. |
I ran the same model via C++ API and there are no leaks. I will check Python side when I get a chance. |
I ran it with It seems that the top offender is uuid module. Memory gains from onnxruntime appear to be minimal and are likely to be GCed. I will check if we are leaking those strings in Pybind. The PID of this script is: 28556 [ Top 10 differences ] [ Top 10 differences ] [ Top 10 differences ] [ Top 10 differences ] [ Top 10 differences ] |
There is a memory leak in PyBind layer. This is only manifested when data is not fed via numpy array. Until it is fixed, the simple workaround is to feed data with numpy array. |
I wrapped the data with numpy array. import uuid
import onnxruntime
import psutil
import numpy
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import StringTensorType
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline
def create_trained_model():
x = [str(uuid.uuid4()) for _ in range(100)]
y = ["a" for _ in range(50)] + ["b" for _ in range(50)]
pipeline = Pipeline(
steps=[
("vectorizer", CountVectorizer()),
("classifier", RandomForestClassifier())
]
)
pipeline.fit(x, y)
return pipeline
def convert_sklearn_model_to_onnx_session(sklearn_model):
onnx_model_proto_string = convert_sklearn(
sklearn_model,
initial_types=[('features', StringTensorType((None,)))],
verbose=False
).SerializeToString()
options = onnxruntime.SessionOptions()
options.enable_cpu_mem_arena = False
return onnxruntime.InferenceSession(
onnx_model_proto_string,
providers=["CPUExecutionProvider"],
sess_options=options
)
def get_one_prediction(onnx_model, input_data):
return onnx_model.run(
[onnx_model.get_outputs()[1].name],
{onnx_model.get_inputs()[0].name: input_data}
)
def get_used_megabytes():
# https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_full_info
# uss (Linux, macOS, Windows): aka “Unique Set Size”,
# this is the memory which is unique to a process and which would be freed
# if the process was terminated right now.
return psutil.Process().memory_full_info().uss / (1024 * 1024)
def run_model_many_times(onnx_model, count, *, dummy):
print("run_model_many_times, dummy = ", dummy)
print("Memory in the beginning = ", get_used_megabytes())
for i in range(count):
input_data = numpy.array([str(uuid.uuid4()) for _ in range(20)])
if not dummy:
get_one_prediction(onnx_model, input_data)
if i % 10000 == 0:
print("Memory in the middle = ", get_used_megabytes())
print("Memory in the end = ", get_used_megabytes())
def main():
sklearn_model = create_trained_model()
onnx_model = convert_sklearn_model_to_onnx_session(sklearn_model)
count = 100000
run_model_many_times(onnx_model, count, dummy=False)
run_model_many_times(onnx_model, count, dummy=True)
if __name__ == '__main__':
main() Code above is producing the following output
|
Describe the issue
In then following scenario I see that memory consumption of python process is continuously growing:
To reproduce
The following code run run_model_many_times function two twice. Once with running onnxruntime.InferenceSession method and once without. While code is running it is printing memory consumption in megabytes, which is close to macOs activity monitor numbers.
In the first case memory consumption is continuously growing and in the second case is not.
Example of printing:
Urgency
On production server python process consumed 100 GB of RAM memory after two months. The only solution is restart the process. So yes, it's urgent.
Platform
Linux
OS Version
both of MacOs 15.0 and Ubuntu 20
ONNX Runtime Installation
Released Package
ONNX Runtime Version or Commit ID
1.19.2
ONNX Runtime API
Python
Architecture
X64
Execution Provider
Default CPU
Execution Provider Library Version
No response
The text was updated successfully, but these errors were encountered: