-
|
After looking at the documentation I assume that an engine cannot be used simultaneously by various threads. But since (in principle), we can load various engines simultaneously, I would like to know if we can load the same engine multiple times for concurrent use. I know this also depends on whether or not the underlying engine also supports this. So I also posted a question in the MXNet discussion group. Has anyone used multiple instances of the other engines for training in the same process? TIA |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
DJL load engine shared library into java process. Once process can only load the library once. So there is one engine in the Java process. For inference, you can load model once, and use multiple threads to run inference concurrently. DJL make sure it's safe for all the engines that running inference in multithreading fashion. Training is completely different use case. Especially form MXNet, for the best performance in training, you have to MXNet threaded engine, which is not thread safe. If you want to run training and at the same time run multithreaded inference, you might crash your JVM. |
Beta Was this translation helpful? Give feedback.
-
|
Multi-process then. |
Beta Was this translation helpful? Give feedback.
DJL load engine shared library into java process. Once process can only load the library once. So there is one engine in the Java process.
For inference, you can load model once, and use multiple threads to run inference concurrently. DJL make sure it's safe for all the engines that running inference in multithreading fashion.
Training is completely different use case. Especially form MXNet, for the best performance in training, you have to MXNet threaded engine, which is not thread safe. If you want to run training and at the same time run multithreaded inference, you might crash your JVM.