You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we expect all deployed models to be a function from a list of inputs to a list of outputs. This allows models the ability to do batching, but it makes the function signature of models more complicated. For simple models that want to operate on a single-input at a time, users have to remember to wrap the input in for-loop.
Instead, we should support deploying two types of models: batch and single-input. Batch models are still given the full available batch at once. Single-input models would be a function from a single input to a single output, and the RPC server embedded in the model container would call the model in a for-loop transparently to the user.
This should make the model deployment process simpler in many common cases.
The text was updated successfully, but these errors were encountered:
Based on discussion in #395, we decided that rather than supporting single-input mode, we should improve our documentation and debugging capabilities to help users use the batching functionality in Clipper correctly. #394 is a good first step in that direction.
Currently, we expect all deployed models to be a function from a list of inputs to a list of outputs. This allows models the ability to do batching, but it makes the function signature of models more complicated. For simple models that want to operate on a single-input at a time, users have to remember to wrap the input in for-loop.
Instead, we should support deploying two types of models: batch and single-input. Batch models are still given the full available batch at once. Single-input models would be a function from a single input to a single output, and the RPC server embedded in the model container would call the model in a for-loop transparently to the user.
This should make the model deployment process simpler in many common cases.
The text was updated successfully, but these errors were encountered: