Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model containers should have a choice of running in batch or single-input mode #285

Closed
dcrankshaw opened this issue Sep 3, 2017 · 1 comment

Comments

@dcrankshaw
Copy link
Contributor

Currently, we expect all deployed models to be a function from a list of inputs to a list of outputs. This allows models the ability to do batching, but it makes the function signature of models more complicated. For simple models that want to operate on a single-input at a time, users have to remember to wrap the input in for-loop.

Instead, we should support deploying two types of models: batch and single-input. Batch models are still given the full available batch at once. Single-input models would be a function from a single input to a single output, and the RPC server embedded in the model container would call the model in a for-loop transparently to the user.

This should make the model deployment process simpler in many common cases.

@dcrankshaw
Copy link
Contributor Author

Based on discussion in #395, we decided that rather than supporting single-input mode, we should improve our documentation and debugging capabilities to help users use the batching functionality in Clipper correctly. #394 is a good first step in that direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants