-
Notifications
You must be signed in to change notification settings - Fork 440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
distributed suggestion service #68
Comments
saving all trials to db sounds like a good first step, but once we can support large number of trials per study, and large number of studies, we might need to do more optimizations
|
@ddysher @YujiOshima @gaocegege Is there specific work related to this issue for 0.3? If so what is the priority? |
It is similar to #33 at the beginning, thus we do not need to add it into 0.3 |
I think we can solve this by solving #33 . But we need a test. |
Now we keep one suggestion for one experiment. But we still need #33 . /close |
@gaocegege: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
suggestion service is inherently stateful, we want to make sure a study is processed by a single suggestion service instance. For example, in current setup, if we run 2 replicas of grid service, they will both receive requests, thus essentially suggesting the same parameters twice.
The simplest solution would be to change the service affinity to 'ClientIP' in kubernetes. However, in the long run, we need to have proper handling in our suggestion services themselves as well, to handle potential failure cases.
/cc @YujiOshima @gaocegege
The text was updated successfully, but these errors were encountered: