-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix flakiness in fixed batch size integration test #387
Conversation
Test FAILed. |
Merged build finished. Test FAILed. |
Test FAILed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Test FAILed. |
Hmm @Corey-Zumar looks like this maybe doesn't fix the flakiness https://amplab.cs.berkeley.edu/jenkins/job/Clipper-PRB/936/console |
@dcrankshaw Based on testing with AWS, it appears that the container crashes when queried. No stack trace is produced. The issue appears to be associated with the serialization and deserialization of functions that call For now, in place of |
Hmm weird. You could just hardcode a new testing docker container that has a sleep in it (similar to the noop container). That seems like a potentially cleaner and more controlled solution. |
Good call. Will do that. |
Test FAILed. |
Test FAILed. |
@dcrankshaw The container-based approach doesn't fare any better. I'm still encountering the same traceless crash. Will need to debug. |
f20ef16
to
df23a4b
Compare
|
||
COPY containers/python/python_closure_container.py containers/python/python_closure_container_entry.sh /container/ | ||
|
||
|
||
CMD ["/container/python_closure_container_entry.sh"] | ||
ENTRYPOINT ["/container/python_closure_container_entry.sh"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changing this from CMD
to ENTRYPOINT
fixes the crash associated with the python call to time.sleep
. I'll have to explore why this is the case. CMD
executes the python script within /bin/sh -c
directive, while ENTRYPOINT
executes the python script in some other fashion; I'd imagine that this accounts for the difference.
@@ -3,12 +3,11 @@ FROM clipper/py-rpc:${CODE_VERSION} | |||
|
|||
COPY clipper_admin/clipper_admin/python_container_conda_deps.txt /lib/ | |||
RUN conda config --set ssl_verify no \ | |||
&& conda install -y -c anaconda cloudpickle=0.5.2 \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Installing this first fixes an installation error.
Test FAILed. |
Test PASSed. |
Test PASSed. |
Fixes #386