Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare for new release #1764

Merged
merged 2 commits into from
Jul 6, 2022
Merged

Prepare for new release #1764

merged 2 commits into from
Jul 6, 2022

Conversation

miguelgfierro
Copy link
Collaborator

Description

Related Issues

Checklist:

  • I have followed the contribution guidelines and code style for this project.
  • I have added tests covering my contributions.
  • I have updated the documentation accordingly.
  • This PR is being made to staging branch and not to main branch.

@miguelgfierro
Copy link
Collaborator Author

@pradnyeshjoshi we have an issue with the spark tuning code, I don't think it is a problem with this particular PR, but it is annoying

@pradnyeshjoshi
Copy link
Collaborator

@pradnyeshjoshi we have an issue with the spark tuning code, I don't think it is a problem with this particular PR, but it is annoying

@miguelgfierro this is weird, I just merged #1763, did not get this error there.

@miguelgfierro
Copy link
Collaborator Author

miguelgfierro commented Jun 30, 2022

@miguelgfierro this is weird, I just merged #1763, did not get this error there.

yeah, I think that precise test is flaky. I have reexecuted the failed jobs again

@miguelgfierro
Copy link
Collaborator Author

@pradnyeshjoshi @anargyri this is annoying, the test failed again. This spark tunning is super flaky:

============================= slowest 10 durations =============================
250.20s call     tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning
225.70s call     tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning
203.54s call     tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning
175.64s call     tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning
166.25s call     tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning
139.28s call     tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning
(4 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/unit/examples/test_notebooks_pyspark.py::test_spark_tuning - pap...

@anargyri
Copy link
Collaborator

anargyri commented Jul 1, 2022

Yes, this is difficult to understand. One thing that is special with this test in particular is that we call ALS.fit() two or three times. So, I would speculate that this may "overload" the resources allocated by Spark in some way. I suggest you try separating the ALS training runs into separate tests and see if this fixes the issue. It's probably feasible to do this by parameterizing the notebook in a creative way.

@miguelgfierro
Copy link
Collaborator Author

Added issue about this: #1770

@miguelgfierro
Copy link
Collaborator Author

@pradnyeshjoshi I have merged staging with your latest changes, hopefully we don't see the error in the tests anymore. Please review this PR

@pradnyeshjoshi
Copy link
Collaborator

@pradnyeshjoshi I have merged staging with your latest changes, hopefully we don't see the error in the tests anymore. Please review this PR

@miguelgfierro I cannot review this PR, it says I need to be a code owner to review,

@pradnyeshjoshi pradnyeshjoshi self-requested a review July 6, 2022 19:07
@miguelgfierro miguelgfierro merged commit 118dfe4 into staging Jul 6, 2022
@miguelgfierro miguelgfierro deleted the miguelgfierro-patch-1 branch July 6, 2022 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants