Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace package_available with module_available #16607

Closed
wants to merge 7 commits into from
Closed

Conversation

Borda
Copy link
Member

@Borda Borda commented Feb 2, 2023

What does this PR do?

switch to rather use module_available until this is resolved: #16595 (comment)
resolves #16810

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda @tchaton @carmocca @justusschock @awaelchli

@Borda Borda added this to the v1.9.x milestone Feb 2, 2023
@Borda Borda added bug Something isn't working priority: 1 Medium priority task labels Feb 2, 2023
@github-actions github-actions bot added app (removed) Generic label for Lightning App package fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Feb 2, 2023
@Borda Borda enabled auto-merge (squash) February 2, 2023 09:02
@github-actions
Copy link
Contributor

github-actions bot commented Feb 2, 2023

⛈️ Required checks status: Has failure 🔴

Warning
This job will need to be re-run to merge your PR. If you do not have write access to the repository, you can ask Lightning-AI/lai-frameworks to re-run it. If you push a new commit, all of CI will re-trigger.

Groups summary

🔴 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.11) failure
pl-cpu (macOS-11, lightning, 3.9, 1.12) failure
pl-cpu (macOS-11, lightning, 3.10, 1.13) failure
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest) failure
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11) failure
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12) failure
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) failure
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) failure
pl-cpu (windows-2022, lightning, 3.9, 1.11) failure
pl-cpu (windows-2022, lightning, 3.10, 1.12) failure
pl-cpu (windows-2022, lightning, 3.10, 1.13) failure
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest) failure
pl-cpu (macOS-11, pytorch, 3.8, 1.13) failure
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) failure
pl-cpu (windows-2022, pytorch, 3.8, 1.13) failure

These checks are required after the changes to src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py, tests/tests_pytorch/graveyard/test_legacy_import_unpickler.py, tests/tests_pytorch/helpers/datamodules.py, tests/tests_pytorch/models/test_hparams.py, tests/tests_pytorch/strategies/launchers/test_subprocess_script.py, tests/tests_pytorch/utilities/test_imports.py, tests/tests_pytorch/utilities/test_torchdistx.py.

🔴 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) failure

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py, tests/tests_pytorch/graveyard/test_legacy_import_unpickler.py, tests/tests_pytorch/helpers/datamodules.py, tests/tests_pytorch/models/test_hparams.py, tests/tests_pytorch/strategies/launchers/test_subprocess_script.py, tests/tests_pytorch/utilities/test_imports.py, tests/tests_pytorch/utilities/test_torchdistx.py, src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py.

🔴 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) failure

These checks are required after the changes to src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py, tests/tests_pytorch/graveyard/test_legacy_import_unpickler.py, tests/tests_pytorch/helpers/datamodules.py, tests/tests_pytorch/models/test_hparams.py, tests/tests_pytorch/strategies/launchers/test_subprocess_script.py, tests/tests_pytorch/utilities/test_imports.py, tests/tests_pytorch/utilities/test_torchdistx.py.

🟢 pytorch_lightning: Azure IPU
Check ID Status
pytorch-lightning (IPUs) success

These checks are required after the changes to src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py, tests/tests_pytorch/graveyard/test_legacy_import_unpickler.py, tests/tests_pytorch/helpers/datamodules.py, tests/tests_pytorch/models/test_hparams.py, tests/tests_pytorch/strategies/launchers/test_subprocess_script.py, tests/tests_pytorch/utilities/test_imports.py, tests/tests_pytorch/utilities/test_torchdistx.py.

🔴 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) failure
make-html (pytorch) success

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py.

🔴 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11) failure
fabric-cpu (macOS-11, lightning, 3.9, 1.12) failure
fabric-cpu (macOS-11, lightning, 3.10, 1.13) failure
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest) failure
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.11) failure
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.12) failure
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) failure
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) failure
fabric-cpu (windows-2022, lightning, 3.9, 1.11) success
fabric-cpu (windows-2022, lightning, 3.10, 1.12) success
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest) success
fabric-cpu (macOS-11, fabric, 3.8, 1.13) failure
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) failure
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success

These checks are required after the changes to src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) success

These checks are required after the changes to src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py.

🔴 lightning_app: Tests workflow
Check ID Status
app-pytest (macOS-11, lightning, 3.8, latest) cancelled 🚫
app-pytest (macOS-11, lightning, 3.8, oldest) success
app-pytest (macOS-11, app, 3.9, latest) cancelled 🚫
app-pytest (ubuntu-20.04, lightning, 3.8, latest) success
app-pytest (ubuntu-20.04, lightning, 3.8, oldest) success
app-pytest (ubuntu-20.04, app, 3.9, latest) success
app-pytest (windows-2022, lightning, 3.8, latest) success
app-pytest (windows-2022, lightning, 3.8, oldest) success
app-pytest (windows-2022, app, 3.8, latest) success

These checks are required after the changes to src/lightning/app/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/lightning_cli.py, src/lightning/app/testing/testing.py.

🟢 lightning_app: Examples
Check ID Status
app-examples (macOS-11, lightning, 3.9, latest) success
app-examples (macOS-11, lightning, 3.9, oldest) success
app-examples (macOS-11, app, 3.9, latest) success
app-examples (ubuntu-20.04, lightning, 3.9, latest) success
app-examples (ubuntu-20.04, lightning, 3.9, oldest) success
app-examples (ubuntu-20.04, app, 3.9, latest) success
app-examples (windows-2022, lightning, 3.9, latest) success
app-examples (windows-2022, lightning, 3.9, oldest) success
app-examples (windows-2022, app, 3.9, latest) success

These checks are required after the changes to src/lightning/app/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/lightning_cli.py, src/lightning/app/testing/testing.py, tests/integrations_app/public/test_multi_node.py.

🔴 lightning_app: Azure
Check ID Status
App.cloud-e2e failure

These checks are required after the changes to src/lightning/app/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/lightning_cli.py, src/lightning/app/testing/testing.py, tests/integrations_app/public/test_multi_node.py.

🟢 lightning_app: Docs
Check ID Status
make-doctest (lightning) success
make-html (lightning) success

These checks are required after the changes to src/lightning/app/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/lightning_cli.py, src/lightning/app/testing/testing.py.

🟢 mypy
Check ID Status
mypy success

These checks are required after the changes to src/lightning/app/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/lightning_cli.py, src/lightning/app/testing/testing.py, src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success
install-pkg (ubuntu-22.04, app, 3.10) success
install-pkg (ubuntu-22.04, fabric, 3.8) success
install-pkg (ubuntu-22.04, fabric, 3.10) success
install-pkg (ubuntu-22.04, pytorch, 3.8) success
install-pkg (ubuntu-22.04, pytorch, 3.10) success
install-pkg (ubuntu-22.04, lightning, 3.8) success
install-pkg (ubuntu-22.04, lightning, 3.10) success
install-pkg (ubuntu-22.04, notset, 3.8) success
install-pkg (ubuntu-22.04, notset, 3.10) success
install-pkg (macOS-12, app, 3.8) success
install-pkg (macOS-12, app, 3.10) success
install-pkg (macOS-12, fabric, 3.8) success
install-pkg (macOS-12, fabric, 3.10) success
install-pkg (macOS-12, pytorch, 3.8) success
install-pkg (macOS-12, pytorch, 3.10) success
install-pkg (macOS-12, lightning, 3.8) success
install-pkg (macOS-12, lightning, 3.10) success
install-pkg (macOS-12, notset, 3.8) success
install-pkg (macOS-12, notset, 3.10) success
install-pkg (windows-2022, app, 3.8) success
install-pkg (windows-2022, app, 3.10) success
install-pkg (windows-2022, fabric, 3.8) success
install-pkg (windows-2022, fabric, 3.10) success
install-pkg (windows-2022, pytorch, 3.8) success
install-pkg (windows-2022, pytorch, 3.10) success
install-pkg (windows-2022, lightning, 3.8) success
install-pkg (windows-2022, lightning, 3.10) success
install-pkg (windows-2022, notset, 3.8) success
install-pkg (windows-2022, notset, 3.10) success

These checks are required after the changes to src/lightning/app/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/lightning_cli.py, src/lightning/app/testing/testing.py, src/lightning/fabric/__init__.py, src/lightning/fabric/cli.py, src/lightning/fabric/loggers/tensorboard.py, src/lightning/fabric/plugins/environments/mpi.py, src/lightning/fabric/plugins/io/xla.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/cli.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/neptune.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/serve/servable_module_validator.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, src/lightning/pytorch/tuner/lr_finder.py, src/lightning/pytorch/utilities/imports.py, src/lightning/pytorch/utilities/model_helpers.py, src/lightning/pytorch/utilities/model_summary/model_summary_deepspeed.py.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@github-actions github-actions bot added the ci Continuous Integration label Feb 2, 2023
@github-actions github-actions bot removed the ci Continuous Integration label Feb 2, 2023
@mergify mergify bot added the has conflicts label Feb 2, 2023
@mergify mergify bot removed the has conflicts label Feb 2, 2023
@mergify mergify bot added the ready PRs ready to be merged label Feb 2, 2023
@awaelchli
Copy link
Contributor

@Borda Have you checked why the spec is None for the lightning package? See the implementation:

@lru_cache()
def package_available(package_name: str) -> bool:
    """Check if a package is available in your environment.

    >>> package_available('os')
    True
    >>> package_available('bla')
    False
    """
    try:
        return find_spec(package_name) is not None
    except ModuleNotFoundError:
        return False

There must be something missing in the lightning package.

@Borda
Copy link
Member Author

Borda commented Feb 2, 2023

Have you checked why the spec is None for the lightning package? See the implementation:

yes, will check just want to patch it until we have it resolved :)

@Borda Borda requested a review from carmocca February 2, 2023 14:26
@Borda Borda removed the priority: 1 Medium priority task label Feb 2, 2023
@codecov
Copy link

codecov bot commented Feb 6, 2023

Codecov Report

Merging #16607 (469cd1b) into master (b30a43f) will decrease coverage by 17%.
The diff coverage is 85%.

Additional details and impacted files
@@            Coverage Diff            @@
##           master   #16607     +/-   ##
=========================================
- Coverage      82%      64%    -17%     
=========================================
  Files         435      416     -19     
  Lines       31517    31310    -207     
=========================================
- Hits        25732    20124   -5608     
- Misses       5785    11186   +5401     

@mergify mergify bot added has conflicts and removed ready PRs ready to be merged labels Feb 8, 2023
@Borda Borda marked this pull request as draft February 11, 2023 02:31
auto-merge was automatically disabled February 11, 2023 02:31

Pull request was converted to draft

@Borda Borda marked this pull request as ready for review February 21, 2023 10:31
@mergify mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Feb 21, 2023
Copy link
Contributor

@awaelchli awaelchli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are incorrect usages of the module_available utilities.

@@ -416,7 +416,7 @@ def run_app(
)


if RequirementCache("lightning-fabric>=1.9.0") or RequirementCache("lightning>=1.9.0"):
if ModuleAvailableCache("lightning-fabric>=1.9.0") or ModuleAvailableCache("lightning>=1.9.0"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this work? module available checks don't support version comparison!

@@ -34,7 +34,7 @@
from lightning.pytorch.utilities.model_summary import ModelSummary
from lightning.pytorch.utilities.rank_zero import rank_zero_only

_NEPTUNE_AVAILABLE = RequirementCache("neptune-client")
_NEPTUNE_AVAILABLE = ModuleAvailableCache("neptune-client")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not a module

@@ -691,7 +691,7 @@ def test_model_with_fsspec_as_parameter(tmpdir):
trainer.test()


@pytest.mark.skipif(RequirementCache("hydra-core<1.1"), reason="Requires Hydra's Compose API")
@pytest.mark.skipif(ModuleAvailableCache("hydra-core<1.1"), reason="Requires Hydra's Compose API")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hydra-core is not a module

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
app (removed) Generic label for Lightning App package bug Something isn't working fabric lightning.fabric.Fabric has conflicts pl Generic label for PyTorch Lightning package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ModuleNotFoundError: Neither tensorboard nor tensorboardX is available.
6 participants