Convert existing tests to pytest + increase coverage #1255

TechnicalPirate · 2021-09-27T14:32:10Z

Problem - Unit test coverage of the cx_freeze project is very low - prior to Pull-1234 the majority were also not working.

Improving test coverage of the product code will support development of cx_freeze by enabling developers and contributors to have a degree of confidence that there changes are non-destructive to existing functionality and expected usage.

Proposed Solution - This PR replaces the existing nose tests with a pytest suite preconfigured to generate coverage reports as required ( Article here going over benefits of PyTest ) as well as these new tests:

test__init__.py -> test_exposed_namespaces
test_exception -> test_raise_exceptions
test_winmsvcr.py -> test_FILES & test_FILES_TO_DUPLICATE
test_winversioninfo.py -> test___init__ and test___init__pads_short_versions

A readme.md is provided in the /tests folder explaining how to execute the tests.

With this PR total coverage of the suite is raised to 27%

... While this is in review - i will begin writing coverage for more modules ( slowly building up to addressing : Issue-594

…entation

…_and_test_readme

Disable failing test Removed unused import

…_and_test_readme

…to_pytest

…ture/convert_existing_tests_to_pytest

TechnicalPirate · 2021-09-27T16:55:56Z

@marcelotduarte - Thanks for reviewing / deciding on outcomes to the other PR's - with those done i have now updated this PR.

Please feel free to try executing the tests and generating the coverage report and let me know what you think. - If you like this direction ~~I'll prepare this branch as it stands for merging~~ Branch prepared/cleaned and I'll open other PR's module by module as i write test coverage to keep things flowing

Cheers and have a good day/night! :)

marcelotduarte · 2021-09-28T15:26:14Z

Hi! In two or three days I'll try to make time to test and review. Good afternoon! ;-)

TechnicalPirate · 2021-09-28T17:05:15Z

No stress, Just wanted to square this one up so i could start on the more complex modules.

I may start suggesting small refactors on those other PR's to make code more "testable" :)

…ing_tests_to_pytest

marcelotduarte

.coverage should be ignored at .gitignore?

marcelotduarte · 2021-10-06T05:27:28Z

tests/test_finder.py

+
+    @pytest.mark.skip("Test skipped, uncertain if no longer supported - waiting for maintainer to comment on:"
+                      "https://github.com/marcelotduarte/cx_Freeze/pull/1234")
+    def test_FindModule_from_zip(self, fix_module_finder, fix_samples_dir, tmpdir):


This test can be marked as skipped for now. In the rewrite of ModuleFinder, I ignored this method, but similar support for importing directing from the wheel (.whl)... Maybe we can delete it. I'll think about it

Your shout - the current usage of @pytest.mark.skip disables execution of the test - if you want to remove the underlying code + test go for it. I didn't want to remove existing tests without discussion with you :)

marcelotduarte · 2021-10-06T05:38:16Z

In windows:

========================================================= test session starts =========================================================
platform win32 -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: c:\clone\cx_Freeze
plugins: cov-2.12.1, mock-3.6.1, timeout-1.4.2
collected 15 items

tests\test___init__.py ....                                                                                                      [ 26%]
tests\test_exception.py ..                                                                                                       [ 40%]
tests\test_finder.py ...s                                                                                                        [ 66%]
tests\test_winmsvcr.py ..                                                                                                        [ 80%]
tests\test_winversioninfo.py ...                                                                                                 [100%]

---------- coverage: platform win32, python 3.8.10-final-0 -----------
Coverage HTML written to dir htmlcov


=================================================== 14 passed, 1 skipped in 20.44s ====================================================

In Linux:

==================================================== test session starts =====================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/marcelo/github/cx_Freeze
plugins: mock-3.6.1, timeout-1.4.2, cov-2.12.1
collected 15 items                                                                                                           

tests/test___init__.py .F..                                                                                            [ 26%]
tests/test_exception.py ..                                                                                             [ 40%]
tests/test_finder.py ...s                                                                                              [ 66%]
tests/test_winmsvcr.py ..                                                                                              [ 80%]
tests/test_winversioninfo.py ...                                                                                       [100%]

========================================================== FAILURES ==========================================================
_______________________________________ test_exposed_namespaces[win32-extra_modules1] ________________________________________

mocker = <pytest_mock.plugin.MockerFixture object at 0x7f757f3cfb20>, platform = 'win32', extra_modules = ['bdist_msi']

    @pytest.mark.parametrize(
        "platform, extra_modules",
        [
            (None, []),
            ("win32", ["bdist_msi"]),
            ("darwin", ["bdist_dmg", "bdist_mac"]),
            ("linux", []),
        ],
    )
    def test_exposed_namespaces(mocker, platform, extra_modules):
        """This test asserts that all the namespaces that should be exposed when `importing cx_Freeze` are available"""
        if platform:  # Mock platform before import :)
            mocker.patch.object(sys, "platform", platform)
    
>       importlib.reload(cx_Freeze)

tests/test___init__.py:21: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.9/importlib/__init__.py:169: in reload
    _bootstrap._exec(spec, module)
<frozen importlib._bootstrap>:613: in _exec
    ???
<frozen importlib._bootstrap_external>:855: in exec_module
    ???
<frozen importlib._bootstrap>:228: in _call_with_frames_removed
    ???
cx_Freeze/__init__.py:25: in <module>
    from .windist import bdist_msi
cx_Freeze/windist.py:1: in <module>
    import distutils.command.bdist_msi
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    Implements the bdist_msi command.
    """
    
    import os
    import sys
    import warnings
    from distutils.core import Command
    from distutils.dir_util import remove_tree
    from distutils.sysconfig import get_python_version
    from distutils.version import StrictVersion
    from distutils.errors import DistutilsOptionError
    from distutils.util import get_platform
    from distutils import log
>   import msilib
E   ModuleNotFoundError: No module named 'msilib'

/usr/lib/python3.9/distutils/command/bdist_msi.py:19: ModuleNotFoundError
================================================== short test summary info ===================================================
FAILED tests/test___init__.py::test_exposed_namespaces[win32-extra_modules1] - ModuleNotFoundError: No module named 'msilib'
========================================== 1 failed, 13 passed, 1 skipped in 8.57s ===========================================

TechnicalPirate · 2021-10-06T08:28:17Z

.coverage should be ignored at .gitignore?

There's 2 coverage files, one is the db used to generate the report (this is the one in the .gitignore ) the other is the .coveragerc which is a configuration file for which modules to be included in the report - this we should keep checked in :)

import msilib
E ModuleNotFoundError: No module named 'msilib'

Makes sense! - Prob will fail on OSX as well 🤔 ... contemplating how to handle this for testing - will get back to you....

marcelotduarte · 2021-10-06T08:55:30Z

.coverage should be ignored at .gitignore?

There's 2 coverage files, one is the db used to generate the report (this is the one in the .gitignore ) the other is the .coveragerc which is a configuration file for which modules to be included in the report - this we should keep checked in :)

I'm talking about a binary (db) file that appears as first file at https://github.com/marcelotduarte/cx_Freeze/pull/1255/files
It is not in .gitignore

TechnicalPirate · 2021-10-06T08:56:13Z

Oh bugger! well spotted :) will fix ( am fixing failing test right now - gimmi 10min :) )

TechnicalPirate · 2021-10-06T10:03:54Z

@marcelotduarte - Test should be fixed now + excess file nuked + gitignore updated :)

marcelotduarte · 2021-10-06T12:17:34Z

==================================================== test session starts =====================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/marcelo/github/cx_Freeze
plugins: mock-3.6.1, timeout-1.4.2, cov-2.12.1
collected 15 items                                                                                                           

tests/test___init__.py .F..                                                                                            [ 26%]
tests/test_exception.py ..                                                                                             [ 40%]
tests/test_finder.py ...s                                                                                              [ 66%]
tests/test_winmsvcr.py ..                                                                                              [ 80%]
tests/test_winversioninfo.py ...                                                                                       [100%]

========================================================== FAILURES ==========================================================
_______________________________________ test_exposed_namespaces[win32-extra_modules1] ________________________________________

mocker = <pytest_mock.plugin.MockerFixture object at 0x7f86e1ff2040>, platform = 'win32', extra_modules = ['bdist_msi']

    @pytest.mark.parametrize(
        "platform, extra_modules", [
            (None, []),
            ("win32", ["bdist_msi"]),
            ("darwin", ["bdist_dmg", "bdist_mac"]),
            ("linux", [])
        ]
    )
    def test_exposed_namespaces(mocker, platform, extra_modules):
        """ This test asserts that all the namespaces that should be exposed when `importing cx_Freeze` are available """
        # Mock platform specific modules so we don't get import errors due modules not being available on test device
        platform_specific_modules = ["msilib", ]
        for mod_name in platform_specific_modules:
            fake_mod = types.ModuleType(mod_name)
            sys.modules[mod_name] = fake_mod
    
        if platform:  # Mock platform before import :)
            mocker.patch.object(sys, "platform", platform)
    
>       importlib.reload(cx_Freeze)

tests/test___init__.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.9/importlib/__init__.py:169: in reload
    _bootstrap._exec(spec, module)
<frozen importlib._bootstrap>:613: in _exec
    ???
<frozen importlib._bootstrap_external>:855: in exec_module
    ???
<frozen importlib._bootstrap>:228: in _call_with_frames_removed
    ???
cx_Freeze/__init__.py:25: in <module>
    from .windist import bdist_msi
cx_Freeze/windist.py:1: in <module>
    import distutils.command.bdist_msi
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    Implements the bdist_msi command.
    """
    
    import os
    import sys
    import warnings
    from distutils.core import Command
    from distutils.dir_util import remove_tree
    from distutils.sysconfig import get_python_version
    from distutils.version import StrictVersion
    from distutils.errors import DistutilsOptionError
    from distutils.util import get_platform
    from distutils import log
    import msilib
>   from msilib import schema, sequence, text
E   ImportError: cannot import name 'schema' from 'msilib' (unknown location)

/usr/lib/python3.9/distutils/command/bdist_msi.py:20: ImportError

----------- coverage: platform linux, python 3.9.5-final-0 -----------
Coverage HTML written to dir htmlcov

================================================== short test summary info ===================================================
FAILED tests/test___init__.py::test_exposed_namespaces[win32-extra_modules1] - ImportError: cannot import name 'schema' fro...
========================================== 1 failed, 13 passed, 1 skipped in 13.07s ==========================================

TechnicalPirate · 2021-10-06T12:37:23Z

Bah! Ok - will fix that up later - watch this space :)

TechnicalPirate · 2021-10-08T14:00:27Z

@marcelotduarte - sorry for delay, have changed approach slightly lets see if it passes now

( Will happily refine tests later, but right now would like to get framework switched over so i can move onto the other tests :) )

marcelotduarte · 2021-10-08T15:16:36Z

==================================================== test session starts =====================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/marcelo/github/cx_Freeze
plugins: mock-3.6.1, timeout-1.4.2, cov-2.12.1
collected 15 items                                                                                                           

tests/test___init__.py .F..                                                                                            [ 26%]
tests/test_exception.py ..                                                                                             [ 40%]
tests/test_finder.py ...s                                                                                              [ 66%]
tests/test_winmsvcr.py ..                                                                                              [ 80%]
tests/test_winversioninfo.py ...                                                                                       [100%]

========================================================== FAILURES ==========================================================
_______________________________________ test_exposed_namespaces[win32-extra_modules1] ________________________________________

mocker = <pytest_mock.plugin.MockerFixture object at 0x7f7335bad6a0>, platform = 'win32', extra_modules = ['bdist_msi']

    @pytest.mark.parametrize(
        "platform, extra_modules", [
            (None, []),
            ("win32", ["bdist_msi"]),
            ("darwin", ["bdist_dmg", "bdist_mac"]),
            ("linux", [])
        ]
    )
    def test_exposed_namespaces(mocker, platform, extra_modules):
        """ This test asserts that all the namespaces that should be exposed when `importing cx_Freeze` are available """
    
        if platform:  # Mock platform before import :)
            mocker.patch.object(sys, "platform", platform)
    
        try:
            importlib.reload(cx_Freeze)
        except ImportError:
            pass  # Pass import errors as some modules are platform specific, we're testing what's exposed in `__all__` here
    
        expected_namespaces = [  # These namespaces are there regardless of platform
            "bdist_rpm",
            "build",
            "build_exe",
            "install",
            "install_exe",
            "setup",
            "ConfigError",
            "ConstantsModule",
            "Executable",
            "Freezer",
            "Module",
            "ModuleFinder",
        ]
        for ns in expected_namespaces:
            assert ns in dir(cx_Freeze)
    
        if platform:
            for ns in extra_modules:
>               assert ns in dir(cx_Freeze)
E               AssertionError: assert 'bdist_msi' in ['ConfigError', 'ConstantsModule', 'Executable', 'Freezer', 'Module', 'ModuleFinder', ...]
E                +  where ['ConfigError', 'ConstantsModule', 'Executable', 'Freezer', 'Module', 'ModuleFinder', ...] = dir(cx_Freeze)

tests/test___init__.py:45: AssertionError

----------- coverage: platform linux, python 3.9.5-final-0 -----------
Coverage HTML written to dir htmlcov

================================================== short test summary info ===================================================
FAILED tests/test___init__.py::test_exposed_namespaces[win32-extra_modules1] - AssertionError: assert 'bdist_msi' in ['Conf...
========================================== 1 failed, 13 passed, 1 skipped in 13.34s ==========================================

TechnicalPirate · 2021-10-08T15:46:26Z

Bah! Ok - Disabled the test for now to unblock the PR. I'll address this in a future PR :)

marcelotduarte · 2021-10-08T17:53:31Z

I think that you can use skipif in the non-windows platform.

TechnicalPirate · 2021-10-08T21:16:07Z

yep i'll split the test into platform specific tests for the platform specific modules - for now ive just disabled the whole thing as its a "New" test and i don't want it to get in the way if you approve of the move to pytest :)

marcelotduarte · 2021-10-09T00:51:56Z

I merged so you can adjust. If you can make small patches to facilitate it will be good, but when this is not possible, for now, in this part of adjusting the tests, no problem.
Thanks!

Joe Deuchar and others added 17 commits August 30, 2021 14:46

Initial commit / skip pycharm config

37ede11

Added essential dependancies

36628ae

Merge remote-tracking branch 'origin/main' into feature/pytest_implem…

7dcfb8a

…entation

Added another developer dependancy + a readme for test execution

3407d13

Merge branch 'marcelotduarte:main' into feature/pipenv_dep_management…

9b77e1c

…_and_test_readme

Fixes for all but one unit test

34f480d

Disable failing test Removed unused import

Merge branch 'marcelotduarte:main' into feature/fix_tests

a27ad4d

Merge branch 'marcelotduarte:main' into feature/pipenv_dep_management…

1074d9c

…_and_test_readme

Added pycharm idea folder to excluded dirs

34bbf76

Updated lock file

df9dac9

Merge branch 'feature/fix_tests' into feature/convert_existing_tests_…

ca2b440

…to_pytest

Initial PyTest setup + import namespace test

1f7a75e

Ported first nose test

07cf1bc

Fully ported test_finder

a3e9c59

Moved remaining files

f3c482f

Delete old test suite

af93b30

added some more very basic tests

c28fed2

TechnicalPirate changed the title ~~Feature/convert existing tests to pytest~~ convert existing tests to pytest + Extend coverage Sep 27, 2021

TechnicalPirate changed the title ~~convert existing tests to pytest + Extend coverage~~ Convert existing tests to pytest + increase coverage Sep 27, 2021

TechnicalPirate and others added 8 commits September 27, 2021 18:08

Generated dev-requirements

de7d0ef

Remove pipenv files

c0642a8

Ignore pipfiles

6530818

Merge branch 'main' into feature/pipenv_dep_management_and_test_readme

2df9638

Update Readme.md

9087a34

Merge branch 'feature/pipenv_dep_management_and_test_readme' into fea…

efb4dba

…ture/convert_existing_tests_to_pytest

Merge branch 'main' into feature/convert_existing_tests_to_pytest

c48a771

Cleaning up reports + ignores

8e7152e

Removed unimplemented tests + improved param call

001aefd

TechnicalPirate marked this pull request as ready for review September 28, 2021 09:50

Improve test to avoid ns pollution

9057119

Merge remote-tracking branch 'origin/main' into feature/convert_exist…

55bd656

…ing_tests_to_pytest

marcelotduarte reviewed Oct 6, 2021

View reviewed changes

Fix failing test + nuke uneeded file + gitignore update

e3833d5

Switch approach for handling namesapce test

93dc1c5

Disabled test temporarily

7b2d3e4

marcelotduarte merged commit b4e1157 into marcelotduarte:main Oct 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert existing tests to pytest + increase coverage #1255

Convert existing tests to pytest + increase coverage #1255

TechnicalPirate commented Sep 27, 2021 •

edited

Loading

TechnicalPirate commented Sep 27, 2021 •

edited

Loading

marcelotduarte commented Sep 28, 2021

TechnicalPirate commented Sep 28, 2021

marcelotduarte left a comment

marcelotduarte Oct 6, 2021

TechnicalPirate Oct 6, 2021

marcelotduarte commented Oct 6, 2021

TechnicalPirate commented Oct 6, 2021 •

edited

Loading

marcelotduarte commented Oct 6, 2021 •

edited

Loading

TechnicalPirate commented Oct 6, 2021

TechnicalPirate commented Oct 6, 2021

marcelotduarte commented Oct 6, 2021

TechnicalPirate commented Oct 6, 2021

TechnicalPirate commented Oct 8, 2021

marcelotduarte commented Oct 8, 2021

TechnicalPirate commented Oct 8, 2021

marcelotduarte commented Oct 8, 2021

TechnicalPirate commented Oct 8, 2021

marcelotduarte commented Oct 9, 2021 •

edited

Loading

Convert existing tests to pytest + increase coverage #1255

Convert existing tests to pytest + increase coverage #1255

Conversation

TechnicalPirate commented Sep 27, 2021 • edited Loading

TechnicalPirate commented Sep 27, 2021 • edited Loading

marcelotduarte commented Sep 28, 2021

TechnicalPirate commented Sep 28, 2021

marcelotduarte left a comment

Choose a reason for hiding this comment

marcelotduarte Oct 6, 2021

Choose a reason for hiding this comment

TechnicalPirate Oct 6, 2021

Choose a reason for hiding this comment

marcelotduarte commented Oct 6, 2021

TechnicalPirate commented Oct 6, 2021 • edited Loading

marcelotduarte commented Oct 6, 2021 • edited Loading

TechnicalPirate commented Oct 6, 2021

TechnicalPirate commented Oct 6, 2021

marcelotduarte commented Oct 6, 2021

TechnicalPirate commented Oct 6, 2021

TechnicalPirate commented Oct 8, 2021

marcelotduarte commented Oct 8, 2021

TechnicalPirate commented Oct 8, 2021

marcelotduarte commented Oct 8, 2021

TechnicalPirate commented Oct 8, 2021

marcelotduarte commented Oct 9, 2021 • edited Loading

TechnicalPirate commented Sep 27, 2021 •

edited

Loading

TechnicalPirate commented Sep 27, 2021 •

edited

Loading

TechnicalPirate commented Oct 6, 2021 •

edited

Loading

marcelotduarte commented Oct 6, 2021 •

edited

Loading

marcelotduarte commented Oct 9, 2021 •

edited

Loading