Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add a commit hook which compare tests in Nightly and all non-ignored tests #2736

Closed
ailisp opened this issue May 27, 2020 · 5 comments · Fixed by #4584
Closed

add a commit hook which compare tests in Nightly and all non-ignored tests #2736

ailisp opened this issue May 27, 2020 · 5 comments · Fixed by #4584
Assignees
@mina86
Labels
A-CI Area: Continuous Integration A-testing Area: Unit testing / integration testing Node Node team T-node Team: issues relevant to the node experience team

Comments

@ailisp
Copy link
Member

ailisp commented May 27, 2020

to make sure no expensive test is forgotten
@bowenwang1996 bowenwang1996 added the A-testing Area: Unit testing / integration testing label May 30, 2020
@stale
Copy link

stale bot commented Jul 1, 2021

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

@stale stale bot added the S-stale label Jul 1, 2021
@bowenwang1996
Copy link
Collaborator

Shouldn't we just add this to CI? @ailisp what is the status of this issue?

@stale stale bot removed the S-stale label Jul 2, 2021
@ailisp
Copy link
Member Author

ailisp commented Jul 5, 2021

@bowenwang1996 I believe this is still nice to have. Yeah it should be part of CI, perhaps some kind of bot, so that if you add an expensive test or a test under pytest dir, a script should check complete_nayduck_set.txt and remind if it's not added

@bowenwang1996 bowenwang1996 added A-CI Area: Continuous Integration T-node Team: issues relevant to the node experience team labels Jul 5, 2021
@mina86
Copy link
Contributor

mina86 commented Jul 17, 2021

Just to confirm that I understand what this issue is about. On high level the request is to list all pytest/tests/**/*.py files and check if they have a corresponding line somewhere in one of the nightly/*.txt files?

@bowenwang1996
Copy link
Collaborator

@mina86 yes

mina86 added a commit to mina86/nearcore that referenced this issue Jul 19, 2021
@mina86
pytest: add scripts/check_pytest.py checking if tests are listed
49d722e 
On high level, this simply lists all pytest/tests/**/*.py files and
checks whether they are listed somewhere in nightly/*.txt files.  The
end goal is to have this as a required check before a commit is
accepted so that no pytest’s are forgotten.

Currently though, there are 18 files which the script complains
about.  The output of the script is:

    Found 18 pytest files (i.e. Python files in pytest/tests directory)
    which are not included in any of the nightly/*.txt files:

     -  pytest/tests/adversarial/gc_rollback.py
     -  pytest/tests/delete_remote_nodes.py
     -  pytest/tests/sanity/concurrent_function_calls.py
     -  pytest/tests/sanity/proxy_example.py
     -  pytest/tests/sanity/restart.py
     -  pytest/tests/sanity/rpc_finality.py
     -  pytest/tests/sanity/rpc_tx_submission.py
     -  pytest/tests/sanity/state_migration.py
     -  pytest/tests/stress/hundred_nodes/100_node_block_production.py
     -  pytest/tests/stress/hundred_nodes/block_chunks.py
     -  pytest/tests/stress/hundred_nodes/collect_logs.py
     -  pytest/tests/stress/hundred_nodes/create_gcloud_image.py
     -  pytest/tests/stress/hundred_nodes/create_prometheus_targets.py
     -  pytest/tests/stress/hundred_nodes/node_rotation.py
     -  pytest/tests/stress/hundred_nodes/start_100_nodes.py
     -  pytest/tests/stress/hundred_nodes/watch_fork.py
     -  pytest/tests/stress/network_stress.py
     -  pytest/tests/stress/saturate_routing_table.py

    Add the tests to one of the lists in nightly/*.txt files.  For example
    as:

        pytest adversarial/gc_rollback.py

    If the test is temporarily disabled, add it to one of the files with an
    appropriate TODO comment.  For example:

        # TODO(near#1234): Enable the test again once <some condition>
        # pytest adversarial/gc_rollback.py

    Note that the TODO comment must reference a GitHub issue (i.e. must
    contain a #<number> string).

    If the file is not a test but a helper library, consider moving it out
    of the pytest/tests directory to pytest/lib or add it to HELPER_SCRIPTS
    list at the top of scripts/check_pytests.py file.

Before this check is incorporated in checks pipeline, the above needs
to be resolved.

Issue: near#2736
@mina86 mina86 mentioned this issue Jul 19, 2021
Merged
mina86 added a commit to mina86/nearcore that referenced this issue Jul 20, 2021
@mina86
pytest: add scripts/check_pytest.py checking if tests are listed
faddf20 
On high level, this simply lists all pytest/tests/**/*.py files and
checks whether they are listed somewhere in nightly/*.txt files.  The
end goal is to have this as a required check before a commit is
accepted so that no pytest’s are forgotten.

Currently though, there are 10 files which the script complains
about.  The output of the script is:

    Found 10 pytest files (i.e. Python files in pytest/tests directory)
    which are not included in any of the nightly/*.txt files:

     -  pytest/tests/adversarial/gc_rollback.py
     -  pytest/tests/delete_remote_nodes.py
     -  pytest/tests/sanity/concurrent_function_calls.py
     -  pytest/tests/sanity/proxy_example.py
     -  pytest/tests/sanity/restart.py
     -  pytest/tests/sanity/rpc_finality.py
     -  pytest/tests/sanity/rpc_tx_submission.py
     -  pytest/tests/sanity/state_migration.py
     -  pytest/tests/stress/network_stress.py
     -  pytest/tests/stress/saturate_routing_table.py

    Add the tests to one of the lists in nightly/*.txt files.  For example
    as:

        pytest sanity/rpc_tx_submission.py

    If the test is temporarily disabled, add it to one of the files with an
    appropriate TODO comment.  For example:

        # TODO(near#1234): Enable the test again once <some condition>
        # pytest sanity/rpc_tx_submission.py

    Note that the TODO comment must reference a GitHub issue (i.e. must
    contain a #<number> string).

    If the file is not a test but a helper library, consider moving it out
    of the pytest/tests directory to pytest/lib or add it to HELPER_SCRIPTS
    list at the top of scripts/check_pytests.py file.

Before this check is incorporated in checks pipeline, the above needs
to be resolved.

Issue: near#2736
mina86 added a commit to mina86/nearcore that referenced this issue Jul 20, 2021
@mina86
pytest: add scripts/check_pytest.py checking if tests are listed
ecb5fa2 
On high level, this simply lists all pytest/tests/**/*.py files and
checks whether they are listed somewhere in nightly/*.txt files.  The
end goal is to have this as a required check before a commit is
accepted so that no pytest’s are forgotten.

Currently though, there are 10 files which the script complains
about.  The output of the script is:

    Found 9 pytest files (i.e. Python files in pytest/tests directory)
    which are not included in any of the nightly/*.txt files:

     -  pytest/tests/adversarial/gc_rollback.py
     -  pytest/tests/sanity/concurrent_function_calls.py
     -  pytest/tests/sanity/proxy_example.py
     -  pytest/tests/sanity/restart.py
     -  pytest/tests/sanity/rpc_finality.py
     -  pytest/tests/sanity/rpc_tx_submission.py
     -  pytest/tests/sanity/state_migration.py
     -  pytest/tests/stress/network_stress.py
     -  pytest/tests/stress/saturate_routing_table.py

    Add the tests to one of the lists in nightly/*.txt files.  For example
    as:

        pytest sanity/rpc_tx_submission.py

    If the test is temporarily disabled, add it to one of the files with an
    appropriate TODO comment.  For example:

        # TODO(near#1234): Enable the test again once <some condition>
        # pytest sanity/rpc_tx_submission.py

    Note that the TODO comment must reference a GitHub issue (i.e. must
    contain a #<number> string).

    If the file is not a test but a helper library, consider moving it out
    of the pytest/tests directory to pytest/lib or add it to HELPER_SCRIPTS
    list at the top of scripts/check_pytests.py file.

Before this check is incorporated in checks pipeline, the above needs
to be resolved.

Issue: near#2736
@mina86 mina86 mentioned this issue Jul 22, 2021
Closed
2 tasks
mina86 added a commit to mina86/nearcore that referenced this issue Jul 26, 2021
@mina86
pytest: add scripts/check_pytest.py checking if tests are listed
53c97b5 
On high level, this simply lists all pytest/tests/**/*.py files and
checks whether they are listed somewhere in nightly/*.txt files.  The
end goal is to have this as a required check before a commit is
accepted so that no pytest’s are forgotten.

Currently though, there are 10 files which the script complains
about.  The output of the script is:

    Found 9 pytest files (i.e. Python files in pytest/tests directory)
    which are not included in any of the nightly/*.txt files:

     -  pytest/tests/adversarial/gc_rollback.py
     -  pytest/tests/sanity/concurrent_function_calls.py
     -  pytest/tests/sanity/proxy_example.py
     -  pytest/tests/sanity/restart.py
     -  pytest/tests/sanity/rpc_finality.py
     -  pytest/tests/sanity/rpc_tx_submission.py
     -  pytest/tests/sanity/state_migration.py
     -  pytest/tests/stress/network_stress.py
     -  pytest/tests/stress/saturate_routing_table.py

    Add the tests to one of the lists in nightly/*.txt files.  For example
    as:

        pytest sanity/rpc_tx_submission.py

    If the test is temporarily disabled, add it to one of the files with an
    appropriate TODO comment.  For example:

        # TODO(near#1234): Enable the test again once <some condition>
        # pytest sanity/rpc_tx_submission.py

    Note that the TODO comment must reference a GitHub issue (i.e. must
    contain a #<number> string).

    If the file is not a test but a helper library, consider moving it out
    of the pytest/tests directory to pytest/lib or add it to HELPER_SCRIPTS
    list at the top of scripts/check_pytests.py file.

Before this check is incorporated in checks pipeline, the above needs
to be resolved.

Issue: near#2736
near-bulldozer bot pushed a commit that referenced this issue Jul 27, 2021
@mina86
pytest: add scripts/check_pytest.py checking if tests are listed (#4539)
f614232 
On high level, this simply lists all `pytest/tests/**/*.py` files and
checks whether they are listed somewhere in `nightly/*.txt` files.
The end goal is to have this as a required check before a commit
is accepted so that no pytest’s are forgotten.

Currently though, there are 9 files which the script complains
about.  The output of the script is:

    Found 9 pytest files (i.e. Python files in pytest/tests directory)
    which are not included in any of the nightly/*.txt files:

     -  pytest/tests/adversarial/gc_rollback.py
     -  pytest/tests/sanity/concurrent_function_calls.py
     -  pytest/tests/sanity/proxy_example.py
     -  pytest/tests/sanity/restart.py
     -  pytest/tests/sanity/rpc_finality.py
     -  pytest/tests/sanity/rpc_tx_submission.py
     -  pytest/tests/sanity/state_migration.py
     -  pytest/tests/stress/network_stress.py
     -  pytest/tests/stress/saturate_routing_table.py

    Add the tests to one of the lists in nightly/*.txt files.  For example
    as:

        pytest sanity/rpc_tx_submission.py

    If the test is temporarily disabled, add it to one of the files with an
    appropriate TODO comment.  For example:

        # TODO(#1234): Enable the test again once <some condition>
        # pytest sanity/rpc_tx_submission.py

    Note that the TODO comment must reference a GitHub issue (i.e. must
    contain a #<number> string).

    If the file is not a test but a helper library, consider moving it out
    of the pytest/tests directory to pytest/lib or add it to HELPER_SCRIPTS
    list at the top of scripts/check_pytests.py file.

Before this check is incorporated in checks pipeline, the above needs
to be resolved.

Issue: #2736
mina86 added a commit to mina86/nearcore that referenced this issue Jul 27, 2021
@mina86
pytest: add a pre-commit check testing if all pytests are enabled
278043d 
Add the scripts/check_pytest.py to the ‘sanity checks’ pre-commit
tests to make sure that all pytest tests are enabled in one of the
nightly/*.txt files.

Because at the moment there are nine files which do not meet that
criteria, add them to nightly/complete_nayduck_set.txt with a comment
documenting the state.

Issue: near#4552
Fixes: near#2736
@mina86 mina86 mentioned this issue Jul 27, 2021
Merged
mina86 added a commit that referenced this issue Jul 27, 2021
@mina86
pytest: add a pre-commit check testing if all pytests are enabled (#4584
8b8fbb5 
)

Add the scripts/check_pytest.py to the ‘sanity checks’ pre-commit
tests to make sure that all pytest tests are enabled in one of the
nightly/*.txt files.

Because at the moment there are nine files which do not meet that
criteria, add them to nightly/complete_nayduck_set.txt with a comment
documenting the state.

Issue: #4552
Fixes: #2736
mina86 added a commit to mina86/nearcore that referenced this issue Jul 28, 2021
@mina86
nayduck: remove --run_type flag; it’s never used by anything
a1b9e4f 
Issue: Near-One/nayduck#8
Issue: near#2736
@mina86 mina86 mentioned this issue Jul 28, 2021
Merged
mina86 added a commit that referenced this issue Jul 28, 2021
@mina86
nayduck: remove --run_type flag; it’s never used by anything (#4586)
dc6377e 
Issue: Near-One/nayduck#8
Issue: #2736
mina86 added a commit to mina86/nearcore that referenced this issue Jul 28, 2021
@mina86
nayduck: make nayduck.py a library; support ./ in scripts/check_*.py
e3cb826 
nightly/nayduck.py script supports a `./<path>` syntax for including
another file when reading contents of particular file.  (This is akin
to shell’s `source` command or C’s `#include` directive).  Sadly,
neither scripts/check_pytests.py nor scripts/check_nightly.py support
that syntax.

Fix this by making the nayduck.py script easier to import as a library
and use it in the aforementioned scripts.

While at it, do minor cleanups to scripts/check_nightly.py script and
move nightly.py to scripts directory.

Issue: near#2736
Issue: near#4552
@mina86 mina86 mentioned this issue Jul 28, 2021
Merged
mina86 added a commit to mina86/nearcore that referenced this issue Jul 28, 2021
@mina86
nayduck: fix ./ handling
cde8109 
Due to bug introduced in previosu commit, inclusion of any `./<path>`
lines in tests file results in the same file being read again rather
than file denoted by <path>.  Fix that.

Issue: near#2736
Issue: near#4552
mina86 added a commit that referenced this issue Jul 28, 2021
@mina86
nayduck: make nayduck.py a library; support ./ in scripts/check_*.py (
b700d29 
#4590)

nightly/nayduck.py script supports a `./<path>` syntax for including
another file when reading contents of particular file.  (This is akin
to shell’s `source` command or C’s `#include` directive).  Sadly,
neither scripts/check_pytests.py nor scripts/check_nightly.py support
that syntax.

Fix this by making the nayduck.py script easier to import as a library
and use it in the aforementioned scripts.

While at it, do minor cleanups to scripts/check_nightly.py script and
move nightly.py to scripts directory.

Issue: #2736
Issue: #4552
@gmilescu gmilescu added the Node Node team label Oct 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mina86 mina86

Labels
A-CI Area: Continuous Integration A-testing Area: Unit testing / integration testing Node Node team T-node Team: issues relevant to the node experience team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pytest: add a pre-commit check testing if all pytests are enabled mina86/nearcore
4 participants
@mina86 @gmilescu @ailisp @bowenwang1996