-
Notifications
You must be signed in to change notification settings - Fork 875
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add format and test scripts #1987
Conversation
@adam-narozniak @danieljanes should we include this before we launch SoR? at the very least the formatting script? |
@jafermarq I think we should add it, now it's ready. There will be a next PR that will automatically test only the changed baseline in the CI, but ppl can use that locally for testing and formatting. |
@adam-narozniak, i feel like we should provide instructions for people to know how to run the # the top level directory for baselines run after sourcing your environment:
./dev/format_baseline.sh <name-of-you-baseline> Also, i think we need to add Maybe you can do the |
* main: Add new baselines test github workflow (#2009) Add simulation to E2E tests (#2074) Add E2E test for Pandas (#2070) Add E2E test for scikit-learn (#2073) Add format and test scripts (#1987) Add missing ruff dependency to baselines (#2075) Add E2E test for MXNet (#2069) Add E2E test for Jax (#2067) Update bare E2E test client (#2068) Update PyTorch E2E test (#2072) Update Tensorflow E2E test (#2071) Fix flake8 error E266 in baseline template (#2065) Fix baseline creation on linux-based systems (#2063) Updates to Baseline Template Readmes (#2059) Refresh FedProx MNIST baseline (#1918) Extend test checking tools config (#1986) Improved documentation (#2006) updated material for 30min FL tutorial (#2005) Update codeowners list (#2004) Create and delete nodes via Fleet API (#1901)
* Add code quality tests * Add a structure test
Issue
The baseline created using the updated way doesn't have the format and test scripts. Additionally, they differ slightly - each has its own pyproject.toml
Description
The test scripts include:
Related issues/PRs
NA
Proposal
Create a script that checks a single baseline.
Explanation
Baselines have to be checked individually since they have different pyproject.tomls. Additionally, check performed on-baseline-basis will help to run only the CI test for the baseline that was modified (all baseline checks, according to me is not feasible to singe the env setup for a single baseline is ~3 minutes, and we plan on having 50 baselines).
The test and format work as follows:
./dev/test-baseline.sh
structure_exceptions=()
in test-baseline-structure.shpython -m pytest . || ([ $? -eq 5 ] || [ $? -eq 0 ])
The 5 return status means: no test detected. The 0 return status means correctly executed. We accept both.