-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test Framework #299
Test Framework #299
Conversation
API specs implemented for 313/649 (48%) APIs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good overall! Needs READMEs, tests, and all the TODOs.
Left small usage/naming comments.
Try to get something that runs under CI first in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested this out.
Changes AnalysisCommit SHA: 078326f API ChangesSummary
ReportThe full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/9291588290/artifacts/1549682775 API Coverage
|
Signed-off-by: Theo Truong <theotr@amazon.com>
This is ready for the view and merged so people can start adding tests. I'll handle the remaining work in followup PRs:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM! @Xtansia any objections to merging as is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's still some parts I think could be improved, but not against merging as is to iterate on it
.addOption(new Option('--spec, --spec_path <path>', 'path to the root folder of the multi-file spec').default('./spec')) | ||
.addOption(new Option('--tests, --tests_path <path>', 'path to the root folder of the tests').default('./tests')) | ||
.addOption(new Option('--tab_size <size>', 'tab size for displayed results').default('4')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CLI args are generally kebab-case. Additionally I'd say it's more commonly referred to as a tab width than a tab size.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
THanks for the callout. Will update them in the next PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#301 with tests
export function overall_result (evaluations: Evaluation[]): Result { | ||
if (evaluations.some(e => e.result === Result.ERROR)) return Result.ERROR | ||
if (evaluations.some(e => e.result === Result.FAILED)) return Result.FAILED | ||
if (evaluations.some(e => e.result === Result.SKIPPED)) return Result.SKIPPED |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this correct? If I have 24 results, 23 passed but 1 skipped, then the overall result is skipped?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is for evaluating components results. A component is skipped when a previous component results in an error (say if a chapter could not be executed for validation, all subsequent chapters will be skipped). However that would result in ERROR
for overall. So for now line 6 is actually redundant.
Hypothetically when we allow the user to skip some component evaluation (say skip evaluating the response body of a new endpoint still being developed), I think we should still mark the parent component as skipped. The print out will show all subcomponents so you can tell which subcomponent causes the SKIPPED
status for the parent component. So:
- If you see ERROR as the result for a component, at least one of its subcomponent has ERROR
- otherwise, If you see FAILED as the result for a component, at least one of its subcomponent has FAILED
- otherwise, If you see SKIPPED as the result for a component, at least one of its subcomponent has SKIPPED
- else, all components have been vested and passed the test.
This PR adds a custom test framework for OpenSearch Open API Spec. Each YAML file in the
./tests
folder (and its sub-folders) is a test story with 3 main components:A JSON Schema for the test stories is also provided so that the IDE can help developers with autocomplete and validation when drafting the stories.
Make sure you have an OpenSearch cluster running at
http://localhost:9200
before running the following command to initiate the test suiteBellow is the result of running
npm run test:spec -- --ignore_passed N
Check start.ts to see the options that can be passed to
npm run test:spec
command.Remaining work:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.