-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added detailed test coverage report. #513
Added detailed test coverage report. #513
Conversation
Signed-off-by: dblock <dblock@amazon.com>
18945cc
to
695a3f4
Compare
Changes AnalysisCommit SHA: 74b4b99 API ChangesSummaryNO CHANGES ReportThe full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10408668486/artifacts/1816987687 API Coverage
|
Spec Test Coverage Analysis
|
TESTING_GUIDE.md
Outdated
```json | ||
{ | ||
"evaluated_paths_count": 214, | ||
"paths_count": 550, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: "paths_count" is ambiguous. Can we do total_paths_count
or exposed_paths_count
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly, but this is existing code, so let's not scope creep it.
The spec test coverage is defined as follows.
export interface SpecTestCoverage {
paths_count: number
evaluated_paths_count: number,
evaluated_paths_pct: number
}
In this context the paths count represents the spec, so adding spec_
would duplicate these semantics.
tools/src/tester/ResultLogger.ts
Outdated
log_coverage_report(results: TestResults): void { | ||
console.log() | ||
console.log(`${results.unevaluated_paths().length} paths remaining.`) | ||
const groups = _.groupBy(results.unevaluated_paths(), (path) => path.split(' ', 2)[1].split('/')[1]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was wondering where the space ' '
character came from because an URL path doesn't contain any space character. Then I saw this. The combination of an http verb and an url path constitutes an API operation. I think we should name these entities unevaluated_operations
and such to avoid confusion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, that's a good suggestion, I'll do a bulk rename.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've refactored the path parsing into an Operation
object, too.
Signed-off-by: dblock <dblock@amazon.com>
fdb7f18
to
74b4b99
Compare
Description
Added a new
--coverage-report
option that displays a hierarchy of paths that are not tested. This is helpful to see if an entire API is tested vs. what's missing.For example, the
/_alias
API is missing 4 tests and/_plugins
are missing 46, including 11 in/_plugins/_knn
.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.