Added detailed test coverage report. #513
Conversation
dblock
requested review from
harshavamsi,
sachetalva,
nhtruong,
Xtansia,
VachaShah,
Tokesh and
aabeshov
as code owners
Signed-off-by: dblock <dblock@amazon.com>
dblock
force-pushed
the
test-coverage-report
branch
from
18945cc
to
695a3f4
Compare
Changes AnalysisCommit SHA: 74b4b99 API ChangesSummaryNO CHANGES ReportThe full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10408668486/artifacts/1816987687 API Coverage
|
Spec Test Coverage Analysis
|
TESTING_GUIDE.md
Outdated
| ```json | ||
| { | ||
| "evaluated_paths_count": 214, | ||
| "paths_count": 550, |
There was a problem hiding this comment.
Nit: "paths_count" is ambiguous. Can we do total_paths_count or exposed_paths_count?
There was a problem hiding this comment.
Possibly, but this is existing code, so let's not scope creep it.
The spec test coverage is defined as follows.
export interface SpecTestCoverage {
paths_count: number
evaluated_paths_count: number,
evaluated_paths_pct: number
}
In this context the paths count represents the spec, so adding spec_ would duplicate these semantics.
tools/src/tester/ResultLogger.ts
Outdated
| log_coverage_report(results: TestResults): void { | ||
| console.log() | ||
| console.log(`${results.unevaluated_paths().length} paths remaining.`) | ||
| const groups = _.groupBy(results.unevaluated_paths(), (path) => path.split(' ', 2)[1].split('/')[1]) |
There was a problem hiding this comment.
I was wondering where the space ' ' character came from because an URL path doesn't contain any space character. Then I saw this. The combination of an http verb and an url path constitutes an API operation. I think we should name these entities unevaluated_operations and such to avoid confusion.
There was a problem hiding this comment.
Ok, that's a good suggestion, I'll do a bulk rename.
There was a problem hiding this comment.
I've refactored the path parsing into an Operation object, too.
Signed-off-by: dblock <dblock@amazon.com>
dblock
force-pushed
the
test-coverage-report
branch
from
fdb7f18
to
74b4b99
Compare
Description
Added a new
--coverage-reportoption that displays a hierarchy of paths that are not tested. This is helpful to see if an entire API is tested vs. what's missing.For example, the
/_aliasAPI is missing 4 tests and/_pluginsare missing 46, including 11 in/_plugins/_knn.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.