Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add summaries of scans #1076

Closed
wants to merge 68 commits into from
Closed

[WIP] Add summaries of scans #1076

wants to merge 68 commits into from

Conversation

pombredanne
Copy link
Contributor

This adds summaries for Language, Copyrights and Licenses
see #377 #1043

@codecov
Copy link

codecov bot commented May 31, 2018

Codecov Report

Merging #1076 into develop will decrease coverage by 6.24%.
The diff coverage is 57.05%.

Impacted file tree graph

@@            Coverage Diff             @@
##           develop   #1076      +/-   ##
==========================================
- Coverage    84.54%   78.3%   -6.25%     
==========================================
  Files          105     111       +6     
  Lines        11711   12191     +480     
==========================================
- Hits          9901    9546     -355     
- Misses        1810    2645     +835
Impacted Files Coverage Δ
src/cluecode/copyrights_hint.py 100% <ø> (ø) ⬆️
src/summarycode/category.py 0% <0%> (ø)
src/formattedcode/output_csv.py 15.83% <0%> (-80.57%) ⬇️
src/formattedcode/utils.py 100% <100%> (ø)
src/scancode/resource.py 83.15% <100%> (+0.72%) ⬆️
src/scancode/api.py 93.23% <100%> (+0.1%) ⬆️
src/scancode/plugin_copyright.py 100% <100%> (ø) ⬆️
src/formattedcode/output_spdx.py 29.05% <25%> (-58.82%) ⬇️
src/formattedcode/output_jsonlines.py 51.61% <25%> (-48.39%) ⬇️
src/summarycode/facet.py 35.41% <35.41%> (ø)
... and 50 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9845ea8...edaae8a. Read the comment docs.

@pombredanne pombredanne force-pushed the 377-1043-summaries branch 2 times, most recently from 416f18f to 29cd8bb Compare June 8, 2018 12:14
 * also rename CLI option
 * add tests


Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 * this way this can run from a virtual codebase too

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This is very basic at the moment.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 * instead add an error at the codebase level and keep trucking

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 * "fingerprints" strips bracketed words
 * see https://github.com/alephdata/fingerprints/blob/be2827e329f29f189f40ca2c3d8b6524be20a4e9/tests/test_fingerprints.py#L39
 * Therefore we need to ensure we do not process empty fingerprints
  in some corner cases
 * Also add tracing, improve exceptions handling and error messages

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
See also
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 * this will allow to support facets where one file can be in multiple
   facets

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 * this is generally useful when builoding plugins with options

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
facets are based on path only

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Also move deprecated copyrights.detect() function to test code.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* remove the returning of solo years: this is not used anywhere
* optionally exclude year from returned copyright
* handle some corner cases found in license texts
* add new function for future reorg of returned data structure for #255
  for now, not yet used as functions detect_copyrights2() and
  CopyrightDetector.detect2()


Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
The counters are not a summary

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
It now supports the new copyright data structure

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
It now supports the new copyright data structure

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Manintaining this out of tree duplicated script is too much work
Instead we can use the --from-json and--csv options to the same effect

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Also detect "created by" author statements #1120

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This is designed to store transient data that are not returned
in the scan results.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Also rename the copyright-summary option

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 - there is now a single summary option that summarizes whichever scan
 is available from the copyrights, licenses, programming language
 - the summary is report either as a new codebase-level attribute
 or as both codebase-level and file/directory level when using
 --summary-with-details
 - only json output support summaries for now

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 * Fix test failures (from unstable sort order)
 * Refactor common code where relevant
 * Other minor refinements

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This is following a review of expected failures

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This computes the distance as the number of segments that separate
a Resource from the root of the codebase

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Moved over from scancode-toolkit-contrib and refined accordingly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 This adds four new per-file flags:
 - is_legal True is the file is a legal-like file
 - is_manifest True is the file is a package manifest-like file
 - is_readme True is the file is a readme file


 - is_top_level True if the file is a top-level file

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Copy link
Contributor Author

replaced by #1130

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant