New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor and improve copyright holders detection #930

Closed

pombredanne opened this issue Feb 14, 2018 · 2 comments

Labels

copyright scan new feature

Contributor

pombredanne commented Feb 14, 2018

There are a few pending tickets on holders and this is going to be a master ticket for these.

Make the copyright holder value a single holder Make the copyright holder value a single holder #186
Authors scan in testfiles finds holder although it should not Authors scan in testfiles finds holder although it should not #216
Do not mix multiple copyrights and authors in the returned data Do not mix multiple copyrights and authors in the returned data #255
Describe ScanCode output 'copyrights' type Describe ScanCode output 'copyrights' type #618

And this was originally reported offline by email:

Any codebase that has any non-trivial number of contributors has often some issues with reporting clean holders.

Some examples of cleanups:

Leading and trailing punctuation
Inc vs Incorporated,
X and Y should be, in general, split into two separate entries so you don’t get X and Y, X and Z, Y and Z, …
There appear to be cases where the output has random > and other punctuation.

Some pointers for normalization: https://github.com/alephdata/fingerprints and https://github.com/datamade/probablepeople

pombredanne added copyright scan new feature labels

Contributor Author

pombredanne commented Feb 14, 2018

@tsteenbe @jeffmcaffer ping, just so you are kept abreast of the progress here :P

pombredanne added a commit that referenced this issue


          Add tests for holders to all copyright tests #930

158c503

 * this allow to to setup a good base for improvements

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add support to regen copyright tests #930

4dd2080

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve copyright detection #930

aa97355

 * ensure that POS tagging creates proper names and use this to improve
   holders reporting

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Implement holder deduplication with fingerprints #930

df97e28

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve copyright holder and summary detection #930

2377ee8

 * add tests for holders summaries
 * improve tagging of names
 * improve collection of holders and tracing

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

haikoschol mentioned this issue

Incorrect copyright detection #965

Closed

pombredanne added a commit that referenced this issue


          Improve copyright POS tagging #930

469b254

 * minor changes mostly from scanning several npms

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve replacement of HTML entities #930

4c664b2

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve copyright POS tagging #930

922d00e

 * minor changes mostly from scanning several npms

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve replacement of HTML entities #930

6ee9e06

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve copyright POS tagging #930

7e8c029

 * minor changes mostly from scanning several npms

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Improve replacement of HTML entities #930

a89b8f0

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this issue

Clean up copyrights in summarized data clearlydefined/service#59

Closed

pombredanne mentioned this issue

Improve copyright summary #1043

Closed

pombredanne added a commit that referenced this issue


          Streamline copyright detection #930

7e8f56b

* remove the returning of solo years: this is not used anywhere
* optionally exclude year from returned copyright
* handle some corner cases found in license texts
* add new function for future reorg of returned data structure for #255
  for now, not yet used as functions detect_copyrights2() and
  CopyrightDetector.detect2()


Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Streamline copyright detection #930

d3f9a93

* remove the returning of solo years: this is not used anywhere
* optionally exclude year from returned copyright
* handle some corner cases found in license texts
* add new function for future reorg of returned data structure for #255
  for now, not yet used as functions detect_copyrights2() and
  CopyrightDetector.detect2()


Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Contributor Author

pombredanne commented Aug 14, 2018

This has been done and merged in develop

pombredanne closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment