Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collect as-installed system and application packages #2058

Closed
pombredanne opened this issue Jun 2, 2020 · 3 comments
Closed

Collect as-installed system and application packages #2058

pombredanne opened this issue Jun 2, 2020 · 3 comments
Assignees
@steven-esser
Labels
new feature package scan package-at-toplevel package-files
Milestone
v31.0

Comments

@pombredanne
Copy link
Contributor

We can detect a good bunch of as-installed packages through their manifests.
This is about something a tad different to collect as-installed package details based on either:

  • specific layouts and installation manifests (such as a Python site-packages and dist-info or a Rubygems vendor or npm's node_module)
  • installation databases such as Debian dpkg database files, RPM installed db, alpine installed, Arch metadata, etc.

In addition to getting this data we should also collect the files corresponding to each package as reported in #437 and #1554

Some related tickets: #2023 , #253
@pombredanne
Copy link
Contributor Author

@MaJuRG ping ^ ... your take and help would be mucho welcomed there, starting with Debian (and debut?)

steven-esser added a commit that referenced this issue Jun 4, 2020
@steven-esser
Add support for dpkg status files
3ac57cf 
* Add debut dependency and bump attrs version
* Add function to parse a dpkg status file into debian Packages

Addresses: #2058

Signed-off-by: Steven Esser <sesser@nexb.com>
@pombredanne pombredanne mentioned this issue Jun 4, 2020
Closed
steven-esser added a commit that referenced this issue Jun 5, 2020
@steven-esser
Add additional functionality to DebianPackage
537672e 
* Add ability to return a list of paths and md5sums from a dpkg .md5sums
  file
* Add proper `parties` data to build_package()
* Add proper `source` data to build_packages()
* Add additional test cases

Addresses: #2058

Signed-off-by: Steven Esser <sesser@nexb.com>
steven-esser added a commit that referenced this issue Jun 5, 2020
@steven-esser
Improve DebianPackage api functionality
c296300 
* Fix multi-arch mapping bug
* Add end-to-end integration test
* Add get_installed_packages() function for a given rootfs dir location
* Add basic test case for get_installed_packages()

Addresses: #2058

Signed-off-by: Steven Esser <sesser@nexb.com>
steven-esser added a commit that referenced this issue Jun 8, 2020
@steven-esser
Handle cases where the md5sum file does not exist for a DebianPackage
a66e4cd 
* get_list_of_installed_files() now returns a list instead of a
  generator
* Tests added for missing md5sum file

Addresses: #2058

Signed-off-by: Steven Esser <sesser@nexb.com>
pombredanne added a commit that referenced this issue Jun 9, 2020
@pombredanne
Add missing test file #2058
dbf6c9d 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Jun 9, 2020
@pombredanne
Add new test and fix existing ones #2058
067b490 
Somehow a test file was damaged. This restore the test and expectations
to what they should be. And add a new test.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Jun 9, 2020
@pombredanne
Optionally collect license with "try_as_expression" #2058
3d423fe 
get_normalized_expression() was always trying to parse a license
as an exception. This is not an option.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Jun 9, 2020
@pombredanne
Detect licenses in debian copyright files #2058
3e0f277 
We now can detecte license "smartly" in Debian copyright files
using a combo of license name mappings, and parsngs dep-5 debian machine
reable copyright files and treat each section correctly.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Jun 9, 2020
@pombredanne
Do license detection of installed debian package #2058
17eb2f6 
Also add a few end to end tests on a mock rootfs

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
JonoYang added a commit that referenced this issue Jul 7, 2020
@JonoYang
Update debut and its about files #2058
d89c8c6 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 7, 2020
@JonoYang
Get installed alpine packages #2058
f8e8717 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 8, 2020
@JonoYang
Allow kwargs in get_installed_packages #2058
c6df6d9 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 8, 2020
@JonoYang
Add installed_files field to AlpinePackage #2058
86196f2 
Signed-off-by: Jono Yang <jyang@nexb.com>
pombredanne added a commit that referenced this issue Jul 15, 2020
@pombredanne
Improve ABOUT file documentation for debut #2058
d4e314f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
Move and rename InstalledFile to models.py #2058
a65e47c 
The new name is PackageFile and we also support more than just md5 as
checksums.
Also move common test code to package_test_utils.py

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
Allow comments in debian license mappings #2058
3f7b59e 
Comments lines can start with #

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
Improve Alpine installed packages support #2058
7cc381c 
Collect the list of installed files, dependencies and additional data.
Collect and normalize licenses and add basic tests.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
Ad new license rule for CC-BY #2058
a5ecfab 
Found in a binary on Alpine

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
Enable license detection bug with env var #2058
1465c64 
Setting SCANCODE_DEBUG_LICENSE enables tracing

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
Fix failing debian tests from latest refactorings #2058
e99c101 
debian.py is now its own module

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 3, 2020
@pombredanne
File with colon are not legal on Windows #2058
58dfa73 
Therefore we sip these tests there entirely.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 4, 2020
@pombredanne
File with colon are not legal on Windows #2058
12f6416 
Therefore we sip these tests there entirely.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 4, 2020
@pombredanne
Do not report dirs as debian installed files #2058
4874847 
* Fall back to /var/lib/dpkg/info .list files when the .md5sum is not
  present.

* Avoid including directories either because it is a well known root
  directory or because this is a parent of an existing file.

* Always prefix file paths with / since we are in a rootfs

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 6, 2020
@pombredanne
Bump pip and setuptools #2058
0ee5112 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Aug 25, 2020
@pombredanne
Support lists and md5sum for debian packages #2058
95131d8 
Somehow sometimes things are not clean and clear as we liked them to be
and md5sums could be missing

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne pombredanne mentioned this issue Sep 1, 2020
Closed
pombredanne added a commit that referenced this issue Sep 7, 2020
@pombredanne
Rename rules to avoid overlaping names #2058
ecece24 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 7, 2020
@pombredanne
Remove duplicated mongodb-sspl rules #2058
5b4596c 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Rename Debian test files to .expected.json #2058
5f049c9 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Move Debian expected files close to tests #2058
dc00ae4 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Add new license rules for linux firmware #2058
487c3b7 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Improve Debian copyright tests #2058
7a9eb3f 
And run Debian test only on Python3 #2058

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Do not always dedupe Debian licenses #2058
6ee677d 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Improve Debian licenses mappings #2058
a5037e2 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Update Linux proprietary license expectations #2058
b7bd395 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Do not run debian tests on Py2 #2058
36708ed 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Sep 8, 2020
@pombredanne
Do not run alpine tests on Py2 #2058
6e7a321 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit to aboutcode-org/scancode-plugins that referenced this issue Feb 17, 2021
@pombredanne
Add new pre-built RPM binary plugin
745ccc9 
This is to support these tickets:

aboutcode-org/scancode-toolkit#437
aboutcode-org/scancode.io#6
aboutcode-org/scancode-toolkit#2058

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne pombredanne mentioned this issue Feb 19, 2021
Open
pombredanne added a commit that referenced this issue Mar 1, 2021
@pombredanne
Rename to rpm_installed #2058
c136fe6 
 - adopt laest license headers
 -

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Mar 3, 2021
@pombredanne
Rename RPM test files #2058
ec5e911 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Mar 3, 2021
@pombredanne
Move installed_files to Package model #2058
eb52eb5 
And add size to PackageFile model

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Mar 3, 2021
@pombredanne
Collect RPM installed packages #2058
232a2ac 
- Use common RpmPackage model
- Add license detection
- Use rpm command to collect XMLish
- Other minor refactorings

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne added a commit that referenced this issue Mar 4, 2021
@pombredanne
Use proper function name #2058
702c492 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne pombredanne mentioned this issue Mar 5, 2021
Merged
4 tasks
@pombredanne pombredanne added this to the v31.0 milestone Sep 24, 2021
@pombredanne
Copy link
Contributor Author

This is now all in develop

@pombredanne
Copy link
Contributor Author

This is now all in develop, merged.

pombredanne pushed a commit that referenced this issue May 9, 2022
@jeanas
Remove now redundant re.UNICODE and (?u) (#2058)
5e75009
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@steven-esser steven-esser

Labels
new feature package scan package-at-toplevel package-files
Projects
None yet
Milestone
v31.0
Development

No branches or pull requests
2 participants
@pombredanne @steven-esser