-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Review how and which archive to extract in pipelines #251
Comments
pombredanne
added a commit
that referenced
this issue
Jul 23, 2021
Create issue to track extraction issue See #251 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
JonoYang
pushed a commit
that referenced
this issue
Jul 26, 2021
Create issue to track extraction issue See #251 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
JonoYang
pushed a commit
that referenced
this issue
Aug 2, 2021
Create issue to track extraction issue See #251 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
tdruez
added a commit
that referenced
this issue
Aug 4, 2021
* Use newer version of container libraries Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com> * Use new container-inspector structures Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com> * Add minimal support for Windows containers Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com> * Update Windows package getter * The windows_helper module from scancode is not available on pypi Signed-off-by: Jono Yang <jyang@nexb.com> * Use newer version of container libraries Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com> * Update call to windows_helper to win_reg Signed-off-by: Jono Yang <jyang@nexb.com> * Create new pipeline for Windows Docker images * Create Windows specific tag_uninteresting_windows_codebase_resources function Signed-off-by: Jono Yang <jyang@nexb.com> * Add function to find packages at well-known paths * Update tests Signed-off-by: Jono Yang <jyang@nexb.com> * Add step to tag known software in pipeline * Change name of Docker step from "find_images_linux_distro" to "find_images_os_and_distro" Signed-off-by: Jono Yang <jyang@nexb.com> * Get version from path in tag_known_software #238 * Update docstrings * Pin fetchcode dep Signed-off-by: Jono Yang <jyang@nexb.com> * Troubleshoot regex patterns #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Report Program File contents as packages #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Update Windows-specific regex * Add more file names and file extensions to be ignored * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com> * Do not ignore .mui files #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Filter using extension field rather than path #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Update scanpipe/pipes/docker.py Create issue to track extraction issue See #251 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com> * Fix scancode-toolkit pinned version in base.txt #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Create pipeline step to tag ignorable files #252 Signed-off-by: Jono Yang <jyang@nexb.com> * Update formatting #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Generalize regex expressions #238 * Modify regex used for Windows container analysis so it can be used outside the context of a Windows Docker image * Update tests Signed-off-by: Jono Yang <jyang@nexb.com> * Create new pipes for ignoring files #238 * Create pipes that ignore media files and data files with no clues * Update test results Signed-off-by: Jono Yang <jyang@nexb.com> * Add more file extensions to ignore #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Bump dep versions #238 Signed-off-by: Jono Yang <jyang@nexb.com> * Update docstring #238 * Use InstalledWindowsProgram object instead of Package Signed-off-by: Jono Yang <jyang@nexb.com> * Improve regex used in tag_known_software #238 * Update tests with more paths to test regex patterns Signed-off-by: Jono Yang <jyang@nexb.com> * Adjust code for consistency across the codebase #181 Signed-off-by: Thomas Druez <tdruez@nexb.com> * Address PR comments #238 * Use re.match instead of re.split * Rename WindowsDocker pipeline to DockerWindows * Set the default value of the q_objects argument for tag_installed_package_files to be a tuple Signed-off-by: Jono Yang <jyang@nexb.com> * Add is_media field to CodebaseResource #238 * Update test results Signed-off-by: Jono Yang <jyang@nexb.com> * Simplify tag_media_files_as_unintersting() #238 * Update test Signed-off-by: Jono Yang <jyang@nexb.com> * Refine windows pipes #238 Signed-off-by: Thomas Druez <tdruez@nexb.com> Co-authored-by: Jono Yang <jyang@nexb.com> Co-authored-by: Thomas Druez <tdruez@nexb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There are a few ways we deal with archives both at the pipeline input level and inside pipelines and many style of archives.
We should review and ensure we are using a consistent approach across pipes and pipelines for these
Also we are not doing much of anything beyond a shallow extraction of a project input which means we may miss out on certain metadata in some cases.
This is in the context of these issues and PRs:
(This is a follow up from https://github.com/nexB/scancode.io/pull/181/files#diff-56006e2ba488ba623c840bef5b9f94e07e9a37381af5befa665add2f2f6e13faR64 )
The text was updated successfully, but these errors were encountered: