Skip to content

Commit

Permalink
Merge branch 'main' into 6-rpm-support
Browse files Browse the repository at this point in the history
Signed-off-by: Thomas Druez <tdruez@nexb.com>

# Conflicts:
#	etc/requirements/base.txt
  • Loading branch information
tdruez committed Apr 2, 2021
2 parents 40f7145 + 94be24c commit 67edc8f
Show file tree
Hide file tree
Showing 62 changed files with 1,738 additions and 1,226 deletions.
21 changes: 21 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,27 @@

### v1.1.1 (unreleased)

- Add a new codebase resource detail view including the file content.
Detected value can be displayed as annotation in the file source.
https://github.com/nexB/scancode.io/issues/102

- Download URLs can be provided as inputs on the project form.
Each URL is fetched and added to the project input directory.
https://github.com/nexB/scancode.io/issues/100

- Run celery worker with the "threads" pool implementation.
Implement parallelization with ProcessPoolExecutor for file and package scans.
Add a SCANCODE_PROCESSES settings to control the multiprocessing CPUs count.
https://github.com/nexB/scancode.io/issues/70

- Optimize "tag" type pipes using the update() API in place of save() on the QuerySet
iteration.
https://github.com/nexB/scancode.io/issues/70

- Use the extractcode API for the Docker pipeline.
This change helps with performance and results consistency between pipelines.
https://github.com/nexB/scancode.io/issues/70

- Implement cache to prevent scanning multiple times a duplicated codebase resource.
https://github.com/nexB/scancode.io/issues/70

Expand Down
5 changes: 3 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ services:
build: .
command: celery --app scancodeio worker
--loglevel=INFO
--soft-time-limit=21600 --time-limit=22000
--concurrency 1 --pool threads
--events -Ofair --prefetch-multiplier=1
--soft-time-limit=21600 --time-limit=22000
env_file:
- docker.env
volumes:
Expand All @@ -32,7 +33,7 @@ services:
command: sh -c "
./manage.py migrate &&
./manage.py collectstatic --no-input &&
gunicorn scancodeio.wsgi:application --bind :8000 --timeout 600 --workers 3"
gunicorn scancodeio.wsgi:application --bind :8000 --timeout 600 --workers 2"
env_file:
- docker.env
expose:
Expand Down
2 changes: 1 addition & 1 deletion docs/docker-image.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Run your image as a container::
At this point, the ScanCode.io app should be running at port 80 on your Docker host.
Go to http://localhost/ on a web browser to access the web UI.

You can also run a one-off ``scanpipe`` command through the Docker command line
You can also execute a one-off ``scanpipe`` command through the Docker command line
interface, for example::

docker-compose run web ./manage.py create-project project_name
Expand Down
40 changes: 27 additions & 13 deletions docs/scanpipe-command-line.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ ScanPipe's own commands are listed under the ``[scanpipe]`` section::
create-project
graph
output
run
execute
show-pipeline


Expand All @@ -34,10 +34,11 @@ Display help for the provided subcommand.
For example::

$ scanpipe create-project --help
usage: scanpipe create-project [--pipeline PIPELINES] [--input INPUTS] [--execute] name
usage: scanpipe create-project [--input-file INPUTS_FILES]
[--input-url INPUT_URLS] [--pipeline PIPELINES] [--execute] name

Create a ScanPipe project.

positional arguments:
name Project name.

Expand All @@ -50,27 +51,40 @@ be unique.

Optional arguments:

- ``--pipeline PIPELINES`` Pipelines names to add on the project.
- ``--pipeline PIPELINES`` Pipelines names to add on the project.

- ``--input-file INPUTS_FILES`` Input file locations to copy in the :guilabel:`input/`
work directory.

- ``--input INPUTS`` Input file locations to copy in the :guilabel:`input/` workspace
- ``--input-url INPUT_URLS`` Input URLs to download in the :guilabel:`input/` work
directory.

- ``--execute`` Execute the pipelines right after project creation.
- ``--execute`` Execute the pipelines right after project creation.

.. warning::
The pipelines are added and will be executed in the order of the provided options.

`$ scanpipe add-input --project PROJECT <input ...>`
----------------------------------------------------

Copy the file found at the ``<input>`` path to the project named ``PROJECT`` workspace
:guilabel:`input/` directory.
You can use more than one ``<input>`` to copy multiple files at once.
`$ scanpipe add-input --project PROJECT [--input-file FILES] [--input-url URLS]`
--------------------------------------------------------------------------------

Add input files in a project work directory.

- ``--input-file INPUTS_FILES`` Input file locations to copy in the :guilabel:`input/`
work directory.

- ``--input-url INPUT_URLS`` Input URLs to download in the :guilabel:`input/` work
directory.

For example, assuming you have created beforehand a project named "foo", this will
copy ``~/docker/alpine-base.tar`` to the foo project :guilabel:`input/` directory::

$ scanpipe add-input --project foo ~/docker/alpine-base.tar
$ scanpipe add-input --project foo --input-file ~/docker/alpine-base.tar

You can also provide URLs of files to be downloaded to foo project :guilabel:`input/`
directory::

$ scanpipe add-input --project foo --input-url https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar


`$ scanpipe add-pipeline --project PROJECT PIPELINE_NAME [PIPELINE_NAME ...]`
Expand All @@ -80,7 +94,7 @@ Add the ``PIPELINE_NAME`` to the provided ``PROJECT``.
You can use more than one ``PIPELINE_NAME`` to add multiple pipelines at once.

.. warning::
The pipelines are added and will be running in the order of the provided options.
The pipelines are added and will be executed in the order of the provided options.

For example, assuming you have created beforehand a project named "foo", this will
add the docker pipeline to your project::
Expand Down
2 changes: 1 addition & 1 deletion docs/scanpipe-concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Pipelines
---------

A pipeline is a Python script that contains a series of steps from start to end
to run in order to **perform a code analysis**.
to execute in order to **perform a code analysis**.

It usually starts from the uploaded input files, and may extract these then
generates ``CodebaseResource`` records in the database accordingly.
Expand Down
16 changes: 8 additions & 8 deletions docs/scanpipe-tutorial-1.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Step-by-step

- Add the test Docker image tarball to the project workspace's :guilabel:`input/` directory::

$ scanpipe add-input --project staticbox ~/30-alpine-nickolashkraus-staticbox-latest.tar
$ scanpipe add-input --project staticbox --input-file ~/30-alpine-nickolashkraus-staticbox-latest.tar

.. note::
The command output will let you know where is the project workspace :guilabel:`input/` directory
Expand All @@ -39,29 +39,29 @@ Step-by-step

- Add the docker pipeline to your project::

$ scanpipe add-pipeline --project docker
$ scanpipe add-pipeline --project staticbox docker

- Check that the docker pipeline was added to your project::

$ scanpipe show-pipeline --project staticbox

.. note::
The ``scanpipe show-pipeline`` command lists all the pipelines added to the
project and their planned runs.
project and their planned execution.
You can use this to get a quick overview of the pipelines that have been running already
(with their "SUCCESS" or "FAILURE" status) and those that will be running next.

- Run the docker pipeline on this project::

$ scanpipe run --project staticbox
$ scanpipe execute --project staticbox

- Executing the ``show-pipeline`` command again will confirm the success of the
pipeline run::
pipeline execution::

$ scanpipe show-pipeline --project staticbox
"[SUCCESS] docker"

- Get the results of the pipeline run as a JSON file using the ``output`` command::
- Get the results of the pipeline execution as a JSON file using the ``output`` command::

$ scanpipe output --project staticbox --format json

Expand All @@ -75,10 +75,10 @@ Step-by-step
An ``execute`` option is also available to start the pipeline execution right
after the project creation.
For example, the following command will create a project named ``staticbox2``,
copy the test docker image to the project's inputs, add the docker pipeline,
download the test docker image to the project's inputs, add the docker pipeline,
and execute the pipeline in one operation::

$ scanpipe create-project staticbox2 \
--input ~/30-alpine-nickolashkraus-staticbox-latest.tar \
--input-url https://github.com/nexB/scancode.io-tutorial/releases/download/sample-images/30-alpine-nickolashkraus-staticbox-latest.tar \
--pipeline docker \
--execute
4 changes: 2 additions & 2 deletions docs/scanpipe-tutorial-2.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@ Step-by-step
add the ``scan_codebase`` pipeline, and execute it::

$ scanpipe create-project asgiref \
--input ~/asgiref-3.3.0-py3-none-any.whl \
--input-file ~/asgiref-3.3.0-py3-none-any.whl \
--pipeline scan_codebase \
--execute

.. note::
The content of the :guilabel:`input/` directory will be copied in the
:guilabel:`codebase/` directory where ``extractcode`` will be run before
:guilabel:`codebase/` directory where ``extractcode`` will be executed before
running ``scancode``.
Alternatively, the codebase content can be manually copied to the
:guilabel:`codebase/` directory in which case the ``--input`` option can be
Expand Down
4 changes: 2 additions & 2 deletions etc/nginx/conf.d/default.conf
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
client_max_body_size 100M;
proxy_read_timeout 600s;
client_max_body_size 10G;
proxy_read_timeout 600s;
}

location /static/ {
Expand Down
27 changes: 27 additions & 0 deletions etc/nginx/examples/ssl.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
upstream gunicorn_app {
server web:8000;
}

server {
listen 80;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl;
ssl_certificate /etc/nginx/conf.d/fullchain1.pem;
ssl_certificate_key /etc/nginx/conf.d/privkey1.pem;

location / {
proxy_pass http://gunicorn_app;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
client_max_body_size 10G;
proxy_read_timeout 600s;
}

location /static/ {
alias /var/scancodeio/static/;
}
}
12 changes: 6 additions & 6 deletions etc/requirements/base.txt
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Configuration tools
importlib_metadata==3.7.0; python_version < "3.8"
importlib_metadata==3.10.0; python_version < "3.8"

# Django related
Django==3.1.7
django-environ==0.4.5
django-crispy-forms==1.11.1
django-crispy-forms==1.11.2
pytz==2021.1
django-filter==2.4.0
djangorestframework==3.12.2
djangorestframework==3.12.4

# Database
psycopg2==2.8.6; sys_platform == "linux"
Expand All @@ -21,16 +21,16 @@ celery==5.0.5
redis==3.5.3

# WSGI server
gunicorn==20.0.4
gunicorn==20.1.0

# Docker
container_inspector>=3.1.2

# ScanCode-toolkit
scancode-toolkit>=21.2.25
scancode-toolkit==21.3.31

# Utilities
XlsxWriter==1.3.7
XlsxWriter==1.3.8

# Profiling
pyinstrument==3.4.1
9 changes: 3 additions & 6 deletions etc/requirements/dev.txt
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
-r base.txt

# Code validation
pycodestyle==2.6.0
pycodestyle==2.7.0
black==20.8b1
isort==5.7.0
isort==5.8.0

# Documentation
Sphinx==3.5.1
Sphinx==3.5.3
sphinx_rtd_theme==0.5.1

# Release
bump2version==1.0.1

# Dependencies lock file
pip_tools==5.5.0
Loading

0 comments on commit 67edc8f

Please sign in to comment.