Collect and parse as-installed Alpine Linux packages

[pombredanne]

As part of #2058 we would need to collect as-installed Alpine packages (and that is in addition to support .apk files in #1803 )
We already have a base for that since @chinyeungli shared this with me:

 #
#  Copyright (c) 2019 nexB Inc. and others. All rights reserved.
#

import click
import csv
import posixpath

# This utility will only parse the P: (Package), V: (Version), U: (URL) and L: (License)
# from the installed text file
def parse_alpine_apk(input, output):
    with open(input) as f:
        content = f.readlines()
        installed_list = []
        package_data = []
        for x in content:
            # Remove white space or newline char
            x = x.strip()
            if x.startswith('P:'):
                if package_data:
                    installed_list.append(package_data)
                package_data = []
                data = x.partition(':')[2]
                package_data.append(data)
            if x.startswith('V:'):
                data = x.partition(':')[2]
                package_data.append(data)
            if x.startswith('U:'):
                data = x.partition(':')[2]
                package_data.append(data)
            if x.startswith('L:'):
                data = x.partition(':')[2]
                package_data.append(data)
        # Append the last package to the installed_list
        if package_data:
                    installed_list.append(package_data)

    with open(output, 'wb') as csvfile:
        cwriter = csv.writer(csvfile, delimiter=',')
        header_row = ['Package', 'Version', 'URL', 'License']
        cwriter.writerow(header_row)
        for r in installed_list:
            cwriter.writerow(r)

    print("FINISHED!!")

@click.command()
@click.argument('input', type=click.Path(exists=True, readable=True))
@click.argument('output', type=click.Path(exists=False), required=True)
def cli(input, output):
    parse_alpine_apk(input, output)

with this documentation by @johnmhoran :

================
Parse Alpine APK
================

Usage
=====

.. code-block:: none

   Usage: parse_alpine_apk [OPTIONS] LOCATION DESTINATION

     Command Line API to parse the apline_apk installed file and store to a CSV file.

   Options:
     --help  Show this message and exit.

Example
=======

.. code-block:: none

   parse_alpine_apk /code/project/layer.tar-extract/lib/apk/db/installed /audit_workspace/scans/installed_packages.csv

Notes
=====

This installed file is usually located at ``/lib/apk/db/installed`` under the ``layer.tar``. This file shows what packages (including dependencies) are installed.

This utility will only parse the P: (Package), V: (Version), U: (URL) and L: (License) from the installed text file.

The format of a installed file is a set of KEY=. A sample file would look like:

.. code-block:: none

   C:Q1t20ETDkiZIy7BdegtC/woQiNnuE=
   P:musl
   V:1.1.19-r10
   A:x86_64
   S:372275
   I:602112
   T:the musl c library (libc) implementation
   U:http://www.musl-libc.org/
   L:MIT
   o:musl
   m:Timo Teräs <timo.teras@iki.fi>
   t:1529394495
   c:ed42835662421a72dbc1c47397a2805306203860
   p:so:libc.musl-x86_64.so.1=1
   F:lib
   R:libc.musl-x86_64.so.1
   a:0:0:777
   Z:Q17yJ3JFNypA4mxhJJr0ou6CzsJVI=
   R:ld-musl-x86_64.so.1
   a:0:0:755
   Z:Q1lUgQ+IQG9688iKiZSRz3ITg7h/Y=
   F:usr
   F:usr/lib

Other useful sources include:

• the specs
  • https://wiki.alpinelinux.org/wiki/Apk_spec
  • https://wiki.alpinelinux.org/wiki/Alpine_package_format
• a nice library : https://code.foxkit.us/adelie/apkkit by @awilfox ... not sure if this is still maintained in the future
• https://github.com/privazio/alpine_release_info/
• https://github.com/knqyf263/apkindex-archive/blob/master/update.py#L111
• https://github.com/tungel/pmbootstrap/blob/master/pmb/parse/apkindex.py (which is a full fledged parser but is GPL-licensed)
• https://github.com/ownport/alpine-pkg-mirror/blob/f080d94a4582ef97b517fb2470c7d9c8979871bb/mirror/utils.py#L5 (apache-licensed)
• https://github.com/forward3d/alpinist (mit)
• https://github.com/pombredanne/code.foxkit.us-adelie-apkkit/blob/02c75f02551dff67e67a5823fe3015f3b7368435/apkkit/base/index.py
  See http://dl-cdn.alpinelinux.org/alpine/ for an index example
  http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/ for recent releases of apk

[awilfox]

apkkit is considered, for all intents and purposes, to be "complete". That is, if any bugs are found, we will fix them - but no new features are planned at this time.

[pombredanne]

apkkit is considered, for all intents and purposes, to be "complete". That is, if any bugs are found, we will fix them - but no new features are planned at this time.

@awilfox thanks you ++

[pombredanne]

@JonoYang you can also check this aboutcode-org/debian-inspector#3 where the debian parser (which is essentially the built stdlib email header parser actually works quite well)

[pombredanne]

@JonoYang the code actually handling Alpine could be quite similar to what in WIP for Debian in this PR https://github.com/nexB/scancode-toolkit/tree/2058-collect-system-packages and could likely be added on top

[pombredanne]

This has been completed and is available in the latest release ... follow up tickets are tracked separately