split up the required dependencies by backend

[williballenthin]

right now, we install all dependencies for all (library) users:

capa/pyproject.toml

Lines 35 to 52 in 40d9587

	"tqdm==4.66.1",
	"pyyaml==6.0.1",
	"tabulate==0.9.0",
	"colorama==0.4.6",
	"termcolor==2.3.0",
	"wcwidth==0.2.8",
	"ida-settings==2.1.0",
	"viv-utils[flirt]==0.7.9",
	"halo==0.0.31",
	"networkx==3.1",
	"ruamel.yaml==0.17.35",
	"vivisect==1.1.1",
	"pefile==2023.2.7",
	"pyelftools==0.30",
	"dnfile==0.14.1",
	"dncil==1.0.2",
	"pydantic==2.1.1",
	"protobuf==4.23.4",

however, some users might not be using IDA while others might not use Ghidra/dnfile/BN/etc.

we should categorize the dependencies by backend and install only what is needed. for example pip install capa[ida,dncil]. core capa should require as few dependencies as possible.

[williballenthin]

    "tqdm==4.66.1",
    "pyyaml==6.0.1",
    "tabulate==0.9.0",
    "colorama==0.4.6",
    "termcolor==2.4.0",
    "wcwidth==0.2.13",
    "ida-settings==2.1.0",
    "viv-utils[flirt]==0.7.9",
    "halo==0.0.31",
    "networkx==3.1",
    "ruamel.yaml==0.18.5",
    "vivisect==1.1.1",
    "pefile==2023.2.7",
    "pyelftools==0.30",
    "dnfile==0.14.1",
    "dncil==1.0.2",
    "pydantic==2.4.0",
    "protobuf==4.23.4",

core:
    "pyyaml==6.0.1",
    "pydantic==2.4.0",
    "protobuf==4.23.4",
    "networkx==3.1",     ; used only for loop detection

utils:
    "ruamel.yaml==0.18.5",  ; capafmt

cli:
    "tqdm==4.66.1",
    "tabulate==0.9.0",
    "colorama==0.4.6",
    "termcolor==2.4.0",
    "wcwidth==0.2.13",  ; used for packaging tabulate, see: https://github.com/mandiant/capa/blob/a3a8e36911cb0f4711a465d6b1db0ff7d6735056/.github/pyinstaller/pyinstaller.spec#L23-L29
    "halo==0.0.31",     ; spinners, dead project

vivisect:
    "viv-utils[flirt]==0.7.9",
    "vivisect==1.1.1",

ida:
    "ida-settings==2.1.0",

dotnet:
    "dnfile==0.14.1",
    "dncil==1.0.2",

pefile:
    "pefile==2023.2.7",

elffile
    "pyelftools==0.30",

[williballenthin]

currently importing capa.main requires viv due to reference in elf.py

[mr-tz]

We should then also test all install configurations... 😟

[williballenthin]

I think we should at least test "with everything" and "with nothing".

[williballenthin]

i'm not sure if its possible to provide a default collection, like:

pip install flare-capa

but if the user provides something more specific, use that instead, like:

pip install flare-capa[core,utils,pefile-backend]

otherwise, its a breaking change, and we'll have to encourage users to use pip install flare-capa[all] unless they have special needs.

[mr-tz]

so we should keep this as simple as possible, e.g.:

• core
• ui
• optional

then we could have checks in the code that have fallbacks if an UI/optional module is not available

[williballenthin]

I think it's been a good exercise to review the dependencies and update documentation (esp. with pyproject.toml). Given that there are no external requests to split up the dependencies into collections, I don't think we should pursue that at this time. It seems possible to do, but it's a substantial breaking change, and there's not an immediate tangible benefit.

I plan to open a PR shortly with some tweaks to importing (deferring some imports to closer to their usage) and documenting the dependencies with pyproject.toml. Then I'll propose that we close out this issue.

[mr-tz]

sounds good!