Introduce filters

Filters are functions that can be applied to inspect and transform the EDAM
before it gets handed off to Edalize

doc/source/user/build_system/filters.rst

@@ -0,0 +1,99 @@
+.. _ug_build_system_filters:
+
+Filters: Make system-wide modifications to the EDAM structure
+=============================================================
+
+The last thing FuseSoC does before handing over to Edalize, is to prepare an EDAM file containing a description of the complete system and everything that the EDA tools need to know. It is sometimes useful to make system-wide changes after the system is assembled, and this is where filters come in. Filters are additional tasks that can be run to analyze and modify the EDAM structure, and through that structure the filters have access to the complete dependency tree and all source files.
+
+FuseSoC ships with a few built-in filters for generally useful tasks:
+
+* autotype: Sets file types according to the file name prefix for files that don't have an explicit file type set.
+* custom: Runs the command specified with the environment variable `FUSESOC_CUSTOM_FILTER`. Two arguments are passed to the command. The first specifies the EDAM (yaml) file the custom command is supposed to read. The second specifies the name of the file that the filter should use to return the modified EDAM struct.
+* dot: Creates a GraphViz dot file of the dependency tree
+
+All filters are run from the work root directory.
+
+.. note::
+    Technically, an Edalize frontend can perform the exact same task as a FuseSoC EDAM filter. The difference is more philosophical in that a filter can be seen as something that fixes up the system before it is ready to be consumed by an EDA tool, while an Edalize frontend typically *is* an EDA tool. The filters will also possibly have access to more FuseSoC internals in the future.
+
+Using filters
+-------------
+
+There are three way to enable which filters to be applied. These three options have various use-cases.
+
+Filters in targets
+~~~~~~~~~~~~~~~~~~
+
+Filters specified in a target section of a core are typically required for the the build to work. In order to enable a filter for a target, add a list of filters using the `filters` key.
+
+.. code-block:: yaml
+
+   # An excerpt from a core file.
+   targets:
+     sim:
+       # ...
+       filters: [autotype, dot] # Apply the autotype and dot filters in that order
+
+Filters in the configuration file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Filters can be set in the configuration file as a list of space-separated strings in the `main` section. These can be set with the fusesoc config cli, e.g. `fusesoc config filters "filter1 filter2"`. Filters set in the config file are typically used when all targets of the cores in the workspace need some filter to be applied, or just as a convenienence for users who like to have som particular filter always enabled. These filters are applied after the ones from the core file targets.
+
+Filters on the command-line
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is also possible to set filters on the command-line. This is typically used for one-off filters, e.g. to generate some debug info. They are enabled with the `--filter` parameters, e.g. `fusesoc run --filter=dot --target=sim ...`. The `--filter` parameter can be specified multiple times to add more filters. Filters on the command-line are applied after the ones from the configuration file.
+
+
+Creating additional filters
+---------------------------
+
+A filter is a Python module that contains a class with the same name as the module, but capitalized. The class needs to implement a function called `run` which takes the EDAM struct as the first argumen and the work root as the second argument. The function needs to return the new EDAM struct even if it is unmodified.
+
+Filter template::
+
+    import logging
+    import os
+
+    logger = logging.getLogger(__name__)
+
+
+    class Customfilter:
+        def run(self, edam, work_root):
+            # Print file sizes of all verilog files in the system
+            for f in edam["files"]:
+                if f["file_type"].startswith("verilogSource"):
+                    size = os.path.getsize(os.path.join(work_root, f["name"]))
+                    print(f"{f['name']} is {size} bytes")
+
+            # Add an additional parameter
+            edam["parameters"]["my_number"] = {"datatype" : "int",
+                                               "paramtype": "vlogdefine",
+                                               "default" : 5446}
+
+            # Change the system name
+            edam["name"] = "bestsystemever"
+
+            # Return modified EDAM struct
+            return edam
+
+FuseSoC implements support for implicit namespace packages (https://peps.python.org/pep-0420/) This means that subclasses that logically belong to FuseSoC can be distributed over several physical locations and is something we can take advantage of to add new filters outside of the FuseSoC code base.
+
+In order to do that we will create a directory structure that mirrors the structure of FuseSoC like the example below::
+
+  externalplugin/
+      fusesoc/
+          filters/
+	      customfilter.py
+	      anothercustomfilter.py
+
+There are two common options for making the above `customfilter.py` and `anothercustomfilter.py` available to FuseSoC.
+
+The first way is to add the `externalplugin` path to ``PYTHONPATH``. The other is to add a `setup.py` in the `externalplugin` directory and install the filter plugin with pip as with other Python packages.
+
+A `setup.py` in its absolutely most minimal form is listed below and is enough to install the plugin as a package in development mode using ``pip install --user -e .`` from the `externalplugin` directory.::
+
+  from setuptools import setup
+  setup()
+
+A real `setup.py` like the one used by FuseSoC normally contains a lot more information.

doc/source/user/build_system/index.rst

@@ -35,6 +35,7 @@ A full reference documentation on the CAPI2 core file format can be found in the
    core_files.rst
    eda_flows.rst
    dependencies.rst
+   filters.rst
    flags.rst
    generators.rst
    hooks.rst

fusesoc/capi2/core.py

@@ -176,6 +176,10 @@ def get_flags(self, target_name):
             raise RuntimeError(f"'{self.name}' has no target '{target_name}'")
         return flags
 
+    def get_filters(self, flags):
+        target_name, target = self._get_target(flags)
+        return target.get("filters", [])
+
     def get_flow(self, flags):
         self._debug("Getting flow for flags {}".format(str(flags)))
         flow = None

fusesoc/capi2/json_schema.py

@@ -508,6 +508,13 @@
                 "type": "string"
               }
             },
+            "^filters(_append)?$": {
+              "description": "EDAM filters to apply",
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            },
             "^generate(_append)?$": {
               "description": "Parameterized generators to run for this target with optional parametrization",
               "type": "array",

fusesoc/config.py

@@ -153,6 +153,16 @@ def _arg_or_val(self, arg, val):
         else:
             return val
 
+    @property
+    def filters(self):
+        return self._cp.get(
+            Config.default_section, "filters", fallback=""
+        ).split() + getattr(self, "args_filters", [])
+
+    @filters.setter
+    def filters(self, val):
+        self._set_default_section("filters", val)
+
     @property
     def build_root(self):
         return self._arg_or_val("args_build_root", self._get_build_root())

fusesoc/edalizer.py

@@ -9,6 +9,7 @@
 import pathlib
 import shutil
 from filecmp import cmp
+from importlib import import_module
 
 from fusesoc import utils
 from fusesoc.capi2.coreparser import Core2Parser
@@ -89,6 +90,20 @@ def discovered_cores(self):
         """Get a list of all cores found by fusesoc"""
         return self.core_manager.db.find()
 
+    def apply_filters(self, global_filters):
+        filters = self.edam.get("filters", []) + global_filters
+        for f in filters:
+            try:
+                filter_class = getattr(
+                    import_module(f"fusesoc.filters.{f}"), f.capitalize()
+                )
+                logger.info(f"Applying filter {f}")
+                self.edam = filter_class().run(self.edam, self.work_root)
+            except ModuleNotFoundError:
+                logger.warning(f"Ignoring unknown EDAM filter '{f}'")
+            except Exception as e:
+                logger.error(f"Filter error: {str(e)}")
+
     def run(self):
         """Run all steps to create a EDAM file"""
 
@@ -224,6 +239,9 @@ def create_edam(self):
                     self.flags["tool"]: core.get_tool_options(_flags)
                 }
 
+            # Extract EDAM filters
+            snippet["filters"] = core.get_filters(_flags)
+
             # Extract flow options
             snippet["flow_options"] = core.get_flow_options(_flags)
 

fusesoc/filters/autotype.py

@@ -0,0 +1,28 @@
+import logging
+import os
+
+logger = logging.getLogger(__name__)
+
+
+class Autotype:
+    def run(self, edam, work_root):
+        type_map = {
+            ".c": "cSource",
+            ".cpp": "cppSource",
+            ".sv": "systemVerilogSource",
+            ".tcl": "tclSource",
+            ".v": "verilogSource",
+            ".vlt": "vlt",
+            ".xdc": "xdc",
+        }
+        for f in edam["files"]:
+            if not "file_type" in f:
+                fn = f["name"]
+                (_, ext) = os.path.splitext(fn)
+                ft = type_map.get(ext, "")
+                if ft:
+                    f["file_type"] = ft
+                    logger.debug(f"Autoassigning file type {ft} to {fn}")
+                else:
+                    logger.warning("Could not autoassign type for " + fn)
+        return edam

fusesoc/filters/custom.py

@@ -0,0 +1,31 @@
+import logging
+import os
+import subprocess
+
+from yaml import dump, load
+
+try:
+    from yaml import CDumper as Dumper
+    from yaml import CLoader as Loader
+except ImportError:
+    from yaml import Dumper, Loader
+
+logger = logging.getLogger(__name__)
+
+
+class Custom:
+    def run(self, edam, work_root):
+        fin = "custom_filter_input.yml"
+        fout = "custom_filter_output.yml"
+        cmd = os.environ.get("FUSESOC_CUSTOM_FILTER")
+        if not cmd:
+            logger.error("Environment variable FUSESOC_CUSTOM_FILTER was not set")
+            return
+        with open(os.path.join(work_root, fin), "w") as f:
+            dump(edam, f, Dumper=Dumper)
+
+        subprocess.run([cmd, fin, fout], cwd=work_root)
+        with open(os.path.join(work_root, fout)) as f:
+            edam = load(f, Loader=Loader)
+
+        return edam

fusesoc/filters/dot.py

@@ -0,0 +1,24 @@
+import os
+
+
+class Dot:
+    def run(self, edam, work_root):
+        nodes = []
+        edges = []
+        deps = edam.get("dependencies", {})
+        for k, v in deps.items():
+            nodes.append(k)
+            for e in v:
+                edges.append((k, e))
+
+        s = "digraph G {\n"
+        s += 'rankdir="LR"\n'
+        for n in nodes:
+            s += f'"{n}";\n'
+        for e in edges:
+            s += f'"{e[0]}" -> "{e[1]}";\n'
+        s += "}\n"
+        print(edam["name"])
+        with open(os.path.join(work_root, edam["name"] + ".gv"), "w") as f:
+            f.write(s)
+        return edam

fusesoc/fusesoc.py

@@ -153,6 +153,7 @@ def get_backend(self, core, flags, backendargs=[]):
         try:
             edalizer.run()
             edalizer.export()
+            edalizer.apply_filters(self.config.filters)
             edalizer.parse_args(backend_class, backendargs)
         except SyntaxError as e:
             raise RuntimeError(e.msg)

fusesoc/main.py

@@ -581,6 +581,12 @@ def get_parser():
         "--work-root",
         help="Output directory for build. VLNV will not be appended (overrides build-root)",
     )
+    parser_run.add_argument(
+        "--filter",
+        help="Add filter. Can be specified multiple times to add multiple filters",
+        action="append",
+        default=[],
+    )
     parser_run.add_argument("--setup", action="store_true", help="Execute setup stage")
     parser_run.add_argument("--build", action="store_true", help="Execute build stage")
     parser_run.add_argument("--run", action="store_true", help="Execute run stage")
@@ -666,13 +672,15 @@ def args_to_config(args, config):
     if hasattr(args, "system_name") and args.system_name and len(args.system_name) > 0:
         setattr(config, "args_system_name", args.system_name)
 
+    if hasattr(args, "filter"):
+        config.args_filters = args.filter
+
 
 def fusesoc(args):
     Fusesoc.init_logging(args.verbose, args.monochrome, args.log_file)
 
     config = Config(args.config)
     args_to_config(args, config)
-
     fs = Fusesoc(config)
 
     # Run the function

pyproject.toml

@@ -45,7 +45,7 @@ Changelog = "https://github.com/olofk/fusesoc/blob/main/NEWS"
 version_file = "fusesoc/version.py"
 
 [tool.setuptools.packages.find]
-include = ["fusesoc", "fusesoc.capi2", "fusesoc.provider", "fusesoc.parser"]
+include = ["fusesoc", "fusesoc.capi2", "fusesoc.provider", "fusesoc.parser", "fusesoc.filters"]
 
 [project.scripts]
 fusesoc = "fusesoc.main:main"

tests/capi2_cores/misc/filters.core

@@ -0,0 +1,15 @@
+CAPI=2:
+# Copyright FuseSoC contributors
+# Licensed under the 2-Clause BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-2-Clause
+
+name: ::filters:0
+targets:
+  nofilters:
+    toplevel : none
+  emptyfilters:
+    filters : []
+    toplevel : none
+  filters:
+    filters : [corefilter1, corefilter2]
+    toplevel : none

tests/test_capi2.py

@@ -274,6 +274,18 @@ def test_capi2_type_check():
     assert "Error validating" in str(excinfo.value)
 
 
+def test_capi2_get_filters():
+    from fusesoc.capi2.coreparser import Core2Parser
+    from fusesoc.core import Core
+
+    core = Core(Core2Parser(), os.path.join(cores_dir, "filters.core"))
+    assert [] == core.get_filters({"is_toplevel": True, "target": "nofilters"})
+    assert [] == core.get_filters({"is_toplevel": True, "target": "emptyfilters"})
+    assert ["corefilter1", "corefilter2"] == core.get_filters(
+        {"is_toplevel": True, "target": "filters"}
+    )
+
+
 def test_capi2_get_flags():
     from fusesoc.capi2.coreparser import Core2Parser
     from fusesoc.core import Core

tests/test_config.py

@@ -43,6 +43,36 @@ def test_config():
     assert conf.library_root == library_root
 
 
+import pytest
+
+
+@pytest.mark.parametrize("from_cli", [False, True])
+@pytest.mark.parametrize("from_config", [False, True])
+def test_config_filters(from_cli, from_config):
+    import tempfile
+
+    from fusesoc.config import Config
+
+    if from_config:
+        tcf = tempfile.NamedTemporaryFile(mode="w+")
+        tcf.write("[main]\nfilters = configfilter1 configfilter2\n")
+        tcf.seek(0)
+        config = Config(tcf.name)
+    else:
+        config = Config()
+
+    if from_cli:
+        config.args_filters = ["clifilter1", "clifilter2"]
+
+    expected = {
+        (False, False): [],
+        (False, True): ["configfilter1", "configfilter2"],
+        (True, False): ["clifilter1", "clifilter2"],
+        (True, True): ["configfilter1", "configfilter2", "clifilter1", "clifilter2"],
+    }
+    assert config.filters == expected[(from_cli, from_config)]
+
+
 def test_config_relative_path():
     with tempfile.TemporaryDirectory() as td:
         config_path = os.path.join(td, "fusesoc.conf")