Metric Comparison + API Updates (#274)

* `openlane.common`
  * Created new function `get_latest_file`, that finds the latest file with a certain filename in a specific folder
  * Created new `Filter` object aggregating the functionality of wildcards
  * Created new module `openlane.common.metrics` to aggregate functions dealing with metrics, replacing previous file
    * `__main__` file of this module has three commands, `compare`, `compare-multiple`, `compare-main`:
      * `compare`: Compares two metrics files and generates an output summary
      * `compare-multiple`: Compares two folders of metrics files and generates an output summary
      * `compare-main`: Compares a folder of metric files with metrics fetched from `efabless/openlane-metrics` and generates an output summary
    * New `Metric` object created, encapsulating the metric name, its aggregator, and some other functionality related to evaluating changes in metrics
      * Also allowed some metrics to exclude some groups from aggregation, e.g. `iter` for routing metrics
    * Created "library" of `Metrics` of the various metrics we aggregated so far
      * `aggregate_metrics` now uses the metric library for aggregators as a default value
    * Updated parsing of metrics to more closely match https://github.com/ieee-ceda-datc/datc-rdf-Metrics4ML: the format for modifiers was wrong
* `openlane.state`
   * Created `__main__` file that returns the latest state for a run directory (using `get_latest_file`, with an options to extract the metrics thereof for a file (useful for scripts) 
* `openlane.config.Config`:
  * Updated `refg::` to return the glob string verbatim if the glob returns no results
* `openlane.config.Macro`:
  * Created new method `from_state` that automatically creates a `Macro` object from a `State` object
* Created new object, `ScopedFile`, which is a file that's deleted as soon as it's deconstructed
* `KLayout.*`:
  * Unified generation of commandline arguments using `.get_cli_args()`
  * Updated scripts to use `%OL_METRIC_I` instead of having to use other communication channels
  * Made sure all paths passed to KLayout commands are absolute because KLayout loves to not use CWD for relative paths
* `Magic.*`:
  * Fixed bug when `MAGICRC` is a relative path
* `Netgen.LVS`, `Checker.LVS`:
  * Updated metric names for grammatical consistency 
* Added `httpx` to replace `requests` as the dependency for HTTP functions

## CI and Testing
* Added function in CI to test every PR against the current `main` function
* Added `pytest-xdist` to dev dependencies, so multiple tests can be run in parallel

---------

Co-authored-by: Kareem Farid <kareefardi@users.noreply.github.com>

.gitattributes

@@ -1 +1,2 @@
-*.v linguist-detectable=false
+openlane/smoke_test_design/* linguist-detectable=false
+.github/* linguist-detectable=false

.github/actions/build_nix/action.yml

@@ -35,7 +35,7 @@ runs:
       shell: ${{ inputs.shell }}
       run: |
         echo "#################################################################"
-        nix-shell ${{ inputs.nix_build_args }} --run "pytest --step-rx '.'"
+        nix-shell ${{ inputs.nix_build_args }} --run "pytest --step-rx '.' -n auto"
     - name: Smoke Test
       shell: ${{ inputs.shell }}
       run: |

.github/scripts/compare_main.js

@@ -0,0 +1,42 @@
+#!/usr/bin/env node
+// Copyright 2023 Efabless Corporation
+//
+// Licensed under the Apache License, Version 2.0(the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+//     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+/**
+ * @param {string} verbosity
+ * @param {string} token
+ * @returns {string}
+ */
+function get(verbosity, table_out = null, token = "") {
+    const { spawnSync } = require("child_process");
+
+    let tableOutOpts = table_out ? ["--table-out", table_out] : [];
+
+    let child = spawnSync("python3", ["-m", "openlane.common.metrics", "compare-main", "current", "--table-verbosity", verbosity, "--token", token].concat(tableOutOpts), { encoding: "utf8" });
+
+    let result = "";
+    if (child.status != 0) {
+        throw new Error("Failed to create report: \n\n```\n" + child.stderr + "\n```");
+    } else {
+        result += "Metric comparisons are in beta. Please report bugs under the issues tab.\n---\n";
+        result += "> To create this report yourself, grab the metrics artifact from the CI run, extract them, and invoke `python3 -m openlane.common.metrics compare-main <path to directory>`.\n\n" + child.stdout;
+    }
+
+    return result.trim();
+};
+
+module.exports = get;
+
+if (require.main === module) {
+    console.log(get("ALL", "table.md", process.argv[2]));
+}

.github/scripts/post_metrics.js

@@ -0,0 +1,129 @@
+#!/usr/bin/env node
+// Copyright 2023 Efabless Corporation
+//
+// Licensed under the Apache License, Version 2.0(the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+//     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+const compareMain = require("./compare_main.js");
+
+/**
+ * 
+ * @param {Octokit} github 
+ * @param {object} context 
+ * @param {string} botUsername 
+ * @param {string} body 
+ */
+async function postOrUpdateComment(github, context, botUsername, body) {
+    const METRIC_REPORT_MARK = "<!-- MARK: METRICS REPORT -->";
+
+    let page = 1;
+    let allComments = [];
+    let comments = [];
+    do {
+        let response = await github.request("GET /repos/{owner}/{repo}/issues/{issue_number}/comments?page={page}&per_page=100", {
+            issue_number: context.issue.number,
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            page,
+        });
+        comments = response.data;
+        allComments = allComments.concat(comments);
+        page += 1;
+    } while (comments.length != 0);
+
+    let found = null;
+    for (let comment of allComments) {
+        if (comment.body.includes(METRIC_REPORT_MARK) && comment.user.login == botUsername) {
+            found = comment;
+            break;
+        }
+    }
+
+    let fn = github.rest.issues.createComment;
+    let request = {
+        issue_number: context.issue.number,
+        owner: context.repo.owner,
+        repo: context.repo.repo,
+        body: `${METRIC_REPORT_MARK}\n\n${body}`
+    };
+
+    if (found) {
+        request.comment_id = found.id;
+        fn = github.rest.issues.updateComment;
+    }
+
+    await fn(request);
+}
+
+/**
+ * 
+ * @param {Octokit} github 
+ * @param {object} context 
+ * @param {string} botUsername 
+ * @param {string} botToken 
+ */
+async function main(github, context, botUsername, botToken) {
+    const fs = require("fs");
+
+    let body;
+    try {
+        body = compareMain("ALL", "./tables_all.md", botToken);
+        let tables = fs.readFileSync("./tables_all.md", { encoding: "utf8" });
+
+        let gistResponse = await github.rest.gists.create({
+            public: true,
+            description: `Results for ${context.repo.owner} / ${context.repo.repo}#${context.issue.number} (Run ${context.runId})`,
+            files: {
+                "10-ALL.md": {
+                    content: tables
+                }
+            }
+        });
+
+        body += `\n\nFull tables ► ${gistResponse.data.html_url}\n`;
+    } catch (e) {
+        body = e.message;
+        console.error(e.message)
+    }
+
+    await postOrUpdateComment(github, context, botUsername, body)
+}
+
+module.exports = main;
+
+if (require.main === module) {
+    // Test
+    try {
+        require("@octokit/rest");
+    } catch (error) {
+        console.error("Run 'yarn add @octokit/rest @octokit/plugin-paginate-rest'")
+        process.exit(-1);
+    }
+    const { Octokit } = require("@octokit/rest");
+
+    const context = {
+        repo: {
+            owner: "efabless",
+            repo: "openlane2"
+        },
+        issue: {
+            number: process.argv[3]
+        },
+        runId: "api_test"
+    };
+
+    let octokit = new Octokit({
+        auth: process.argv[2],
+    });
+
+    main(octokit, context, "openlane-bot", process.argv[2]);
+}

.github/test_sets/get_test_matrix.py

@@ -65,7 +65,7 @@ def main(scls, use_json, test_sets):
                 ol_dir, "test", "designs", design_name, config_filename
             )
             run_folder = os.path.join(
-                ol_dir, "test", "designs", design_name, "runs", "CI"
+                ol_dir, "test", "designs", design_name, "runs", f"{pdk}-{scl}"
             )
             designs.append(
                 {

.github/workflows/ci.yml

@@ -266,7 +266,7 @@ jobs:
           then
             nix-shell --run "\
               python3 -m openlane\
-              --run-tag CI\
+              --run-tag ${{ matrix.design.pdk }}-${{ matrix.design.scl }}\
               --pdk ${{ matrix.design.pdk }}\
               --scl ${{matrix.design.scl}}\
               ${{ matrix.design.config }}\
@@ -282,6 +282,67 @@ jobs:
         with:
           name: ${{ matrix.design.name }}-${{ matrix.design.pdk }}-${{ matrix.design.scl }}
           path: ${{ matrix.design.run_folder }}
+      - name: Fetch Metrics
+        if: ${{ always() }}
+        run: |
+          nix-shell --run "\
+            python3 -m openlane.state latest\
+              ${{ matrix.design.run_folder }}\
+              --extract-metrics-to ${{ matrix.design.pdk }}-${{ matrix.design.scl }}-${{ matrix.design.name }}.metrics.json\
+          "
+      - name: Upload Metrics
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: metrics
+          path: ${{ matrix.design.pdk }}-${{ matrix.design.scl }}-${{ matrix.design.name }}.metrics.json
+  upload_metrics:
+    name: Upload Metrics
+    runs-on: ubuntu-22.04
+    needs: [test]
+    if: ${{ always() }}
+    steps:
+      - name: Check out repo
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+      - name: Setup
+        run: |
+          sudo apt-get install -y python3-tk
+          python3 -m pip install -e .
+          echo "BRANCH_NAME=${GITHUB_REF##*/}" >> $GITHUB_ENV
+      - name: Download Metrics
+        uses: actions/download-artifact@v3
+        with:
+          name: metrics
+          path: current
+      - name: "[PRs] Compare to main and comment result"
+        uses: actions/github-script@v6
+        if: ${{ github.event_name == 'pull_request' }}
+        with:
+          github-token: ${{ secrets.GH_TOKEN }}
+          script: |
+            require("./.github/scripts/post_metrics.js")(
+              github,
+              context,
+              "${{ vars.BOT_USERNAME }}",
+              "${{ secrets.GH_TOKEN }}",
+            ).then(()=>{console.log("Done.");});
+      - name: "[Push] Upload Metrics"
+        if: ${{ github.event_name == 'push' }} # && env.BRANCH_NAME == 'main' }}
+        run: |
+          CURRENT_SHA=$(git rev-parse HEAD)
+          REPO=efabless/openlane-metrics
+          BRANCH_NAME="commit-$CURRENT_SHA"
+          echo "Uploading to 'github.com/$REPO@$BRANCH_NAME'…"
+          cd current
+          git init -b $BRANCH_NAME
+          git add .
+          git config user.name "${{ vars.BOT_NAME }}"
+          git config user.email "${{ vars.BOT_EMAIL }}"
+          git commit -m "Upload"
+          git remote add origin "https://${{ vars.BOT_USERNAME }}:${{ secrets.GH_TOKEN }}@github.com/$REPO.git"
+          git push -fu origin $BRANCH_NAME # Force for if we have to re-run the CI for some reason
   publish:
     runs-on: ubuntu-22.04
     needs: [build-linux-amd64, build-mac-amd64, build-docker, build-py]

.gitignore

@@ -29,11 +29,17 @@ logs/
 failures_report.json
 abc.history
 slpp_all/
-*/runs
+runs/
 
 # Testing
 .coverage
 htmlcov/
 prof/
 sandbox/
 /designs
+
+# Testing CI
+package.json
+node_modules
+yarn.lock
+tables_*.md

Makefile

@@ -37,13 +37,13 @@ lint: venv/manifest.txt
 
 .PHONY: test
 test: venv/manifest.txt
-	./venv/bin/coverage run -m pytest
+	./venv/bin/coverage run -m pytest -n auto
 	./venv/bin/coverage report
 	./venv/bin/coverage html
 
 .PHONY: test-all
 test-all: venv/manifest.txt
-	./venv/bin/coverage run -m pytest --step-rx "."
+	./venv/bin/coverage run -m pytest --step-rx "." -n auto
 	./venv/bin/coverage report
 	./venv/bin/coverage html
 

default.nix

@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 {
+  system,
   lib,
   clangStdenv,
   fetchFromGitHub,
@@ -51,7 +52,7 @@
   psutil,
   pytestCheckHook,
   pyfakefs,
-  system,
+  httpx,
 }:
 buildPythonPackage rec {
   name = "openlane";
@@ -115,6 +116,7 @@ buildPythonPackage rec {
       libparse
       ioplace-parser
       psutil
+      httpx
       klayout-pymod
     ]
     ++ includedTools;

docs/source/reference/configuration.md

@@ -141,8 +141,10 @@ asterisk as a wildcard to pick multiple files in a specific folder.
 
 * Outside the design directory, this is disabled for security reasons and the
   final path will continue to include the asterisk.
-* `refg::` will always return an array, even if only one element was found,
-  for consistency.
+* `refg::` will always return an array, even if only one element was found, for
+  consistency.
+  * If no elements were found, the glob string is returned verbatim as a single
+    element in array.
 
 As shown below, `refg::$DESIGN_DIR/src/*.v` would find all files ending with `.v`
 in the `src` folder inside the design directory.

docs/source/usage/hardening_macros.md

@@ -251,7 +251,7 @@ Routing in general goes through these phases:
 
 The subsections include some notes:
 
-#### Antenna Mitigation
+### Antenna Mitigation
 To help mitigate the antenna effect, after Global Placement there are also three
 other steps you may choose to enable:
 

docs/source/usage/writing_custom_flows.md

@@ -215,7 +215,7 @@ its `StepError`s:
 ---
 language: python
 start-after: "step_list.append(step)"
-end-before: "self.end_stage()"
+end-before: "raise FlowError(str(e))"
 ---
 ```
 

nix/create-shell.nix

@@ -23,6 +23,7 @@ with pkgs; let
           openlane
           pyfakefs
           pytest
+          pytest-xdist
           pillow
           mdformat
         ]