Merge nim-lang#187 nim-lang#188

187: debug: clean up formatting for the vmgen debugging r=saem a=haxscramper

- Improve formatting for the vmgen code listing reports
- Split VM debuggint defines in two - `nimVMDebugExecute` to trace actually
  executed vm opcodes, and `nimVMDebugGenerate` to see generated code.
- Allow filtering out generated code by proc name, using
  `--define:expandVmListing=vmTarget`
```nim
type
  Obj = object
    charField: char
    intField: int

proc vmTarget(arg: Obj) =
  echo arg.charField.int + arg.intField

static:
  vmTarget(Obj())

```

```
nim c --filenames:canonical --define:expandVmListing=vmTarget file.nim
```

```
Code listing for vmTarget
  LdConst      r3     $     1279                system.nim(2005, 30)
  LdObj        r6     r1     r0                 file.nim(7, 11)
  NodeToReg    r5     r6     r0                 file.nim(7, 11)
  Conv         r6     r5     int   char         file.nim(7, 21)
  LdObj        r7     r1     r1                 file.nim(7, 31)
  NodeToReg    r5     r7     r0                 file.nim(7, 31)
  AddInt       r4     r6     r5                 file.nim(7, 26)
  IndCallAsgn  r2     r3     nim-lang#2                 file.nim(7, 26)
  Echo         r2     r1     r0                 file.nim(7, 26)
  Ret          r0     r0     r0                 file.nim(7, 8)
  Eof          r0     r0     r0                 file.nim(7, 8)
```

188: separate documentation publishing from building r=saem a=alaviss

Since docs were published as a part of the main CI run, we were forced
to re-run CI on devel even though bors has performed the check and built
the documentation. This wastes resources especially since our test
matrix is expanding.

This commit implemented a separate publisher that publishes artifacts
built during staging (currently docs). The publisher uses a pull
architecture, that is, to query producers and obtain necessary artifacts
from them.

Main CI will now always build and upload generated documentation, which
has a nice bonus of allowing us to review docgen changes more throughly
in PRs.

An another adjustment made is that the "Edit" link will now be
hard-coded to the `devel` branch instead of being inferred from the
branch being built on. This lets us reuse docs built during the time
spent in staging branch.

Closes nim-lang#182 

Co-authored-by: haxscramper <haxscramper@gmail.com>
Co-authored-by: Leorize <leorize+oss@disroot.org>

.github/workflows/ci.yml

@@ -3,6 +3,9 @@ on:
   push:
     # Empty configuration means use default (ie. test all branches)
     branches-ignore:
+      # Everything would have passed bors before going into devel
+      - devel
+      # Bors temporary branches
       - staging.tmp
       - trying.tmp
       - staging-squash-merge.tmp
@@ -265,24 +268,22 @@ jobs:
 
       - name: Build docs
         run: |
-          branch=${{ github.ref }}
-          # Remove refs/heads/ prefix
-          branch=${branch##*/}
-
           ./koch.py doc \
             --git.url:'https://github.com/${{ github.repository }}' \
             --git.commit:'${{ github.sha }}' \
-            --git.devel:"$branch"
+            --git.devel:devel
+
+          # Remove leftover nimcache
+          rm -rf doc/html/nimcache
 
-      - name: Publish
-        if: |
-          github.event_name == 'push' && github.ref == 'refs/heads/devel' &&
-          matrix.target.publish_docs
-        uses: crazy-max/ghaction-github-pages@v2.6.0
+      - name: Publish to artifacts
+        if: matrix.target.publish_docs
+        uses: actions/upload-artifact@v2.3.1
         with:
-          build_dir: doc/html
-        env:
-          GITHUB_TOKEN: ${{ github.token }}
+          # If this name is updated, tweak publisher.yml
+          name: Generated docs
+          path: doc/html/
+          if-no-files-found: error
 
   passed:
     name: All check passed

.github/workflows/publisher.yml

@@ -0,0 +1,39 @@
+name: Publish built artifacts
+on:
+  push:
+    branches:
+      - devel
+
+# Run every script actions in bash
+defaults:
+  run:
+    shell: bash
+
+# Since we will be pushing, make sure that only one instance can run at a time.
+concurrency: publisher
+
+jobs:
+  publisher:
+    runs-on: ubuntu-latest
+
+    steps:
+      # Publish action needs a checkout
+      - uses: actions/checkout@v2.4.0
+
+      # Download the latest instance of generated documentation from the build
+      # during bors staging.
+      - name: Download generated docs
+        uses: dawidd6/action-download-artifact@v2.16.0
+        with:
+          workflow: ci.yml
+          workflow_conclusion: completed
+          commit: ${{ github.event.after }}
+          # Keep up-to-date with ci.yml
+          name: Generated docs
+          path: doc/html
+
+      - name: Publish docs
+        uses: JamesIves/github-pages-deploy-action@v4.2.2
+        with:
+          branch: gh-pages
+          folder: doc/html

compiler/ast/lineinfos.nim

@@ -58,9 +58,13 @@ proc computeNotesVerbosity(): tuple[
     # debug report for transition of the configuration options
     result.base.incl {rdbgOptionsPush, rdbgOptionsPop}
 
-  when defined(nimVMDebug):
+  when defined(nimVMDebugExecute):
+    result.base.incl {
+      rdbgVmExecTraceFull # execution of the generated code listings
+    }
+
+  when defined(nimVMDebugGenerate):
     result.base.incl {
-      rdbgVmExecTraceFull, # execution of the generated code listings
       rdbgVmCodeListing    # immediately generated code listings
     }
 

compiler/front/cli_reporter.nim

@@ -19,7 +19,8 @@ import
     strformat,
     tables,
     intsets,
-    json
+    json,
+    strtabs
   ],
   ast/[
     lineinfos,
@@ -101,7 +102,7 @@ proc formatPath(conf: ConfigRef, path: string): string =
     when compileOption"excessiveStackTrace":
       # instLoc(), when `--excessiveStackTrace` is used, generates full
       # paths that /might/ need to be filtered if `--filenames:canonical`.
-      const compilerRoot = currentSourcePath().parentDir()
+      const compilerRoot = currentSourcePath().parentDir().parentDir()
       if conf.filenameOption == foCanonical and
          path.startsWith(compilerRoot):
         result = path[(compilerRoot.len + 1) .. ^1]
@@ -3170,43 +3171,60 @@ proc reportBody*(conf: ConfigRef, r: DebugReport): string =
       result = "cfg trace '" & r.str & "'"
 
     of rdbgVmCodeListing:
+      result.add "Code listing"
+      let l = r.vmgenListing
+      if not l.sym.isNil():
+        result.addf(
+          " for the '$#' $#\n\n",
+          l.sym.name.s,
+          conf.toStr(l.sym.info))
+
+      else:
+        result.add "\n\n"
+
       for e in r.vmgenListing.entries:
         if e.isTarget:
-          result.add("L:\n", e.pc)
+          result.add("L:", e.pc, "\n")
 
+        func `$<`[T](arg: T): string = alignLeft($arg, 5)
+        func `$<`(opc: TOpcode): string = alignLeft(opc.toStr, 12)
+
+        var line: string
         case e.opc:
           of {opcIndCall, opcIndCallAsgn}:
-            result.addf("\t$#\tr$#, r$#, nargs:$#", e.opc, e.ra, e.rb, e.rc)
+            line.addf("  $# r$# r$# #$#", $<e.opc, $<e.ra, $<e.rb, $<e.rc)
 
           of {opcConv, opcCast}:
-            result.addf(
-              "\t$#\tr$#, r$#, $#, $#",
-              $e.opc.toStr,
-              $e.ra,
-              $e.rb,
-              $e.types[0].typeToString(),
-              $e.types[1].typeToString())
+            line.addf(
+              "  $# r$# r$# $# $#",
+              $<e.opc,
+              $<e.ra,
+              $<e.rb,
+              $<e.types[0].typeToString(),
+              $<e.types[1].typeToString())
 
           elif e.opc < firstABxInstr:
-            result.addf("\t$#\tr$#, r$#, r$#", e.opc.toStr, $e.ra, $e.rb, $e.rc)
+            line.addf("  $# r$# r$# r$#", $<e.opc, $<e.ra, $<e.rb, $<e.rc)
 
           elif e.opc in relativeJumps + {opcTry}:
-            result.addf("\t$#\tr$#, L$#", e.opc.toStr, $e.ra, $e.idx)
+            line.addf("  $# r$# L$#", $<e.opc, $<e.ra, $<e.idx)
 
           elif e.opc in {opcExcept}:
-            result.addf("\t$#\t$#, $#", $e.opc.toStr, $e.ra, $e.idx)
+            line.addf("  $# $# $#", $<e.opc, $<e.ra, $<e.idx)
 
           elif e.opc in {opcLdConst, opcAsgnConst}:
-            result.addf(
-              "\t$#\tr$#, $# ($#)",
-              $e.opc.toStr, $e.ra, $e.ast.renderTree(), $e.idx)
+            line.addf(
+              "  $# r$# $# $#",
+              $<e.opc, $<e.ra, $<e.ast.renderTree(), $<e.idx)
 
           else:
-            result.addf("\t$#\tr$#, $#", e.opc.toStr, $e.ra, $e.idx)
+            line.addf("  $# r$# $#", $<e.opc, $<e.ra, $<e.idx)
 
-        result.add("\t# ")
-        result.add(toStr(conf, e.info))
-        result.add("\n")
+        result.add(
+          line,
+          tern(line.len <= 48, repeat(" ", 48 - line.len), ""),
+          toStr(conf, e.info),
+          "\n")
 
 proc reportFull*(conf: ConfigRef, r: DebugReport): string =
   assertKind r
@@ -3457,7 +3475,24 @@ proc reportHook*(conf: ConfigRef, r: Report): TErrorHandling =
   # comment
   assertKind r
 
-  if (
+  if (r.kind == rdbgVmCodeListing) and (
+    (
+     # If special expand target is not defined, debug all generated code
+     ("expandVmListing" notin conf.symbols)
+    ) or (
+     # Otherwise check if listing target is not nil, and it's name is the
+     # name we are targeting.
+     (not r.debugReport.vmgenListing.sym.isNil()) and
+     (
+       r.debugReport.vmgenListing.sym.name.s ==
+       conf.symbols["expandVmListing"]
+  ))):
+    echo conf.reportFull(r)
+
+  elif r.kind == rdbgVmCodeListing:
+    return
+
+  elif (
      (conf.isEnabled(r) and r.category == repDebug and tryhack) or
      # Force write of the report messages using regular stdout if tryhack is
      # enabled
@@ -3490,6 +3525,8 @@ proc reportHook*(conf: ConfigRef, r: Report): TErrorHandling =
     # Return without writing
     return
 
+
+
   elif r.kind == rsemProcessing and conf.hintProcessingDots:
     # REFACTOR 'processing with dots' - requires special hacks, pretty
     # useless, need to be removed in the future.

compiler/vm/vm.nim

@@ -63,7 +63,7 @@ import ast/ast except getstr
 
 
 const
-  traceCode = defined(nimVMDebug)
+  traceCode = defined(nimVMDebugExecute)
 
 when hasFFI:
   import evalffi

compiler/vm/vmgen.nim

@@ -64,7 +64,7 @@ when defined(nimCompilerStacktraceHints):
   import std/stackframes
 
 const
-  debugEchoCode* = defined(nimVMDebug)
+  debugEchoCode* = defined(nimVMDebugGenerate)
 
 when hasFFI:
   import vm/evalffi

doc/intern.rst

@@ -212,6 +212,131 @@ You can also bisect using custom options to build the compiler, for example if
 you don't need a debug version of the compiler (which runs slower), you can replace
 `./koch.py temp`:cmd: by explicit compilation command, see `Rebuilding the compiler`_.
 
+Debugging the compiler
+======================
+
+How to debug different subsystems of the compiler using built-in tooling.
+
+Special defines
+---------------
+
+Debugging functionality can usually be accessed only when compiler itself
+has been built with special defines - this is made to avoid runtime
+overhead, because some debugging tools are not exactly cheap to run.
+
+==================== =======
+Define               Enables
+-------------------- -------
+nimVMDebugExecute    Print out every instruction executed by the VM
+nimVMDebugGenerate   List VM code generated for every procedure called at compile-time
+nimDebugUtils        Enable semantic analysis execution tracer
+
+==================== =======
+
+
+Semantic analysis
+-----------------
+
+When `nimDebugUtils` is enabled you can annotate section of the code to
+trace how semantic analysis is performed on this block of code. Each
+procedure that is annotated with one of the `addInNimDebugUtils()`
+overloads is going to be traced in the execution tree. For example -
+`sem.semOverloadedCall`, triggered when it is necessary to analyze call to
+overloaded procedure.
+
+.. code-block:: nim
+
+    proc semOverloadedCall(c: PContext, n, nOrig: PNode,
+                           filter: TSymKinds, flags: TExprFlags): PNode {.nosinks.} =
+      addInNimDebugUtils(c.config, "semOverloadedCall")
+      # Other implementaion parts ...
+
+If you compile your test file with `nim c -d:nimDebugUtils
+--filenames:canonical file.nim` and annotate code block with the
+
+.. code-block:: nim
+
+    {.define(nimCompilerDebug).}
+    vmTarget(Obj())
+    {.undef(nimCompilerDebug).}
+
+You will get all the call entries traced
+
+.. code-block:: literal
+
+    >>] trace start
+    #0]> semExpr @ sem/semexprs.nim(2948, 21) from sem/semstmts.nim(2446, 1)
+    #1]  > semOverloadedCallAnalyseEffects @ sem/semexprs.nim(941, 21) from sem/semexprs.nim(1133, 1)
+    #2]    > semOverloadedCall @ sem/semcall.nim(569, 21) from sem/semexprs.nim(950, 1)
+    #3]      > semExpr @ sem/semexprs.nim(2948, 21) from sem/semexprs.nim(43, 1)
+    #4]        > semTypeNode @ sem/semtypes.nim(1903, 21) from sem/semobjconstr.nim(469, 1)
+    #4]        < semTypeNode @ sem/semtypes.nim(1903, 21) from sem/semobjconstr.nim(469, 1)
+    #3]      < semExpr @ sem/semexprs.nim(2948, 21) from sem/semexprs.nim(43, 1)
+    #2]    < semOverloadedCall @ sem/semcall.nim(569, 21) from sem/semexprs.nim(950, 1)
+    #1]  < semOverloadedCallAnalyseEffects @ sem/semexprs.nim(941, 21) from sem/semexprs.nim(1133, 1)
+    #0]< semExpr @ sem/semexprs.nim(2948, 21) from sem/semstmts.nim(2446, 1)
+    #0]> semExpr @ sem/semexprs.nim(2948, 21) from sem/semstmts.nim(2446, 1)
+    <<] trace end
+
+Formatting of the report is implemented in the `cli_reporter.nim` as well
+(all debug reports are also transferred using regular reporting pipeline).
+It has a lot of information, but general parts for each call parts are:
+
+.. code-block:: literal
+
+   #2]    < semOverloadedCall @ sem/semcall.nim(569, 21) from sem/semexprs.nim(950, 1)
+   ^      ^ ^                   ^                             ^
+   |      | |                   |                             Where proc has been called from
+   |      | |                   Location of the `addInNimDebugUtils()` - the proc itsemf
+   |      | Name of the proc
+   |      Whether proc has been entered or exited
+   Depth of the traced call tree
+
+
+
+VM codegen and execution
+------------------------
+
+VM code generation prints all of the generated procedures. If this is not
+needed (which would be the majority of use cases) you can add
+`--define:expandVmListing=vmTarget` and only code for the specific proc
+would be printed. For example (generated listing might not match exactly)
+
+.. code-block:: nim
+
+    type
+      Obj = object
+        charField: char
+        intField: int
+
+    proc vmTarget(arg: Obj) =
+      echo arg.charField.int + arg.intField
+
+    static:
+      vmTarget(Obj())
+
+
+.. code-block:: cmd
+
+  nim c --filenames:canonical --define:expandVmListing=vmTarget file.nim
+
+
+.. code-block:: literal
+
+    Code listing for the 'vmTarget' file.nim(6, 6)
+
+      LdConst      r3     $     1279                system.nim(2005, 30)
+      LdObj        r6     r1     r0                 file.nim(7, 11)
+      NodeToReg    r5     r6     r0                 file.nim(7, 11)
+      Conv         r6     r5     int   char         file.nim(7, 21)
+      LdObj        r7     r1     r1                 file.nim(7, 31)
+      NodeToReg    r5     r7     r0                 file.nim(7, 31)
+      AddInt       r4     r6     r5                 file.nim(7, 26)
+      IndCallAsgn  r2     r3     #2                 file.nim(7, 26)
+      Echo         r2     r1     r0                 file.nim(7, 26)
+      Ret          r0     r0     r0                 file.nim(7, 8)
+      Eof          r0     r0     r0                 file.nim(7, 8)
+
 
 Runtimes
 ========