CIRCT Integration

[abejgonzalez]

• Adds CIRCT firtool integration (changes barstools GenerateTopAndHarness executable to preprocess Firrtl if need be). Dramatically speeds up FIRRTL compile times!!!
• Emits design verilog as one module per verilog file

TODO:

• Add/build firtool conda package
• Create .f manifest files for top/harness level modules (do people care about the black-box .f files)
• Cleanly add SiFive module annotation to .anno.json
• Add a clean way to bypass the CIRCT compiler completely if wanted (which changes SFC to dump the same output as the SFC + CIRCT flow) - This will be probably more effort than its worth so unless this is absolutely necessary I'm ignoring this
• Change HARNESS_* to MODEL_* for consistency
• Change some FIRTOOL_* stuff to SFC_FIRTOOL_* (or a better name?)
• Check output SyncReadMem verilog output (FIRTOOL output acts like SRAMS with async writes, requiring the user to inject a cycle manually before writes to replicate the SFC generated SRAM behavior)

Related PRs / Issues:

Type of change:

• Bug fix
• New feature
• Other enhancement

Impact:

• RTL change
• Software change (RISC-V software)
• Build system change
• Other

Contributor Checklist:

• Did you set main as the base branch?
• Is this PR's title suitable for inclusion in the changelog and have you added a changelog:<topic> label?
• Did you state the type-of-change/impact?
• Did you delete any extraneous prints/debugging code?
• Did you mark the PR with a changelog: label?
• (If applicable) Did you update the conda .conda-lock.yml file if you updated the conda requirements file?
• (If applicable) Did you add documentation for the feature?
• (If applicable) Did you add a test demonstrating the PR?

• (If applicable) Did you mark the PR as Please Backport?

[abejgonzalez]

Note this only works up until make verilog

[jerryz123]

!!!

[abejgonzalez]

Notes (so I don't forget):

The script to create .f files expects that the module name == file name. This breaks when handling the black-box files (which don't have this necessarily). Right now, I just have a separate BB .f file that needs to be added in addition to the other *.f files. For more clarity this is what happens:

• *.top.f - Includes all Chisel module files under top (and maybe some BB files - if the mod == filename)
• *.tb.f - Same as top version but contains modules under TB only (not under the DUT)
• *.all.f - Just a concat of the top and TB .fs (so may not include all BBs)
• firrtl_bb_rsrc_files.final.f - List of BB files

[abejgonzalez]

@harrisonliew could you modify the VLSI makefile and try this out with a Hammer flow? Also, could you link me a custom xform that you might want to run in the future?

[jerryz123]

What's breaking the spi tests?

[abejgonzalez]

Maybe @seldridge you have some advice on how to fix this.

In our FIRRTL that is passed to CIRCT, we see this:

attach (spi_mem_0.dq[0], chiptop.spi_0.dq[0])

This generates the following:

`ifdef verilator
  `error "Verilator does not support alias and thus cannot arbitrarily connect bidirectional wires and ports"
`else
  alias _dq_0_wire = _spi_0_dq_0_wire;
`endif

Basically, according to llvm/circt#738, CIRCT doesn't support alias types (see also llvm/circt#788).

I don't know of a way to bypass this. Both using <> and attach for Analog wires create the same FIRRTL output (and thus same Verilog). I don't see any visible plans to have a Verilator alternative here.

[seldridge]

What is the SFC doing here and what is more of the input FIRRTL? Is this a usage of an analog that is only attached port-to-instance or is it being routed up/down?

The answer in the past has been to make CIRCT work harder to sink inout into instances. However, we are usually very careful to never route analog other than port-to-instance.

[abejgonzalez]

Here is the FIRRTL:

circuit TestHarness :
  module TestHarness :
    input clock : Clock
    input reset : UInt<1>
    output io : { success : UInt<1>}

    wire buildtopClock : Clock @[TestHarness.scala 86:27]
    wire buildtopReset : Reset @[TestHarness.scala 87:27]
    inst chiptop of ChipTop @[TestHarness.scala 90:19]
    io.success <= UInt<1>("h0") @[TestHarness.scala 92:14]
    node dutReset = asAsyncReset(buildtopReset) @[TestHarness.scala 94:32]
    wire dtm_success : UInt<1>
    dtm_success <= UInt<1>("h0")
    when dtm_success : @[HarnessBinders.scala 251:28]
      io.success <= UInt<1>("h1") @[HarnessBinders.scala 251:41]
    wire jtag_wire : { TCK : Clock, TMS : UInt<1>, TDI : UInt<1>, flip TDO : { data : UInt<1>, driven : UInt<1>}} @[HarnessBinders.scala 252:29]
    jtag_wire.TDO.data <= chiptop.jtag.TDO @[HarnessBinders.scala 253:28]
    jtag_wire.TDO.driven <= UInt<1>("h1") @[HarnessBinders.scala 254:30]
    chiptop.jtag.TCK <= jtag_wire.TCK @[HarnessBinders.scala 255:15]
    chiptop.jtag.TMS <= jtag_wire.TMS @[HarnessBinders.scala 256:15]
    chiptop.jtag.TDI <= jtag_wire.TDI @[HarnessBinders.scala 257:15]
    inst SimJTAG of SimJTAG @[HarnessBinders.scala 258:26]
    SimJTAG.exit is invalid
    SimJTAG.init_done is invalid
    SimJTAG.enable is invalid
    SimJTAG.jtag is invalid
    SimJTAG.reset is invalid
    SimJTAG.clock is invalid
    node _T = asUInt(buildtopReset) @[HarnessBinders.scala 258:107]
    node _T_1 = asUInt(buildtopReset) @[HarnessBinders.scala 258:134]
    node _T_2 = not(_T_1) @[HarnessBinders.scala 258:115]
    jtag_wire.TCK <= SimJTAG.jtag.TCK @[Periphery.scala 217:15]
    jtag_wire.TMS <= SimJTAG.jtag.TMS @[Periphery.scala 218:15]
    jtag_wire.TDI <= SimJTAG.jtag.TDI @[Periphery.scala 219:15]
    SimJTAG.jtag.TDO.driven <= jtag_wire.TDO.driven @[Periphery.scala 220:17]
    SimJTAG.jtag.TDO.data <= jtag_wire.TDO.data @[Periphery.scala 220:17]
    SimJTAG.clock <= buildtopClock @[Periphery.scala 222:14]
    SimJTAG.reset <= _T @[Periphery.scala 223:14]
    inst plusarg_reader of plusarg_reader_134 @[PlusArg.scala 80:11]
    plusarg_reader.out is invalid
    SimJTAG.enable <= plusarg_reader.out @[Periphery.scala 225:18]
    SimJTAG.init_done <= _T_2 @[Periphery.scala 226:18]
    node _dtm_success_T = eq(SimJTAG.exit, UInt<1>("h1")) @[Periphery.scala 230:26]
    dtm_success <= _dtm_success_T @[Periphery.scala 230:15]
    node _T_3 = geq(SimJTAG.exit, UInt<2>("h2")) @[Periphery.scala 231:19]
    when _T_3 : @[Periphery.scala 231:27]
      node _T_4 = dshr(SimJTAG.exit, UInt<1>("h1")) @[Periphery.scala 232:59]
      node _T_5 = bits(reset, 0, 0) @[Periphery.scala 232:13]
      node _T_6 = eq(_T_5, UInt<1>("h0")) @[Periphery.scala 232:13]
      when _T_6 : @[Periphery.scala 232:13]
        printf(clock, UInt<1>("h1"), "*** FAILED *** (exit code = %d)\n", _T_4) : printf @[Periphery.scala 232:13]
      node _T_7 = bits(reset, 0, 0) @[Periphery.scala 233:11]
      node _T_8 = eq(_T_7, UInt<1>("h0")) @[Periphery.scala 233:11]
      when _T_8 : @[Periphery.scala 233:11]
        stop(clock, UInt<1>("h1"), 1) : stop @[Periphery.scala 233:11]
    node _bits_T = asClock(UInt<1>("h0")) @[SerialAdapter.scala 26:32]
    inst bits_out_queue of AsyncQueue @[SerialAdapter.scala 27:29]
    bits_out_queue.clock <= _bits_T
    bits_out_queue.reset <= UInt<1>("h0")
    bits_out_queue.io.enq <= chiptop.serial_tl.bits.out @[SerialAdapter.scala 28:24]
    bits_out_queue.io.enq_clock <= chiptop.serial_tl.clock @[SerialAdapter.scala 29:30]
    node _bits_out_queue_io_enq_reset_T = asUInt(buildtopReset) @[SerialAdapter.scala 30:39]
    bits_out_queue.io.enq_reset <= _bits_out_queue_io_enq_reset_T @[SerialAdapter.scala 30:30]
    bits_out_queue.io.deq_clock <= buildtopClock @[SerialAdapter.scala 31:30]
    node _bits_out_queue_io_deq_reset_T = asUInt(buildtopReset) @[SerialAdapter.scala 32:39]
    bits_out_queue.io.deq_reset <= _bits_out_queue_io_deq_reset_T @[SerialAdapter.scala 32:30]
    inst bits_in_queue of AsyncQueue_1 @[SerialAdapter.scala 33:28]
    bits_in_queue.clock <= _bits_T
    bits_in_queue.reset <= UInt<1>("h0")
    chiptop.serial_tl.bits.in <= bits_in_queue.io.deq @[SerialAdapter.scala 34:20]
    bits_in_queue.io.deq_clock <= chiptop.serial_tl.clock @[SerialAdapter.scala 35:29]
    node _bits_in_queue_io_deq_reset_T = asUInt(buildtopReset) @[SerialAdapter.scala 36:38]
    bits_in_queue.io.deq_reset <= _bits_in_queue_io_deq_reset_T @[SerialAdapter.scala 36:29]
    bits_in_queue.io.enq_clock <= buildtopClock @[SerialAdapter.scala 37:29]
    node _bits_in_queue_io_enq_reset_T = asUInt(buildtopReset) @[SerialAdapter.scala 38:38]
    bits_in_queue.io.enq_reset <= _bits_in_queue_io_enq_reset_T @[SerialAdapter.scala 38:29]
    wire bits : { flip in : { flip ready : UInt<1>, valid : UInt<1>, bits : UInt<32>}, out : { flip ready : UInt<1>, valid : UInt<1>, bits : UInt<32>}} @[SerialAdapter.scala 39:25]
    bits_in_queue.io.enq <= bits.in @[SerialAdapter.scala 40:23]
    bits.out.bits <= bits_out_queue.io.deq.bits @[SerialAdapter.scala 41:19]
    bits.out.valid <= bits_out_queue.io.deq.valid @[SerialAdapter.scala 41:19]
    bits_out_queue.io.deq.ready <= bits.out.ready @[SerialAdapter.scala 41:19]
    inst ram of SerialRAM @[SerialAdapter.scala 64:24]
    ram.clock <= buildtopClock
    ram.reset <= buildtopReset
    ram.io.ser <= bits @[SerialAdapter.scala 65:19]
    node _success_T = asUInt(buildtopReset) @[HarnessBinders.scala 307:112]
    inst success_sim of SimSerial @[SerialAdapter.scala 101:23]
    success_sim.exit is invalid
    success_sim.serial is invalid
    success_sim.reset is invalid
    success_sim.clock is invalid
    success_sim.clock <= buildtopClock @[SerialAdapter.scala 102:20]
    success_sim.reset <= _success_T @[SerialAdapter.scala 103:20]
    success_sim.serial <= ram.io.tsi_ser @[SerialAdapter.scala 104:21]
    when success_sim.exit : @[HarnessBinders.scala 308:24]
      io.success <= UInt<1>("h1") @[HarnessBinders.scala 308:37]
    inst plusarg_reader_1 of plusarg_reader_151 @[PlusArg.scala 80:11]
    plusarg_reader_1.out is invalid
    chiptop.custom_boot <= plusarg_reader_1.out @[HarnessBinders.scala 329:21]
    chiptop.clock.clock <= buildtopClock @[HarnessBinders.scala 340:17]
    node _chiptop_reset_T = asAsyncReset(buildtopReset) @[HarnessBinders.scala 342:51]
    chiptop.reset <= _chiptop_reset_T @[HarnessBinders.scala 342:31]
    inst spi_mem_0 of SimSPIFlashModel @[SPIFlash.scala 79:27]
    spi_mem_0.reset is invalid
    spi_mem_0.dq is invalid
    spi_mem_0.cs is invalid
    spi_mem_0.sck is invalid
    spi_mem_0.sck <= chiptop.spi_0.sck @[SPIFlash.scala 81:22]
    spi_mem_0.cs[0] <= chiptop.spi_0.cs[0] @[SPIFlash.scala 83:24]
    attach (spi_mem_0.dq[0], chiptop.spi_0.dq[0]) @[SPIFlash.scala 84:61]
    attach (spi_mem_0.dq[1], chiptop.spi_0.dq[1]) @[SPIFlash.scala 84:61]
    attach (spi_mem_0.dq[2], chiptop.spi_0.dq[2]) @[SPIFlash.scala 84:61]
    attach (spi_mem_0.dq[3], chiptop.spi_0.dq[3]) @[SPIFlash.scala 84:61]
    node _spi_mem_0_io_reset_T = asUInt(buildtopReset) @[SPIFlash.scala 85:33]
    spi_mem_0.reset <= _spi_mem_0_io_reset_T @[SPIFlash.scala 85:24]
    inst simdram of SimDRAM @[HarnessBinders.scala 190:23]
    simdram.axi is invalid
    simdram.reset is invalid
    simdram.clock is invalid
    simdram.axi <= chiptop.axi4_mem_0.bits @[HarnessBinders.scala 191:18]
    simdram.clock <= chiptop.axi4_mem_0.clock @[HarnessBinders.scala 210:20]
    simdram.reset <= chiptop.axi4_mem_0.reset @[HarnessBinders.scala 211:20]
    inst uart_sim_0 of UARTAdapter @[UARTAdapter.scala 132:28]
    uart_sim_0.clock <= clock
    uart_sim_0.reset <= reset
    uart_sim_0.io.uart.txd <= chiptop.uart_0.txd @[UARTAdapter.scala 134:28]
    chiptop.uart_0.rxd <= uart_sim_0.io.uart.rxd @[UARTAdapter.scala 135:18]
    wire refClkBundle : { clock : Clock, reset : Reset} @[TestHarness.scala 39:27]
    buildtopClock <= refClkBundle.clock @[TestHarness.scala 103:17]
    wire _buildtopReset_WIRE : Reset
    _buildtopReset_WIRE <= refClkBundle.reset
    buildtopReset <= _buildtopReset_WIRE @[TestHarness.scala 104:17]
    wire implicitHarnessClockBundle : { clock : Clock, reset : Reset} @[TestHarness.scala 106:40]
    implicitHarnessClockBundle.clock <= clock @[TestHarness.scala 107:36]
    implicitHarnessClockBundle.reset <= reset @[TestHarness.scala 108:36]
    refClkBundle.clock <= implicitHarnessClockBundle.clock @[TestHarness.scala 73:47]
    refClkBundle.reset <= implicitHarnessClockBundle.reset @[TestHarness.scala 74:47]

And here is the corresponding Chisel: https://github.com/ucb-bar/testchipip/blob/f99b1eb59a34d7934059904f03237d5bc8d4a680/src/main/scala/SPIFlash.scala#L84

Basically Analog ports from two modules (under the TestHarness) are being connected (one module is our DUT and the other is a SPI flash module). Are you saying this instance-to-instance connection isn't supported (only port-to-instance)?

[seldridge]

Nah, this should work. Anytime we can avoid emitting an alias/bi-directional assign is good as tool support is extremely spotty. We just don't happen to do this and only ever pass analog up or down (as far as I know).

Tracking issue: llvm/circt#4112

[seldridge]

This issue is now fixed (thanks to a contribution by @tymcauley!).

[abejgonzalez]

I just saw it. Thanks shoutout @tymcauley!

[seldridge]

Super cool. Thanks for getting this together!

[seldridge]

FYI: we are currently running with: --lowering-options=emittedLineLength=2048,verifLabels,disallowLocalVariables,explicitBitcast,locationInfoStyle=wrapInAtSquareBracket. If you do any running with Verilator, the last part will emit source locators as // @[foo.bar line:col] instead of // foo.bar line:col. This works around a rare bug where Verilator will interpret any comment that starts with "verilator" as a directive it is supposed to process, regardless of case or spacing. This only comes up if you have any files in your code base that start with "verilator".

[tymcauley]

So, the sed -i strategy gave me a few hiccups on macOS. First, I ran this and got some errors since macOS sed doesn't have the same -i flag as GNU sed. Homebrew provides the coreutils package, which gives you gsed, so I just replaced all the sed calls in here to use that.

Then I re-ran the build, and got an error related to the SFC not being able to parse the memory config statements. That was because this sed -i command failed, but this rule was never re-run, since FIRTOOL_SMEMS_CONF was already generated. So the sed -i strategy doesn't feel like it meshes so cleanly with a make-based flow. Or maybe this is such a narrow error case that we should be fine with it?

FWIW when I cleared the output directory and re-ran after replacing sed with gsed, it all worked (at least when just building the default RocketConfig).

[abejgonzalez]

I tried doing sed -i calls to avoid having a bunch of extra temporary files that are really unused. I generally am comfortable with adding sed -i's but if there is a better recommendation I'd be happy to use it.

In the interim, I think I'll add a flag to override sed (so you don't have to hack it in).

Side note: Do you use the Conda setup on Mac? I haven't tested it there and assumed it didn't work on Mac.

[tymcauley]

Sounds great to me! I'm actually not using the Conda setup on Mac. I basically just use my Mac for Verilog builds, not for simulations. I just built the firtool binary myself from source, and have it on my PATH for testing out this branch.

[jerryz123]

Great stuff. Everything is in place now. Lets get this in.