AXI Memory Over TL Serial Link

[abejgonzalez]

Related issue: ucb-bar/testchipip#121

Type of change: new feature

Impact: rtl change

Release Notes
This PR adds support for simulating an AXI memory interface over the default TL serial link in Chipyard/FireSim.

• Add WithSerialTLBackingMemory config frag. to move ExtMem params to SerialTLKey
• New harness binders to connect the Serial Link + Passthrough Clock (for AXI clock domain)
• New example configs to demonstrate setup (MulticlockAXIOverSerialConfig, FireSimMulticlockAXIOverSerialConfig)
• Modified config. fragment to specify tile clocks (overall or just by hart id)
• Split FireSim "tweaks" into 2 fragments... without clocking information and overall. This lets users ignore FireSim clock defaults if they want.
• Add support in both Chipyard and FireSim to request a clock in the harness (and harness binder). Chipyard provides both a clock/reset while FireSim just provides a clock. Note that (AXI Memory Over TL Serial Link #812 (comment)) applies to both FireSim and Chipyard's test harnesses.

This was tested in FireSim @ 30MHz with the added config.

[jerryz123]

Like this change.
Maybe call this WithFireSimClockingTweaks, and keep the name of the old one?

[jerryz123]

Actually, its probably safer to rename the old one as well, WithFireSimDesignTweak, perhaps?

[abejgonzalez]

This WithFireSimConfigTweaks combines the clocking with the "design" tweaks. In order to avoid changing FireSim itself (changing the build recipes), I will at least keep this name WithFireSimConfigTweaks.

[davidbiancolin]

Could you make a WithFireSimDefaultFrequencies or something and compose the two of them to produce the old tweaks.

[davidbiancolin]

I had to unresolve this.

[abejgonzalez]

There is a distinction that needs to be made in the harness with these changes:

harnessClock/harnessReset =/= clock/reset in the TestHarness

In the case that the reference clock coming from ChipTop isn't the overall reference clock of the entire simulation (i.e. isn't the clock/reset coming into the TestHarness module), the harnessClock/harnessReset can differ from the implicit TestHarness clock/reset. Most of the time this won't be the case... but the current MulticlockAXIOverSerialConfig exposes this behavior. Here is a snippet of the clock summaries:

Here you can see that the reference clock for ChipTop is 500MHz but the reference clock for the entire system is actually 1GHz. So the implicit TestHarness clock/reset will be 1GHz while the harness<Clock/Reset> will be 500MHz.

[jerryz123]

In the example you provide, the mem_over_serial_tl_clock is the clock for the AXI memory?

[abejgonzalez]

In the example you provide, the mem_over_serial_tl_clock is the clock for the AXI memory?

Yup that's correct.

[alonamid]

Will this setup come with a docs section under the "Advanced Concepts" section?

[abejgonzalez]

Will this setup come with a docs section under the "Advanced Concepts" section?

Sure. Though I think I might re-work some of the "Communicating with the DUT" section a bit instead of adding a completely new page.

Edit: Probably need to have a separate page telling people that they can request clocks in the harness...

[abejgonzalez]

Ok this PR is ready for a final review. One thing to note though. If not using the load_mem feature in FireSim, loading a binary is extremely slow (have to send the binary over the serial link to the chip, then back over the serial link to DRAM). While this is slow to begin with, this is exacerbated by FESVR step size in FireSim being very very large (https://github.com/firesim/firesim/blob/7e32128bea5390ca528052dcdffb13656ee41162/sim/firesim-lib/src/main/cc/bridges/serial.cc#L20) which slows down the binary loading. A future PR can address this since this PR is fairly big to begin with.

[jerryz123]

This LGTM, although I think there can be some more deduplication that we can do later.

[alonamid]

LGTM (minor nit comment)

[colinschmidt]

Looks pretty good, a couple of comments/potential issues.
Also did we already have some discussion about what to do about images and the source that generated them? If so what was the decision?

[abejgonzalez]

Also did we already have some discussion about what to do about images and the source that generated them? If so what was the decision?

I think we had one in passing. And we didn't make a definitive decision. For now, I think the sources should not be put here since Chipyard is already big (we could just give a Drive link with the relevant sources and update them there).

[colinschmidt]

LGTM.
I think the image source size issue doesn't make sense to me as I bet the source for those pngs is smaller than the output, but I'm not willing to die on this bike shed.