Add dual and quad SPI support #479

elipsitz · 2024-09-14T15:56:50Z

Fixes #49

This set of commits introduces a few changes, in order to support dual and quad SPI:

SpiDriver::new_dual and SpiDriver::new_quad constructors, to allow creating a SpiDriver while specifying additional pins and modes.
LineWidth enum
And a TransactionExt struct (and functions that use it), which is an analog of the spi_transaction_ext_t in esp-idf.

By adding TransactionExt, this also enables hardware support for command, address, and dummy phases.

Overall, this is a little bit of a weird addition. I don't think it fits in super well with the existing API, but I couldn't think of a better way to do it.

In particular, some things that wouldn't work:

Adding LineWidth to Operation, to allow using the same methods: this won't work because Operation comes from embedded_hal, and can't be modified
Adding line width to SpiConfig: this seemed like a nice solution, but the problem is that multi-line spi devices generally don't do all their communications with quad spi -- often they have a 1-bit wide command phase, and then a 1 or 4-bit wide address, and then a 4-bit wide data phase. If this is just set on SpiConfig, then it wouldn't be possible to have narrower and wider phases.

Internally, everything had to be changed from spi_transaction_t to spi_transaction_ext_t -- this allows specifying the width of the command/address/dummy phases, which is needed because the older methods just use the data phase, and the newer methods (may) use those other phases.

The wider widths can only be used in half-duplex mode. With the current SPI api, there isn't a good way to encode this constraint with types (like in esp_hal: https://docs.esp-rs.org/esp-hal/esp-hal/0.20.0/esp32s3/esp_hal/spi/master/trait.HalfDuplexReadWrite.html). However, esp-idf checks this, and logs an error and returns an error code.

ivmarkov · 2024-09-15T09:23:17Z

@elipsitz Thank you for your contribution!

This is currently not passing CI. Would you look into that?

@Vollbrecht Given that you are the more knowledgeable than me in SPI, would you spend some time looking into this PR? I'll follow up with an API review, but I feel I need your feedback on the inner workings of quad / dual SPI support.

Vollbrecht

Thanks for digging into the SPI driver and try to improve it !

Since the spi driver is somewhat complicated and a bit intertwined i think we should try to focus on the two aspects with clear boundaries.

How to provide a ergonomic story for command / addr definition.
Support for (dual / quad / octo?) line support.

Yes its unfortunate that the embedded-hal Operation doesn't directly provide a mechanism for something like a command or address phase. Internally we map the Operation to an OperationIter enum variant. Maybe there is space here for us to squeeze in a way to define addr/comand as an operation. Though this is just a wild thought.

I am ok with bolting something onto the driver to make it work in general, but i want to be careful to not make the driver worse for people that just use it as a regular single line spi. I assume the switch to spi_transaction_ext_t makes hopefully only a minimal impact. We might want to test it before assuming it has no drastic impact. Though for basic dual / quad and/or (cmd/addr) support its not strictly needed to use spi_transaction_ext_t only when you additional want the a variable length in cmd/addr length right?

I will have a closer look later though that just my first thought that came to mind.

Vollbrecht · 2024-09-15T09:28:12Z

src/spi.rs

+        }
+    }
+
+    /// Half-duplex read


Why mark this as Half-Duplex? Does the dual and quad transaction behave differently than there single counter part. E.g Half and Full duplex have a special meaning -> You can have a "read only" in FullDuplex as in "HalfDuplex".

If i understand correctly you can have both "Half" and "Full-Duplex" in dual and quad mode. Though if you also want a command and address phase than you can only use "Half" duplex. But if you are running in "Half-Duplex" mode on dual/quad mode than you cannot use DMA memory afaik.

We need to be clear about the gotchas overall here.

@Vollbrecht I would assume this is clarified in the meantime. If so, would you close it?

src/spi.rs

elipsitz · 2024-09-15T17:07:57Z

Thanks for taking a look! Let me try to address each point:

This is currently not passing CI. Would you look into that?

Looks like that's because changing from spi_transaction_t to spi_transaction_ext_t increased the stack usage / future size past the Clippy threshold. I'll hold off for now as we discuss the approach we want to take.

Why mark this as Half-Duplex? Does the dual and quad transaction behave differently than there single counter part.

Marking it there might have been the wrong place, but from my understanding, you can only do dual/quad transactions in a half-duplex transaction.

"Full-duplex" would mean that the controller and peripheral are exchanging data at the same time -- with single SPI, that would be the controller clocking out data on MOSI, and the peripheral sending it back, simultaneously, in MISO.

"Half-duplex" still allows both to send data, but not at the same time. So in single-spi, you'd have the controller clocking out over MOSI, and then (or instead) clocking in data over MISO. OR, in 3-wire single spi, you'd have the same physical line used for both.

And dual/quad are similar, where you have 2 or 4 lines, but only one side is using them at a time. The ESP32 chips, from what I can tell, don't support any sort of full-duplex dual/quad mode (which I guess would be 4 lines, half being used by the controller, half being used by the peripheral)?

Anyway, the underlying esp-idf driver does verify all of this (e.g. you would need to set HalfDuplex in the Device Config in order to use dual/quad mode). But it's hard or impossible, given the current API, to enforce this at compile-time.

I assume the switch to spi_transaction_ext_t makes hopefully only a minimal impact.

There's no functional impact, but the main impact is that the struct is bigger (12 bytes larger each, afaik), so more stack space is used. Probably not a huge issue on the ESP32 chips (there's so much RAM!) but it does have an impact (e.g. see the Clippy lint causing the CI failures).

i think we should try to focus on the two aspects with clear boundaries.

Yeah, fair. I thought adding support for spi_transaction_ext_t was the simplest way to add dual/quad/octal support, but yeah there's definitely other ways to do that.

It also has other benefits (command/address/dummy -- it's more efficient to use hardware support than having separate Operations to mimick that with a delay between each phase).

Anyway, here's another idea: what about adding a SpiDeviceDriver::transaction_with_width? Similar to https://docs.esp-rs.org/esp-idf-hal/esp_idf_hal/spi/struct.SpiDeviceDriver.html#method.transaction, but it would look like:

pub fn transaction_with_width(
    &mut self,
    operations: &mut [(Operation<'_, u8>, LineWidth)],
) -> Result<(), EspError>

So a LineWidth is added to each Operation. Note that this is the esp_idf_hal::spi::Operation type, so we could modify the existing variants, but I don't think there's a backwards-compatible way to do that.

Another option would be to add additional variants to the Operation enum:

ReadWithWidth(&'a mut [Word], LineWidth),
WriteWithWidth(&'a [Word], LineWidth),

That fits in a little bit better but maybe starts overloading the Operation too much.

elipsitz · 2024-09-17T17:30:01Z

@Vollbrecht -- what do you think about the alternative transaction_with_width or ReadWithWidth / WriteWithWidth approaches? Do you think that's a better way to add dual/quad functionality?

Vollbrecht · 2024-09-17T20:38:24Z

Regarding the full_duplex stuff. Yeah i misremembered the datasheet a bit, i could have swarm that it had this wired mode to also allow to split input output lines in a 2/2 way, but that is total bogus :D

We try to be stable with our public api, though if it make sense i have no problem with a breaking change. But we should be smart about it and not willy nelly changing stuff for people that they actually don't wan't.

Luckily for us the SpiOperation is not public and thouse we have a bit of leeway here to play around anyway. I think that could be an escape hatch to add a 3rd variant, eg a ExtOperation(spi_transaction_ext_t), and leave the standard operation as is.

We would than internally need to adapt the run method a bit and when traversing the iter chain check if we either have a chained ExtOperation or a normal one. Well and then would duplicate maybe the other function that are used for the SpiBus stuff, for the case when you want to use ExtOperation. (Maybe bringing a arbitrary constraint in that the user can only use either Operation & delay or ExtOperant & delay if that makes our life simpler) With that our existing public api would not change much or anything at all, and we only would see additions. So that way we would put anything with cmd/addr and linewith into the ExtOperation, while not strictly needed.

At least that is were my thinking currently goes into a bit.

elipsitz · 2024-09-17T20:43:31Z

What would adding another SpiOperation variant with spi_transaction_ext_t add? The only change that switching from spi_transaction_t to spi_transaction_ext_t brings is the additional size of the struct, but if we add another enum variant, SpiOperation will grow to the largest enum variant size anyway.

Do you think it makes sense to explore adding line width support without command/address/dummy support? If so, which API makes the most sense to you? Adding pub fn transaction_with_width to SpiDeviceDriver, adding ReadWithWidth / WriteWithWidth to the public Operation enum, or something else?

Vollbrecht · 2024-09-17T22:48:32Z

Ok to take a step back. With the current approach as is you got

a implementation that has no breaking api change ( big plus)
slight overhead introduced since everything is now a spi_transaction_ext_t (minus)
- the current impl always sets dynamic addr / cmd length (slight minus)
the ExtTransaction api always create only a single transaction and doesn't take a slice of transactions, so it breaks a bit with the rest of the api here in its usage pattern.
- For the SpiBus we always take just &[u8] slices and split them accordingly internally, though splitting with add/ cmd and data is a bit more complicated. So i understand here the usage of maybe a single transaction type of api.
- For the SpiDeviceDriver we have both single transactions but also the &[Operation] pattern.

You are using spi_transaction_ext_t so you also get dynamic length of addr/ cmd and the dummy bits. Is that something you want specifically? I am asking since its not needed for addr/cmd if one wants to live with both beeing a fixed default size. Its also not needed for the linewith since this is set in spi_transaction_t.

I think if we would just go with the PR as it's now but only go back to spi_transaction_t then i would not see any big blockers. Though maybe still adding a option on the SpiDevice api that would allow to pass something like a &[ExtOperation] since sometimes you want a chip select cycle for multiple operations, and this can only work if we have some sort iterative list.

I will sleep over it and hopefully give tomorrow a bit more feedback :D

elipsitz · 2024-09-17T23:02:15Z

Though maybe still adding a option on the SpiDevice api that would allow to pass something like a &[ExtOperation] since sometimes you want a chip select cycle for multiple operations, and this can only work if we have some sort iterative list.

Yeah, to be useful, if we don't support command/address phases, we would need a way of putting multiple operations with different line widths in the same transaction (same lowering of the CS pin).

The extending the transaction function seems like the best way to do that, which we can do by expanding the Operation enum to add width (in a backwards-compatible way, with new variants rather than changing existing ones). Only Read and Write make sense to have a width, hence my suggestion of a new ReadWithWidth and WriteWithWidth. (edit: nope, Operation comes from embedded_hal, so we can't do this)

ivmarkov · 2024-09-20T06:51:54Z

The extending the transaction function seems like the best way to do that, which we can do by expanding the Operation enum to add width (in a backwards-compatible way, with new variants rather than changing existing ones). Only Read and Write make sense to have a width, hence my suggestion of a new ReadWithWidth and WriteWithWidth.

I don't follow the conversation too closely, but I also don't want to keep us stuck on this topic and you waiting on our feedback.
So how about you just go with ReadWithWidth and WriteWithWidth?

ivmarkov · 2024-09-20T06:54:03Z

BTW, since Operation is NOT marked with #[non_exhaustive], adding the extra variants to it would be - strictly speaking - non-backwards compatible. But the next release of the esp-idf-* crates anyways breaks compatibility here and there, and this particular case is a "small breakage" anyway, as I don't expect the user code to match on the Operation of course. It is the driver that does it.

elipsitz · 2024-09-22T22:00:36Z

Oops... the rustdoc is misleading. Operation is actually a re-export of https://docs.rs/embedded-hal/latest/embedded_hal/spi/enum.Operation.html -- so we can't add the variants there. I actually mentioned this in my first post, I just forgot.

Adding a few fn transaction_with_width, then, seems like the most unintrusive way to add this functionality.

ivmarkov · 2024-09-23T08:34:05Z

Oops... the rustdoc is misleading. Operation is actually a re-export of https://docs.rs/embedded-hal/latest/embedded_hal/spi/enum.Operation.html -- so we can't add the variants there. I actually mentioned this in my first post, I just forgot.

Adding a few fn transaction_with_width, then, seems like the most unintrusive way to add this functionality.

We could also add our own Operation enum, and then implement From<embedded_hal::spi::Operation> for Operation.
And just alter all e-hal trait implementations to call operation.into()?

elipsitz · 2024-09-23T15:36:19Z

The SPI api is fairly unintuitive at the moment -- I personally had some difficulty understanding which structs to use and how to compose them when I first started using this crate -- so I'd be wary of bolting on another Operation enum without re-evaluating the complexity of the API.

But anyway, if we had our own, crate specific Operation, it seems reasonable to have that be the entrypoint for exposing the additional features of esp-idf/esp32 beyond what embedded_hal exposes... which brings this back to spi_transaction_t / spi_transaction_ext_t, and the command/address/data phases.

ivmarkov · 2024-09-23T16:21:54Z

The SPI api is fairly unintuitive at the moment -- I personally had some difficulty understanding which structs to use and how to compose them when I first started using this crate -- so I'd be wary of bolting on another Operation enum without re-evaluating the complexity of the API.

I'm not sure I follow how having our own Operation is increasing the complexity of the API. Even now, we do have our own Operation if you squint a little. The fact that it is currently just a type-alias for the e-hal one was done by (me) out of pure laziness - yet - it is something the user should neither know, nor care about, as I would anticipate that either she is using the driver via the e-hal traits (and then she cares about the e-hal Operation), or via our own public API (and then she cares about our own Operation which just happens to currently be a type-alias).

But anyway, if we had our own, crate specific Operation, it seems reasonable to have that be the entrypoint for exposing the additional features of esp-idf/esp32 beyond what embedded_hal exposes... which brings this back to spi_transaction_t / spi_transaction_ext_t, and the command/address/data phases.

Do you mean the fact that we have to swallow the extra 12-bytes of spi_transaction_ext_t if we unconditionally switch to it?
(BTW not really sure how we ended up with 12 bytes extra. I do get it, that alignof(spi_transaction_t) is likely u64, but these are only 8 bytes, and the ext struct only adds 3 extra bytes.

Anyway... I'm thinking... maybe if the difference between the existing future sizes and the new ones is within 10% - 15% perhaps we can live with that?

elipsitz · 2024-09-23T16:43:51Z

Even now, we do have our own Operation if you squint a little. The fact that it is currently just a type-alias for the e-hal one was done by (me) out of pure laziness - yet - it is something the user should neither know, nor care about

Ah, I see your point. If you're okay with this breaking change (changing the re-export of Operation to a custom Operation), that seems fine to me. I suppose, for some level of compatibility, you'd want to keep the Operation name for the new enum.

BTW not really sure how we ended up with 12 bytes extra.

I think I just misread the struct. It's probably only 4 or 8 more bytes. Although the futures seemed to grow by quite a bit -- I probably just need to look into that.

So as Vollbrecht pointed out, I conflated two things originally: adding line-width support, and supporting command/address/data phases.

Copying the existing embedded_hal::spi::Operation enum into this crate, and then adding ReadWithWidth and WriteWithWidth variants would work for line-width support. At some point in the future, a TransactionExt variant could be added, with command/address/data support too.

Does that seem reasonable? To keep compatibility, Operation would have to stay an enum with the same variants.

Additionally, SpiBusDriver doesn't have any method that takes Operation. Adding a transaction and transaction_async would match the other two (e.g. https://docs.esp-rs.org/esp-idf-hal/esp_idf_hal/spi/struct.SpiDeviceDriver.html#method.transaction), but it's arguably not actually a "transaction" on SpiBusDriver. I think it makes sense to keep the same names for consistency though.

To summarize:

Copy embedded_hal operation into this crate (breaking change, hopefully not too bad)
Add ReadWithWidth and WriteWithWidth variants
Add transaction and transaction_async to SpiBusDriver?
Some day in the future, perhaps even soon, add TransactionExt variant to Operation.

ivmarkov · 2024-09-23T17:17:42Z

Give me a few hours till tmr to think about this with a fresh head.

I think the plan is OK except introducing transaction* named methods to SpiBusDriver, as you are 100% correct that there are no "transactions" when you own the bus, so this sounds confusing to me. Maybe it is just about how to name these methods.

But let me think about this tmr morning...

ivmarkov · 2024-09-24T08:57:11Z

To summarize:

Copy embedded_hal operation into this crate (breaking change, hopefully not too bad)

Yes.

Add ReadWithWidth and WriteWithWidth variants

Yes.

Add transaction and transaction_async to SpiBusDriver?

Maybe operation and operation_async instead, where these take a single esp-idf-hal Operation instance?
If the user wants to do multiple operations on the bus, she just needs to call the operation(_async) method repeatedly.

Some day in the future, perhaps even soon, add TransactionExt variant to Operation.

Yes. I would not name it TransactionExt though. We are conflating two different meanings of a "transaction":

The notion of ESP IDF "transaction" (spi_transaction(_ext)_t). This is only internal, and not exposed in the API hence we should not talk about it
The notion of e-hal (and this driver's public API's) "transaction". A transaction - in the driver's public API sense is a sequence of operations (plus optional delays) which starts with CS raised and ends with CS lowered.

So if we ever have a TransactionExt variant of our new Operation it should problably be named just Ext or something.

elipsitz · 2024-09-25T19:56:13Z

Ok, I've made those changes.

For SpiBusDriver::operation, I've simplified the implementation by making it so that Operation::DelayNs returns an error. There isn't a point of passing in a Delay (unless you're passing stuff in dynamically).

CI is failing because Clippy can't build esp-idf-sys? Unclear, it doesn't seem to be related to these changes.

Vollbrecht · 2024-09-25T21:10:32Z

I hope i can go through it on the weekend still a bit distracted by work 🥲

The CI seams to be broken because they updated the tooling version they ship within the "release/tag" builds ... Though that should not break things ... This problem was introduced here.

I think the examples/spi_loopback_async.rs:65:21 will not work anymore because of the change on Operation. E.g the removal of the u8 argument to it. Can you check that out please? Thanks !

This replaces the Operation enum which was previously re-exported from embedded_hal. As of this commit, the enum is identical to the embedded_hal Operation, except that there is no Word type parameter, as esp-idf-hal only uses u8 for SPI. Additionally, the `#[non_exhaustive]` attribute has been added to allow for future expansion.

elipsitz · 2024-09-26T00:45:37Z

I think the examples/spi_loopback_async.rs:65:21 will not work anymore because of the change on Operation.

Ah yeah, good point. It's not removing u8 that breaks this specific one, but the type is different. Removing the use embedded_hal::spi::Operation for this and the non-async one fixes the issue

I checked this with cargo +esp check --example spi_loopback

Half-duplex read transactions should not use a write buffer. Previously, only 3-wire half-duplex transactions followed this, but the esp-idf error-checking logic applies to all half-duplex transactions.

src/spi.rs

ivmarkov · 2024-09-26T07:12:53Z

I checked this with cargo +esp check --example spi_loopback

Please also do cargo +esp clippy

elipsitz · 2024-09-26T13:19:16Z

Clippy passes too

ivmarkov · 2024-09-26T14:27:05Z

@elipsitz Let's wait until @Vollbrecht can take a final look at this over the weekend, as I'm not really into the details of dual/quad etc. SPI semantics and I won't have the time to do a deep dive.

As for API/stylistic comments - I don't have any myself so from my POV this is good for merging.

This adds support for multi-line half-duplex SPI operations.

This allows the use of the Operation enum with SpiBusDriver, enabling support for multi-line transfers.

Vollbrecht · 2024-09-29T15:27:09Z

src/spi.rs

-            if duplex == Duplex::Half3Wire {
-                0
-            } else {
+            if duplex == Duplex::Full {


Did you test this here? I think this is wrong from past memory. The underlying parameters here are total_length and rx_length.

I think Half3Wire was the odd one here, but for HalfDuplex you have to set the total_length to the same as the rx_length here. E.g the total length cannot be smaller than the rx_length.

Yeah, I had to set this otherwise this assertion fails:

https://github.com/espressif/esp-idf/blob/46acfdce969f03c02b001fe4d24fa9e98f6adc5e/components/esp_driver_spi/src/gpspi/spi_master.c#L1067

I did test it and it all works on my device (single, dual, quad). But honestly I’m pretty confused about how these fields are intended to be used, so it would be great to double check that.

On what type of esp did you test it? By a quick glance it seams to only assert on non esp32/s2 variants. If that is the case and it works on esp32 differently we may need to add a cfg flag here for conditional compilation based on that.

the corresponding flag would be [#cfg(esp_idf_soc_spi_hd_both_inout_supported)]

Oh, this was on an ESP32-S3. Makes sense then— I won’t be able to test it properly though on an ESP32.

I’m surprised the API would work so differently though between the two devices though. The way it’s worded makes it sound like it’s an actual behavior difference, a hardware feature that either is or isn’t supported on different devices.

Than for now lets add this cfg flag, and if its set, set the length to the same as rx_length, and if its not present put a 0 in it like you did.

If you look at the line above in esp-idf it also mentions this configuration not working on ESP32 when DMA is enabled.

The wording “SPI half duplex mode does not support using DMA with both MOSI and MISO phases.”, to me, sounds like it’s a mode where you first do a write and then do a read (or vice versa). But nothing in this crate would map to that— you’d send separate Operations if you wanted that. So this behavior, to me, sounds like something that’s undesirable anyway.

Yeah it would essentially mean this feature would only fully working on a s2 and its niche. Maybe there are some perf benefits in that specific scenario, though that's not relevant here.

Let's bring the PR into master and see of other people start complaining shell we :D

Vollbrecht

Overall looks good!

if we can clarify the spi_read_transaction chunk_len stuff than i think we are good to merge. Though i want to have this double checked as we had bugs in the past here at the exact same point.

Vollbrecht reviewed Sep 15, 2024

View reviewed changes

elipsitz added 2 commits September 24, 2024 18:16

spi: Add SpiDriver new_dual and new_quad constructors

c0e403c

spi: Add LineWidth enum

522f3ed

elipsitz force-pushed the quad-spi branch from c8b2345 to 426e73c Compare September 25, 2024 19:43

elipsitz force-pushed the quad-spi branch from 426e73c to c5a059c Compare September 26, 2024 00:43

spi: Fix half-duplex read transactions

c9ac2c5

Half-duplex read transactions should not use a write buffer. Previously, only 3-wire half-duplex transactions followed this, but the esp-idf error-checking logic applies to all half-duplex transactions.

elipsitz force-pushed the quad-spi branch from c5a059c to 3ce2f11 Compare September 26, 2024 01:56

elipsitz mentioned this pull request Sep 26, 2024

CI broken due to esp-idf tool 'ninja' extracted as non-executable #485

Open

ivmarkov reviewed Sep 26, 2024

View reviewed changes

src/spi.rs Show resolved Hide resolved

ivmarkov reviewed Sep 26, 2024

View reviewed changes

src/spi.rs Show resolved Hide resolved

elipsitz added 2 commits September 27, 2024 14:27

spi: Add Operation::ReadWithWidth and Operation::WriteWithWidth

e42807f

This adds support for multi-line half-duplex SPI operations.

spi: Add SpiBusDriver::operation and operation_async

22b81ea

This allows the use of the Operation enum with SpiBusDriver, enabling support for multi-line transfers.

elipsitz force-pushed the quad-spi branch from 3ce2f11 to 22b81ea Compare September 27, 2024 18:27

Vollbrecht reviewed Sep 29, 2024

View reviewed changes

Vollbrecht merged commit bba51cd into esp-rs:master Sep 29, 2024
4 of 10 checks passed

Add dual and quad SPI support #479

Add dual and quad SPI support #479

Conversation

elipsitz commented Sep 14, 2024

ivmarkov commented Sep 15, 2024

Vollbrecht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elipsitz commented Sep 15, 2024 • edited Loading

elipsitz commented Sep 17, 2024

Vollbrecht commented Sep 17, 2024 • edited Loading

elipsitz commented Sep 17, 2024

Vollbrecht commented Sep 17, 2024 • edited Loading

elipsitz commented Sep 17, 2024 • edited Loading

ivmarkov commented Sep 20, 2024

ivmarkov commented Sep 20, 2024

elipsitz commented Sep 22, 2024 • edited Loading

ivmarkov commented Sep 23, 2024

elipsitz commented Sep 23, 2024

ivmarkov commented Sep 23, 2024 • edited Loading

elipsitz commented Sep 23, 2024 • edited Loading

ivmarkov commented Sep 23, 2024

ivmarkov commented Sep 24, 2024 • edited Loading

elipsitz commented Sep 25, 2024

Vollbrecht commented Sep 25, 2024

elipsitz commented Sep 26, 2024

ivmarkov commented Sep 26, 2024

elipsitz commented Sep 26, 2024

ivmarkov commented Sep 26, 2024

Vollbrecht Sep 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vollbrecht Sep 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vollbrecht left a comment

Choose a reason for hiding this comment

elipsitz commented Sep 15, 2024 •

edited

Loading

Vollbrecht commented Sep 17, 2024 •

edited

Loading

Vollbrecht commented Sep 17, 2024 •

edited

Loading

elipsitz commented Sep 17, 2024 •

edited

Loading

elipsitz commented Sep 22, 2024 •

edited

Loading

ivmarkov commented Sep 23, 2024 •

edited

Loading

elipsitz commented Sep 23, 2024 •

edited

Loading

ivmarkov commented Sep 24, 2024 •

edited

Loading

Vollbrecht Sep 29, 2024 •

edited

Loading

Vollbrecht Sep 29, 2024 •

edited

Loading