Add a 'skip' parameter to writev1 so that the beginning of a car can … #291
Conversation
|
The remaining potential work here is allow skip to be an arbitrary byte offset in the car file. currently it's limited to block boundaries, and because of the API it's difficult to know which block boundary nearest to the requested skip offset has been used. |
|
I cleaned up the tee-ing writer and added a test that seems to verify that we can use this to advance to arbitrary byte offsets in a car. @rvagg maybe can provide a first round of review |
| type counter struct { | ||
| totalRead uint64 | ||
| // Counter tracks how much data has been read. | ||
| type Counter struct { |
There was a problem hiding this comment.
why do we use a struct for this? is it something to do with needing a pointer to it? why isn't a *uint64 good enough for this purpose?
|
|
||
| // TraverseResumer allows resuming a progress from a previously encountered path in the selector. | ||
| type TraverseResumer interface { | ||
| RewindToPath(from datamodel.Path) error |
There was a problem hiding this comment.
am I right in assuming we don't have a use-case for this at the moment, that we're just supporting it because it's not a difficult extension of doing it to an offset?
got thoughts on what a use-case for might me?
There was a problem hiding this comment.
This is probably the most useful thing to have pointed out. I think the more useful method to expose here will be a RewindToOffset in addition to the current RewindToPath. I'll work on adding that
| var _ io.ReadSeeker = (*SkipWriterReaderSeeker)(nil) | ||
|
|
||
| // NewSkipWriterReaderSeeker creates an io.ReadSeeker around a ReWriter. | ||
| func NewSkipWriterReaderSeeker(ctx context.Context, size uint64, cons ReWriter) *SkipWriterReaderSeeker { |
There was a problem hiding this comment.
could we have a better name than cons? it's not obvious to me what this name is supposed to be (consume?), it's puzzling when we get to c.cons() and it seems like it should be more obvious what it's doing - should it be read as "start consuming now"?
|
This is some super-impressive code @willscott, once I got my head around it. The scope is large and it's doing some really complex stuff and I can't really fault it from a review. Most of my suggested changes are just comments I think might be helpful for anyone else following the breadcrumbs in future. I'm now toying with integrating it into Boost and will report back. |
|
(FYI I messed this branch up a tiny bit so ended up rebasing it on master, removing the merge commits in the process, so the commits have changed but the resulting code is the same) |
|
@rvagg are we able to merge this - was there something else we were waiting for on it? |
|
Yeah, it needs #327 to get closer to working as advertised, but even then it's still not quite right. I stepped back from working on this as the complexity increased and I started questioning the utility since we mainly want this (for now) for the explore-all case and it's the selectors that make this difficult to properly do. As it is, it can't replace Boost's impl because of caching enabled by explore-all shortcuts that we're not allowing ourselves to touch here. I'd be fine merging in #327 to here, but I'm not sure we should merge this in since it's not going to give people what it says it does. |
|
I forgot to add that I'm not intending to suggest this is a lost cause. My thinking, the last time I touched this, was that we should try to implement an efficient explore-all case here, and then iterate on that for the more complex cases. |
Remaining: