VOM should track Presence vrefs used in virtualized data, to inhibit dropImport
#3133
Assignees
Labels
Comments
9 tasks
This was referenced May 21, 2021
warner
added a commit
that referenced
this issue
May 22, 2021
If userspace puts a Presence into the `state` of a virtual object, or somewhere inside the value stored in vref-keyed `makeWeakStore()` entry, it gets serialized and stored as a vref, which doesn't (and should not) keep the Presence object alive. Allowing this Presence to leave RAM, remembering only the vref on disk, is a non-trivial part of the memory savings we obtain by using virtualized data. However, just because there is currently no Presence (for a given vref) does *not* mean that the vat cannot reach the vref. Liveslots will observe the Presence being collected (when the finalizer runs), but if the vref is still stored somewhere in virtualized data, liveslots must not emit a `syscall.dropImport` for it. This changes the virtual object manager to keep track of Presences used in virtualized data, and remember their vref in a Set. When liveslots' wants to `dropImport` a vref that no longer has a Presence, it will ask the VOM first. With this Set, the VOM can inhibit the `dropImport` call until later. At this stage, we simply add the vref to a Set and never remove it. This is safe but conservative. In the future, we'll need some form of refcounting to detect when the vref is no longer mentioned anywhere in virtualized data. At that point, the VOM will need to inform liveslots (or some sort of "reachability manager") that the VOM no longer needs the vref kept alive. The `syscall.dropImport` can be sent when neither the VOM nor a Presence is causing the vref to remain reachable. closes #3133 refs #3106
warner
added a commit
that referenced
this issue
May 24, 2021
If userspace puts a Presence into the `state` of a virtual object, or somewhere inside the value stored in vref-keyed `makeWeakStore()` entry, it gets serialized and stored as a vref, which doesn't (and should not) keep the Presence object alive. Allowing this Presence to leave RAM, remembering only the vref on disk, is a non-trivial part of the memory savings we obtain by using virtualized data. However, just because there is currently no Presence (for a given vref) does *not* mean that the vat cannot reach the vref. Liveslots will observe the Presence being collected (when the finalizer runs), but if the vref is still stored somewhere in virtualized data, liveslots must not emit a `syscall.dropImport` for it. This changes the virtual object manager to keep track of Presences used in virtualized data, and remember their vref in a Set. When liveslots' wants to `dropImport` a vref that no longer has a Presence, it will ask the VOM first. With this Set, the VOM can inhibit the `dropImport` call until later. At this stage, we simply add the vref to a Set and never remove it. This is safe but conservative. In the future, we'll need some form of refcounting to detect when the vref is no longer mentioned anywhere in virtualized data. At that point, the VOM will need to inform liveslots (or some sort of "reachability manager") that the VOM no longer needs the vref kept alive. The `syscall.dropImport` can be sent when neither the VOM nor a Presence is causing the vref to remain reachable. closes #3133 refs #3106
warner
added a commit
that referenced
this issue
May 25, 2021
If userspace puts a Presence into the `state` of a virtual object, or somewhere inside the value stored in vref-keyed `makeWeakStore()` entry, it gets serialized and stored as a vref, which doesn't (and should not) keep the Presence object alive. Allowing this Presence to leave RAM, remembering only the vref on disk, is a non-trivial part of the memory savings we obtain by using virtualized data. However, just because there is currently no Presence (for a given vref) does *not* mean that the vat cannot reach the vref. Liveslots will observe the Presence being collected (when the finalizer runs), but if the vref is still stored somewhere in virtualized data, liveslots must not emit a `syscall.dropImport` for it. This changes the virtual object manager to keep track of Presences used in virtualized data, and remember their vref in a Set. When liveslots' wants to `dropImport` a vref that no longer has a Presence, it will ask the VOM first. With this Set, the VOM can inhibit the `dropImport` call until later. At this stage, we simply add the vref to a Set and never remove it. This is safe but conservative. In the future, we'll need some form of refcounting to detect when the vref is no longer mentioned anywhere in virtualized data. At that point, the VOM will need to inform liveslots (or some sort of "reachability manager") that the VOM no longer needs the vref kept alive. The `syscall.dropImport` can be sent when neither the VOM nor a Presence is causing the vref to remain reachable. closes #3133 refs #3106
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the Problem Being Solved?
As described in #3106 (the section named "VOM tracks Presences in virtualized data, add-only"), if a Presence is used in the
stateof a virtual object or the value of a virtualizedWeakStore, only it's vref is stored, not the Presence JSObjectitself. If/when userspace drops the tangible Presence object, the liveslots GC finalizers will take note, and they want to callsyscall.dropImportto let the kernel know that this vat can no longer reach that Presence (allowing the upstream Remotable to be dropped, eventually). But if some virtual data can be reanimated to re-introduce a new Presence object for that same vref, then the vref isn't really unreachable.When the finalizer runs, the liveslots reachability manager will be asking the virtual object manager about the vref, and the VOM needs to be able to say "I can still reach the vref" or "I cannot".
To answer this question correctly requires refcounting of each
vref(and a return notification mechanism to say "hey reachability manager, earlier I said yes, but that answer just changed to no, so if you don't still have a Presence for that vref, you should dodropImportnow"). But we can defer that work because we don't have any users of this feature yet: the only thing that goes intostateor virtualized data is one Remotable per ERTP issuer (a Brand).So the immediate task is to simply remember a Set of all the Presence
vrefsthat have ever been included in virtualized data, and use that to conservatively answer the reachability manager's question.Description of the Design
reachableVrefsSet whose type is strings (vrefs)statesetter or WeakStore.set()), scan the slots and identify the Presences (o-NN).isVrefReachable(vref), which just doesreturn reachableVrefs.has(vref)Later, when #2660 prepares to call
syscall.dropImport, it will ask the VOM first.Test Plan
Unit tests which create a VOM, forge a Presence, ask
.isVrefReachable()and get backfalse, add the Presence to WeakStore value, ask.isVrefReachable()again and get backtrue. Then a separate test which does the same thing for thestateof a virtual object.For both tests, we should then delete the item and check that
isVrefReachable()still returnstrue. We'll add a note to this test to remind ourselves that a complete implementation would returnfalse, and to change the assertion when we finish that task some day.The text was updated successfully, but these errors were encountered: