Changes to support moving immix #27

udesou · 2023-08-22T23:49:26Z

Opening this draft PR to keep track of the changes to support moving immix. So far they are:
(1) pinning objects that are hashed. Note that Julia hashes on the address of the object, which means that we might get errors if we move an object that has been hashed.
(2) pinning buffer objects: these can be shared between objects, and there are some invariants that should be enforced (if a->is_shared is set to true, I believe we cannot move the buffer), to be on the safe side, we're pinning these objects.
(3) pinning types and typenames: these are used when scanning objects, and not traced from the objects as they are stored in the objects' header.
For (1), until we implement address space hashing, we need to make sure these objects do not move. For (2) and (3), we should be simply allocate these objects in a non-moving space.
There are also small bug fixes. Eg, Julia does not put a reference on the shadow stack, because it comes from a global root. In that case, if that object moves, the pointer to it is not updated. Finally I've also added code to clear the stack of dead tasks, so I have removed the static attribute from free_stack and _jl_free_stack in order to reuse these functions from MMTk.

NB: merge after mmtk/mmtk-core#897 in order to support transitively pinned roots.

…en realloc-ing if pinned; removing static from functions needed to sweep live_tasks array

…a's debug build

…ld be 0 when using MMTk

It's possible for PiNodes to effectively imply statically the condition of a Core.ifelse. For example: ```julia 23 ─ %60 = Core.ifelse(%47, false, true)::Bool │ %61 = Core.ifelse(%47, %58, false)::Union{Missing, Bool} 25 ─ goto #27 if not %60 26 ─ %65 = π (%61, Bool) └─── ... ``` In basic block #26, the PiNode gives us enough information to conclude that `%47 === false` if control flow reaches that point. The previous code incorrectly assumed that this kind of pruning would only be done for PhiNodes. Resolves JuliaLang#50276

qinsoon

LGTM

src/threading.c

src/julia_internal.h

qinsoon · 2023-09-13T06:23:16Z

src/builtins.c

@@ -392,6 +395,9 @@ static uintptr_t immut_id_(jl_datatype_t *dt, jl_value_t *v, uintptr_t h) JL_NOT
        return ~h;
    size_t f, nf = jl_datatype_nfields(dt);
    if (nf == 0 || (!dt->layout->haspadding && dt->layout->npointers == 0)) {
+        if(mmtk_object_is_managed_by_mmtk(v)) {
+            mmtk_pin_object(v);


For mmtk/mmtk-julia#104, probably we will introduce jl_gc_pin or something like that. But it is fine for now.

…nning owner to avoid introspecting it during scanning

…specs

This PR adds support for (partially) moving objects in Julia and should be merged with mmtk/julia#27 and mmtk/mmtk-core#897. - It adds support for pinning/unpinning objects and checking if an object is pinned (the implementation uses a pin bit). (`mmtk_pin_object`, `mmtk_unpin_object` and `mmtk_is_pinned`) - It adds support for providing transitively pinned (`tpinned`) roots . - It implements the `copy` function in `object_model.rs`. Note that arrays with inlined data must be treated specially, as their `a->data` pointer needs to be updated after copying. - It uses Julia's GC bits to store forwarding information and the object's header to store the forwarding pointer. - Currently, all stack roots are transitively pinned. Note that we also need to traverse the `tls->live_tasks` to make sure that any stack root from these tasks are transitively pinned. - `scan_julia_object` had to be adapted to cover a few corner cases: - when an array object contains a pointer to the owner of the data, `a->data` needs to be updated in case the owner moves. - the `using` field inside a `jl_module_t` may also be inlined inside the module, and if that's the case, we need to make sure that field is updated if the module moves. - when marking finalizers, traversing the list of malloced arrays, and the list of live tasks at the end of GC, we need to updated these lists with objects that have possibly been moved. - Added a few debug assertions to capture scanning of misaligned objects and roots. NB: I've only tested moving immix; sticky immix is still non-moving. --------- Co-authored-by: Luis Eduardo de Souza Amorim <eduardo@shrew.moma> Co-authored-by: Luis Eduardo de Souza Amorim <eduardo@bear.moma> Co-authored-by: mmtkgc-bot <mmtkgc.bot@gmail.com>

This PR adds support for (partially) moving objects in Julia and should be merged with mmtk/julia#27 and mmtk/mmtk-core#897. - It adds support for pinning/unpinning objects and checking if an object is pinned (the implementation uses a pin bit). (`mmtk_pin_object`, `mmtk_unpin_object` and `mmtk_is_pinned`) - It adds support for providing transitively pinned (`tpinned`) roots . - It implements the `copy` function in `object_model.rs`. Note that arrays with inlined data must be treated specially, as their `a->data` pointer needs to be updated after copying. - It uses Julia's GC bits to store forwarding information and the object's header to store the forwarding pointer. - Currently, all stack roots are transitively pinned. Note that we also need to traverse the `tls->live_tasks` to make sure that any stack root from these tasks are transitively pinned. - `scan_julia_object` had to be adapted to cover a few corner cases: - when an array object contains a pointer to the owner of the data, `a->data` needs to be updated in case the owner moves. - the `using` field inside a `jl_module_t` may also be inlined inside the module, and if that's the case, we need to make sure that field is updated if the module moves. - when marking finalizers, traversing the list of malloced arrays, and the list of live tasks at the end of GC, we need to updated these lists with objects that have possibly been moved. - Added a few debug assertions to capture scanning of misaligned objects and roots. NB: I've only tested moving immix; sticky immix is still non-moving. --------- Co-authored-by: Luis Eduardo de Souza Amorim <eduardo@shrew.moma> Co-authored-by: Luis Eduardo de Souza Amorim <eduardo@bear.moma> Co-authored-by: mmtkgc-bot <mmtkgc.bot@gmail.com> (cherry picked from commit 1622162) # Conflicts: # julia/mmtk_julia.c # mmtk/Cargo.lock # mmtk/Cargo.toml # mmtk/api/mmtk.h # mmtk/src/julia_scanning.rs # mmtk/src/lib.rs # mmtk/src/scanning.rs

Changes to support moving immix

Testing: - with a package error ``` (SimpleLooper) pkg> precompile Precompiling all packages... ✗ SimpleLooper 0 dependencies successfully precompiled in 2 seconds ERROR: The following 1 direct dependency failed to precompile: SimpleLooper Failed to precompile SimpleLooper [ff33fe5-d8e3-4cbd-8bd9-3d2408ff8cab] to "/Users/ian/.julia/compiled/v1.12/SimpleLooper/jl_PQArnH". ERROR: LoadError: Stacktrace: [1] error() @ Base ./error.jl:53 ``` - with interrupt ``` (SimpleLooper) pkg> precompile Precompiling all packages... ^C Interrupted: Exiting precompilation... ◒ SimpleLooper 1 dependency had output during precompilation: ┌ SimpleLooper │ [57879] signal 2: Interrupt: 2 │ in expression starting at /Users/ian/Documents/GitHub/SimpleLooper.jl/src/SimpleLooper.jl:2 └ ``` - an internal error simulated in the same scope that JuliaLang/Pkg.jl#3984 was failing to throw from ``` JULIA stdlib/release.image Unhandled Task ERROR: Stacktrace: [1] error() @ Base ./error.jl:53 [2] (::Base.Precompilation.var"mmtk#27#65"{Bool, Bool, Vector{Task}, Dict{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}, String}, Dict{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}, String}, Base.Event, Base.Event, ReentrantLock, Vector{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}}, Dict{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}, String}, Vector{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}}, Int64, Vector{Base.PkgId}, Dict{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}, Bool}, Dict{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}, Base.Event}, Dict{Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}, Bool}, Vector{Base.PkgId}, Dict{Base.PkgId, String}, Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, Base.Precompilation.var"#color_string#38"{Bool}, Bool, Base.Semaphore, Bool, String, Vector{String}, Vector{Base.PkgId}, Base.PkgId, Base.CacheFlags, Cmd, Pair{Cmd, Base.CacheFlags}, Tuple{Base.PkgId, Pair{Cmd, Base.CacheFlags}}})() @ Base.Precompilation ./precompilation.jl:819 ```

udesou added 4 commits August 3, 2023 09:44

Supporting moving immix (wip)

ae2fa58

Pushing current task into the stack to set as red root; pin string wh…

2f21eec

…en realloc-ing if pinned; removing static from functions needed to sweep live_tasks array

Fixing issue that prevented building Julia or running tests with Juli…

5fe96d7

…a's debug build

Fixing issue when setting up the number of stock GC threads - it shou…

7985bb2

…ld be 0 when using MMTk

udesou mentioned this pull request Aug 23, 2023

Supporting moving immix mmtk/mmtk-julia#93

Merged

Merge branch 'mmtk:master' into feature/rb-roots

9314e70

udesou requested a review from qinsoon August 23, 2023 06:20

udesou added 2 commits September 6, 2023 06:16

Merge branch 'master' into feature/rb-roots

bbe1f1d

Merge remote-tracking branch 'udesou/master' into feature/rb-roots

d14df55

qinsoon marked this pull request as ready for review September 13, 2023 05:56

qinsoon approved these changes Sep 13, 2023

View reviewed changes

src/threading.c Outdated Show resolved Hide resolved

src/julia_internal.h Show resolved Hide resolved

qinsoon reviewed Sep 13, 2023

View reviewed changes

udesou added 5 commits September 15, 2023 06:49

Checking if object is managed by mmtk before calling pin function; Pi…

f9da153

…nning owner to avoid introspecting it during scanning

Fixing duplicate code from merging mistake

cfb6d90

Merge branch 'master' into feature/rb-roots

25c06e0

Increasing the timeout just in case it's caused by the github runner …

a0c895e

…specs

Refactoring pinning functions and adding comments

3b4ae53

udesou merged commit 861f151 into mmtk:master Feb 12, 2024
1 check passed

udesou added a commit to udesou/julia that referenced this pull request Feb 13, 2024

Merge pull request mmtk#27 from udesou/feature/rb-roots

5a812ad

Changes to support moving immix

udesou mentioned this pull request Feb 13, 2024

Changes to support moving immix #36

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to support moving immix #27

Changes to support moving immix #27

udesou commented Aug 22, 2023 •

edited

Loading

qinsoon left a comment

qinsoon Sep 13, 2023

Changes to support moving immix #27

Changes to support moving immix #27

Conversation

udesou commented Aug 22, 2023 • edited Loading

qinsoon left a comment

Choose a reason for hiding this comment

qinsoon Sep 13, 2023

Choose a reason for hiding this comment

udesou commented Aug 22, 2023 •

edited

Loading