Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Indeterministic illegal instruction: 4 #1626

Closed
mhauru opened this issue Jul 10, 2024 · 17 comments
Closed

Indeterministic illegal instruction: 4 #1626

mhauru opened this issue Jul 10, 2024 · 17 comments
Labels
gc garbage collection more information needed segfault

Comments

@mhauru
Copy link
Contributor

mhauru commented Jul 10, 2024 

module MWE

import DynamicPPL
import AbstractPPL
import Accessors
import BangBang
import LogDensityProblems
import LogDensityProblemsAD
import ADTypes
import Random
using Turing: MvNormal, LogDensityFunction

using Enzyme

Random.seed!(23)

Enzyme.API.runtimeActivity!(true)

function satellite_model_matrix(__model__::DynamicPPL.Model, __varinfo__::DynamicPPL.AbstractVarInfo, __context__::AbstractPPL.AbstractContext, x0_mean::Any, P0::Any, var"##arg#286193"::(DynamicPPL.TypeWrap){TV}=(DynamicPPL.TypeWrap){Matrix{Float64}}();) where {TV}
    T = 2
    state_dim = 2
    x = TV(undef, state_dim, T)
    var"##dist#286197" = MvNormal(x0_mean, P0)
    var"##vn#286194" = (DynamicPPL.resolve_varnames)((AbstractPPL.VarName){:x}((AbstractPPL.concretize)((Accessors.opticcompose)((Accessors.IndexLens)((:, 1))), x)), var"##dist#286197")
    (var"##value#286198", __varinfo__) = (DynamicPPL.tilde_assume!!)(__context__, (DynamicPPL.unwrap_right_vn)((DynamicPPL.check_tilde_rhs)(var"##dist#286197"), var"##vn#286194")..., __varinfo__)
    x = (Accessors.set)(x, (BangBang.AccessorsImpl.prefermutation)((Accessors.opticcompose)((Accessors.IndexLens)((:, 1)))), var"##value#286198")
    var"##value#286198"
    var"##dist#286202" = MvNormal(x[:, 1], P0)
    var"##vn#286199" = (DynamicPPL.resolve_varnames)((AbstractPPL.VarName){:x}((AbstractPPL.concretize)((Accessors.opticcompose)((Accessors.IndexLens)((:, 2))), x)), var"##dist#286202")
    (var"##value#286203", __varinfo__) = (DynamicPPL.tilde_assume!!)(__context__, (DynamicPPL.unwrap_right_vn)((DynamicPPL.check_tilde_rhs)(var"##dist#286202"), var"##vn#286199")..., __varinfo__)
    x = (Accessors.set)(x, (BangBang.AccessorsImpl.prefermutation)((Accessors.opticcompose)((Accessors.IndexLens)((:, 2)))), var"##value#286203")
    var"##value#286203"
    var"##retval#286204" = nothing
    return (var"##retval#286204", __varinfo__)
end

function satellite_model_matrix(x0_mean, P0, var"##arg#286193"::(DynamicPPL.TypeWrap){TV}=(DynamicPPL.TypeWrap){Matrix{Float64}}();) where {TV}
    return (DynamicPPL.Model)(satellite_model_matrix, NamedTuple{(:x0_mean, :P0, Symbol("##arg#286193"))}((x0_mean, P0, var"##arg#286193"));)
end

x0_mean = [0.0, 0.0]
P0 = [0.1 0.0; 0.0 0.1]
model = satellite_model_matrix(x0_mean, P0)

f = LogDensityFunction(model)
x = f.varinfo[:]
Enzyme.autodiff(ReverseWithPrimal, LogDensityProblems.logdensity, Active, Const(f), Enzyme.Duplicated(x, zero(x)))

end

About half of the time when I run this it passes fine, the other half it crashes with

[92394] signal (4.1): Illegal instruction: 4
in expression starting at /Users/mhauru/projects/Enzyme-mwes/satellite_illegal_instruction/mwe.jl:53
satellite_model_matrix at /Users/mhauru/projects/Enzyme-mwes/satellite_illegal_instruction/mwe.jl:21
_evaluate!! at /Users/mhauru/.julia/packages/DynamicPPL/ACaKr/src/model.jl:968 [inlined]
evaluate_threadunsafe!! at /Users/mhauru/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941 [inlined]
evaluate!! at /Users/mhauru/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894 [inlined]
logdensity at /Users/mhauru/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100 [inlined]
logdensity at /Users/mhauru/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0 [inlined]
diffejulia_logdensity_3065_inner_1wrap at /Users/mhauru/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0
Allocations: 103369755 (Pool: 103245526; Big: 124229); GC: 84
Trace/BPT trap: 5

I appreciate the MWE probably needs more minimisation, but if you have any pointers as to where the indeterminism might be coming from, that would be great.

Enzyme v0.12.22.
@wsmoses
Copy link
Member

wsmoses commented Jul 10, 2024

Unfortunately not, this feels like a GC error.

The first thing I'd do is check if it's resolved by the GC fix in Julia 1.10.5 JuliaLang/julia#54851

If not definitely needs more minimization

@mhauru
Copy link
Contributor Author

mhauru commented Jul 11, 2024

I compiled Julia with the backports from that PR, and am still seeing this issue. Any other ideas for where indeterminacy might come from?

@wsmoses
Copy link
Member

wsmoses commented Jul 11, 2024 via email

@wsmoses
Copy link
Member

wsmoses commented Jul 11, 2024 via email

@mhauru mhauru mentioned this issue Jul 12, 2024
Closed
@wsmoses wsmoses added gc garbage collection segfault labels Jul 14, 2024
@mhauru
Copy link
Contributor Author

mhauru commented Jul 15, 2024

MWE that only depends on StdLib:

module MWE

import Random
import Base: (==), length, size
import LinearAlgebra

using Enzyme

Random.seed!(23)

Enzyme.API.runtimeActivity!(true)

struct PDMat{T<:Real,S<:AbstractMatrix}
    mat::S
    chol::LinearAlgebra.Cholesky{T,S}

    PDMat{T,S}(m::AbstractMatrix{T},c::LinearAlgebra.Cholesky{T,S}) where {T,S} = new{T,S}(m,c)
end

function chol_lower(a::LinearAlgebra.Cholesky)
    return LinearAlgebra.LowerTriangular(a.factors')
end

function PDMat(mat::AbstractMatrix,chol::LinearAlgebra.Cholesky{T,S}) where {T,S}
    d = LinearAlgebra.checksquare(mat)
    if size(chol, 1) != d
        throw(DimensionMismatch("Dimensions of mat and chol are inconsistent."))
    end
    PDMat{T,S}(convert(S, mat), chol)
end

size(a::PDMat, i::Int) = size(a.mat, i)
LinearAlgebra.cholesky(a::PDMat) = a.chol

PDMat(mat::AbstractMatrix) = PDMat(mat, LinearAlgebra.cholesky(mat))

LinearAlgebra.det(a::PDMat) = det(a.chol)
LinearAlgebra.logdet(a::PDMat) = LinearAlgebra.logdet(a.chol)

function invquad(a::PDMat, x::AbstractVecOrMat)
    inv_aL_x = chol_lower(LinearAlgebra.cholesky(a)) \ x
    return sum(abs2, inv_aL_x)
end

function mvnormal_c0(d)
    ldcd = logdetcov(d)
    return -(length(d) * oftype(ldcd, π) + ldcd) / 2
end

_logpdf(d, x::AbstractVector) = mvnormal_c0(d) - sqmahal(d, x) / 2

struct MvNormal{T<:Real,Cov,Mean<:AbstractVector}
    μ::Mean
    Σ::Cov
end

function MvNormal(μ::AbstractVector{T}, Σ::PDMat) where {T<:Real}
    size(Σ, 1) == length(μ) || throw(DimensionMismatch("The dimensions of mu and Sigma are inconsistent."))
    MvNormal{T,typeof(Σ),typeof(μ)}(μ, Σ)
end

MvNormal(μ::AbstractVector{<:Real}, Σ::AbstractMatrix{<:Real}) = MvNormal(μ, PDMat(Σ))

@inline function logpdf(
    d, x::AbstractArray{<:Real,M}
) where {M}
    return _logpdf(d, x)
end

length(d::MvNormal) = length(d.μ)
logdetcov(d::MvNormal) = LinearAlgebra.logdet(d.Σ)

sqmahal(d::MvNormal, x::AbstractVector) = invquad(d.Σ, x .- d.μ)

struct TypeWrap{T} end

struct Metadata{TVal}
    vals::TVal
end

struct VarInfo{Tmeta,Tlogp}
    metadata::Tmeta
    logp::Base.RefValue{Tlogp}
end

function getindex(vi::VarInfo)
    range = (1:2)
    return copy(vi.metadata.vals[range])
end

function tilde_assume!!(right, vi)
    y = [1.0, 1.0]
    _ = logpdf(right, y)
    r = getindex(vi)
    logp = 0.0
    vi.logp[] += logp
    return r, vi
end

function satellite_model_matrix(__varinfo__, x0_mean::Any, P0::Any)
    T = 2
    state_dim = 2
    x = Matrix{Float64}(undef, state_dim, T)
    d1 = MvNormal(x0_mean, P0)
    val1, __varinfo__ = tilde_assume!!(d1, __varinfo__)
    x[:, 1] = val1
    return __varinfo__
end

x0_mean = [0.0, 0.0]
P0 = [0.1 0.0; 0.0 0.1]

function f(x)
    vi_new = VarInfo(Metadata(x), Ref{Float64}(0.0))
    vi_result = satellite_model_matrix(vi_new, x0_mean, P0)
    return vi_result.logp[]
end
x = [0.0, 0.0, 0.0, 0.0]
@show f(x)
Enzyme.autodiff(ReverseWithPrimal, f, Active, Enzyme.Duplicated(x, zero(x)))

end

Output:

[4385] signal (4): Illegal instruction: 4
in expression starting at /Users/mhauru/projects/Enzyme-mwes/satellite_illegal_instruction/mwe.jl:121
diffejulia_satellite_model_matrix_3020_inner_1wrap at ./abstractarray.jl:702
macro expansion at /Users/mhauru/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6630 [inlined]
enzyme_call at /Users/mhauru/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6231
Allocations: 53352017 (Pool: 53285780; Big: 66237); GC: 51
Illegal instruction: 4

I get that something like 1/20 times, the rest of the times everything passes without issue.

Tested on Enzyme v0.12.23, Enzyme_jll v0.0.134+0.

This is quite finicky and minimisation is hard, are you good to take this from here @wsmoses?

@yebai yebai mentioned this issue Jul 24, 2024
Closed
@wsmoses
Copy link
Member

wsmoses commented Jul 15, 2024

Presently trying to see if can repro in a debug julia with rr.

@wsmoses
Copy link
Member

wsmoses commented Jul 16, 2024

@mhauru to confirm does this fail still with the backports PR.

I'm having difficulty reproducing. Was there anything that reduced the probability that can be reverted [even if sadly making less simple]

wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-55'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-56'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-57'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-58'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-59'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-60'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-61'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-62'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-63'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-64'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-65'.
f(x) = 0.0
wmoses@beast:~/git/Enzyme.jl ((HEAD detached at origin/main)) $ ~/git/rr/bin/rr record ./julia10/julia --project part.jl 
rr: Saving execution to trace directory `/home/wmoses/.local/share/rr/julia-66'.

@mhauru
Copy link
Contributor Author

mhauru commented Jul 16, 2024

Yeah, I'm running a Julia from commit 48e140b, so all except the very last commit from the backports PR. It's pretty bad with the randomness. Can you confirm that you can at least get it to trigger by running it 100 times, maybe without rr?

mhauru@403-Q4WFPXDFN0:~/projects/Enzyme-mwes/satellite_illegal_instruction (main)$ for i in {1..20}; do echo ${i}; ~/projects/julia/julia --project mwe.jl; done
1
f(x) = 0.0
2
f(x) = 0.0
3
f(x) = 0.0
4
f(x) = 0.0
5
f(x) = 0.0
6
f(x) = 0.0
7
f(x) = 0.0
8
f(x) = 0.0
9
f(x) = 0.0
10
f(x) = 0.0
11
f(x) = 0.0
12
f(x) = 0.0
13
f(x) = 0.0
14
f(x) = 0.0
15
f(x) = 0.0
16
f(x) = 0.0
17
f(x) = 0.0
18
f(x) = 0.0
19
f(x) = 0.0
20
f(x) = 0.0
mhauru@403-Q4WFPXDFN0:~/projects/Enzyme-mwes/satellite_illegal_instruction (main)$ for i in {1..20}; do echo ${i}; ~/projects/julia/julia --project mwe.jl; done
1
f(x) = 0.0
2
f(x) = 0.0
3
f(x) = 0.0
4
f(x) = 0.0
5
f(x) = 0.0
6
f(x) = 0.0
7
f(x) = 0.0
8
f(x) = 0.0

[2245] signal (4): Illegal instruction: 4
in expression starting at /Users/mhauru/projects/Enzyme-mwes/satellite_illegal_instruction/mwe.jl:121
diffejulia_satellite_model_matrix_3020_inner_1wrap at ./abstractarray.jl:702
macro expansion at /Users/mhauru/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6630 [inlined]
enzyme_call at /Users/mhauru/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6231
Allocations: 55025505 (Pool: 54956203; Big: 69302); GC: 55
Illegal instruction: 4
9
f(x) = 0.0
10
f(x) = 0.0
11
f(x) = 0.0
12
f(x) = 0.0
13
f(x) = 0.0
14
f(x) = 0.0
15
f(x) = 0.0

[2973] signal (4): Illegal instruction: 4
in expression starting at /Users/mhauru/projects/Enzyme-mwes/satellite_illegal_instruction/mwe.jl:121
diffejulia_satellite_model_matrix_3020_inner_1wrap at ./abstractarray.jl:702
macro expansion at /Users/mhauru/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6630 [inlined]
enzyme_call at /Users/mhauru/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6231
Allocations: 55629525 (Pool: 55559305; Big: 70220); GC: 60
Illegal instruction: 4
16
f(x) = 0.0
17
f(x) = 0.0
18
f(x) = 0.0
19
f(x) = 0.0
20
f(x) = 0.0

I can probably find something in the git history that fails more often but relies on extra dependencies, give me a moment.

@mhauru
Copy link
Contributor Author

mhauru commented Jul 16, 2024

This seems to error out closer to half the time, or maybe a third:

module MWE

import DynamicPPL
import AbstractPPL
import Accessors
import BangBang
import LogDensityProblems
import LogDensityProblemsAD
import ADTypes
import Random
using Turing: MvNormal, LogDensityFunction

using Enzyme

Random.seed!(23)

Enzyme.API.runtimeActivity!(true)

function satellite_model_matrix(__model__::DynamicPPL.Model, __varinfo__::DynamicPPL.AbstractVarInfo, __context__::AbstractPPL.AbstractContext, x0_mean::Any, P0::Any, var"##arg#286193"::(DynamicPPL.TypeWrap){TV}=(DynamicPPL.TypeWrap){Matrix{Float64}}();) where {TV}
    T = 2
    state_dim = 2
    x = TV(undef, state_dim, T)
    var"##dist#286197" = MvNormal(x0_mean, P0)
    var"##vn#286194" = (DynamicPPL.resolve_varnames)((AbstractPPL.VarName){:x}((AbstractPPL.concretize)((Accessors.opticcompose)((Accessors.IndexLens)((:, 1))), x)), var"##dist#286197")
    (var"##value#286198", __varinfo__) = (DynamicPPL.tilde_assume!!)(__context__, (DynamicPPL.unwrap_right_vn)((DynamicPPL.check_tilde_rhs)(var"##dist#286197"), var"##vn#286194")..., __varinfo__)
    x = (Accessors.set)(x, (BangBang.AccessorsImpl.prefermutation)((Accessors.opticcompose)((Accessors.IndexLens)((:, 1)))), var"##value#286198")
    var"##value#286198"
    var"##dist#286202" = MvNormal(x[:, 1], P0)
    var"##vn#286199" = (DynamicPPL.resolve_varnames)((AbstractPPL.VarName){:x}((AbstractPPL.concretize)((Accessors.opticcompose)((Accessors.IndexLens)((:, 2))), x)), var"##dist#286202")
    (var"##value#286203", __varinfo__) = (DynamicPPL.tilde_assume!!)(__context__, (DynamicPPL.unwrap_right_vn)((DynamicPPL.check_tilde_rhs)(var"##dist#286202"), var"##vn#286199")..., __varinfo__)
    x = (Accessors.set)(x, (BangBang.AccessorsImpl.prefermutation)((Accessors.opticcompose)((Accessors.IndexLens)((:, 2)))), var"##value#286203")
    var"##value#286203"
    var"##retval#286204" = nothing
    return (var"##retval#286204", __varinfo__)
end

function satellite_model_matrix(x0_mean, P0, var"##arg#286193"::(DynamicPPL.TypeWrap){TV}=(DynamicPPL.TypeWrap){Matrix{Float64}}();) where {TV}
    return (DynamicPPL.Model)(satellite_model_matrix, NamedTuple{(:x0_mean, :P0, Symbol("##arg#286193"))}((x0_mean, P0, var"##arg#286193"));)
end

x0_mean = [0.0, 0.0]
P0 = [0.1 0.0; 0.0 0.1]
model = satellite_model_matrix(x0_mean, P0)

f = LogDensityFunction(model)
x = f.varinfo[:]
@show LogDensityProblems.logdensity(f, x)
Enzyme.autodiff(ReverseWithPrimal, LogDensityProblems.logdensity, Active, Const(f), Enzyme.Duplicated(x, zero(x)))

end

@yebai yebai mentioned this issue Jul 18, 2024
Open
@wsmoses
Copy link
Member

wsmoses commented Jul 21, 2024

@mhauru with the fix to the other memory issue [and apparent issues I have reproducing this], can you confirm this still fails?

@wsmoses
Copy link
Member

wsmoses commented Jul 31, 2024 


  %18 = bitcast {} addrspace(10)* %2 to {} addrspace(10)* addrspace(10)*, !dbg !2850
  %19 = addrspacecast {} addrspace(10)* addrspace(10)* %18 to {} addrspace(10)* addrspace(11)*, !dbg !2850
  
  %arraysize_ptr = getelementptr inbounds {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %19, i64 3, !dbg !2850
  %20 = bitcast {} addrspace(10)* addrspace(11)* %arraysize_ptr to i64 addrspace(11)*, !dbg !2850
  %arraysize = load i64, i64 addrspace(11)* %20, align 8, !dbg !2850, !tbaa !81, !range !479, !alias.scope !2853, !noalias !2856


  %.repack5 = getelementptr inbounds { { i8*, double*, i8*, i64, i64 }, { {} addrspace(10)*, i64 }, { {} addrspace(10)*, i64 } }, { { i8*, double*, i8*, i64, i64 }, { {} addrspace(10)*, i64 }, { {} addrspace(10)*, i64 } }* %0, i64 0, i32 0, i32 3, !dbg !2908
  store i64 %arraysize, i64* %.repack5, align 8, !dbg !2908


  call void @augmented_julia_potrf__3629({ { i8*, double*, i8*, i64, i64 }, { {} addrspace(10)*, i64 }, { {} addrspace(10)*, i64 } }* nonnull sret({ { i8*, double*, i8*, i64, i64 }, { {} addrspace(10)*, i64 }, { {} addrspace(10)*, i64 } }) %stack_sret, [2 x {} addrspace(10)*]* nonnull "enzymejl_returnRoots" %6, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %35, {} addrspace(10)* noundef align 16 dereferenceable_or_null(16) %27), !dbg !2980


  %_augmented10.fca.3.gep = getelementptr inbounds { { i8*, double*, i8*, i64, i64 }, { {} addrspace(10)*, i64 }, { {} addrspace(10)*, i64 } }, { { i8*, double*, i8*, i64, i64 }, { {} addrspace(10)*, i64 }, { {} addrspace(10)*, i64 } }* %stack_sret, i64 0, i32 0, i32 3, !dbg !2980
  %_augmented10.fca.3.load = load i64, i64* %_augmented10.fca.3.gep, align 8, !dbg !2980


  %.repack22.repack36 = getelementptr inbounds { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }* %0, i64 0, i32 0, i32 5, i32 3, !dbg !2951
  store i64 %_augmented10.fca.3.load, i64* %.repack22.repack36, align 8, !dbg !2951


  %.repack22.repack38 = getelementptr inbounds { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }* %0, i64 0, i32 0, i32 5, i32 4, !dbg !2951
  store i64 %_augmented10.fca.4.load, i64* %.repack22.repack38, align 8, !dbg !2951




  call void @augmented_julia__cholesky_150_3622({ { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }* nonnull sret({ { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }) %stack_sret, [5 x {} addrspace(10)*]* nonnull "enzymejl_returnRoots" %14, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %3), !dbg !3043



  %_augmented.fca.5.3.gep = getelementptr inbounds { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 } }* %stack_sret, i64 0, i32 0, i32 5, i32 3, !dbg !3043
  %_augmented.fca.5.3.load = load i64, i64* %_augmented.fca.5.3.gep, align 8, !dbg !3043


  %.repack32.repack59.repack73 = getelementptr inbounds { { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }, { { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }* %0, i64 0, i32 0, i32 0, i32 5, i32 3, !dbg !3217
  store i64 %_augmented.fca.5.3.load, i64* %.repack32.repack59.repack73, align 8, !dbg !3217




  call void @augmented_julia_ressatellite_model_matrix_3508({ { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }* nonnull sret({ { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }) %stack_sret, [50 x {} addrspace(10)*]* nonnull "enzymejl_returnRoots" %25, { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } addrspace(11)* nocapture noundef nonnull readonly align 8 dereferenceable(80) %3, { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } addrspace(11)* nocapture noundef nonnull align 8 dereferenceable(80) %"'1", {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %getfield9), !dbg !3347


  %_augmented15.fca.0.5.3.gep = getelementptr inbounds { { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }, { { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }* %stack_sret, i64 0, i32 0, i32 0, i32 5, i32 3, !dbg !3347
  %_augmented15.fca.0.5.3.load = load i64, i64* %_augmented15.fca.0.5.3.gep, align 8, !dbg !3347


  %.fca.13.0.5.3.insert = insertvalue { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* } %.fca.13.0.5.2.insert, i64 %_augmented15.fca.0.5.3.load, 13, 0, 5, 3, !dbg !3347


  %.fca.16.insert = insertvalue { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* } %.fca.15.insert, {} addrspace(10)* %getfield9, 16, !dbg !3347




  %253 = getelementptr inbounds { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }, { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }* %0, i64 0, i32 0, !dbg !3347

  store { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* } %.fca.16.insert, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }* %253, align 8, !dbg !3347

  



  call void @augmented_julia_myevaluate___3502({ { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }* nonnull sret({ { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }) %stack_sret, [64 x {} addrspace(10)*]* nonnull "enzymejl_returnRoots" %25, { [1 x {} addrspace(10)*] } addrspace(11)* nocapture nofree noundef nonnull readonly align 8 dereferenceable(8) %35, { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } addrspace(11)* nocapture noundef nonnull readonly align 8 dereferenceable(80) %45, { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } addrspace(11)* nocapture noundef nonnull align 8 dereferenceable(80) %"'ipc1.i"), !dbg !1628


    %_augmented.i.fca.13.0.5.3.gep = getelementptr inbounds { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }, { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } }, { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } }* %stack_sret, i64 0, i32 0, i32 13, i32 0, i32 5, i32 3, !dbg !1628
  %_augmented.i.fca.13.0.5.3.load = load i64, i64* %_augmented.i.fca.13.0.5.3.gep, align 8, !dbg !1628


  %_augmented.i.fca.13.0.5.3.insert = insertvalue { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* } %_augmented.i.fca.13.0.5.2.insert, i64 %_augmented.i.fca.13.0.5.3.load, 13, 0, 5, 3, !dbg !1628





%_augmented.i.fca.16.insert



diffejulia_myevaluate___3502


  %tapeArg25 = extractvalue { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i1, i1*, i1* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, i64, i64 }, i64, i64, i64 }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, i8* }, { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 }, {} addrspace(10)*, i64, i64 }, i64, i64 }, i1, {} addrspace(10)*, {} addrspace(10)* } %tapeArg, 13, !dbg !2271



tapeArg25


diffejulia_ressatellite_model_matrix_3508


  %tapeArg12 = extractvalue {
    0:{
        { i8*, i8* },
        {} addrspace(10)*,
        {} addrspace(10)*,
        {} addrspace(10)*,
        { i64, i1, i1** },
       5: {
            i8*,
            double*,
            i8*,
          3:  i64,
            i64
        },
        i64,
        i64,
        i64
    },
    {} addrspace(10)*,
    {} addrspace(10)*,
    {} addrspace(10)*,
    {} addrspace(10)*,
    i8*,
    {} addrspace(10)*,
    {
        {} addrspace(10)*,
        {} addrspace(10)*,
        {} addrspace(10)*,
        { i8*, i8* },
        { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i8*, { double*, double* }, i8*, i8*, i64, i64, i64 }, { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, i1, {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)*, i64, i64, i1, i1 },
        {} addrspace(10)*,
        i64,
        i64
    },
    i64,
    i64
  } %tapeArg, 0, !dbg !5354


tapeArg12


diffejulia__cholesky_150_3622
  %tapeArg49 = extractvalue { { i8*, i8* }, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, { i64, i1, i1** }, { i8*, double*, i8*, ######i64, i64 }, i64, i64, i64 } %tapeArg, 5, !dbg !2320


diffejulia_potrf__3629
  %arraysize = extractvalue { i8*, double*, i8*, ######i64, i64 } %tapeArg, 3, !dbg !2376

  %11 = mul i64 %arraysize, %arraysize, !dbg !2394
  %12 = shl nuw i64 %11, 3, !dbg !2394.  <- THIS IS BOGUS

@wsmoses
Copy link
Member

wsmoses commented Jul 31, 2024

https://godbolt.org/z/aGna3GW85

@wsmoses
Copy link
Member

wsmoses commented Jul 31, 2024

https://fwd.gymni.ch/7xDmi9

In the reverse pass, we have an atomic load, from a previous stack to heap'd value. We decide to lower it back to a stack, believing we can recompute it. This is wrong as we can't -- and have an undefined load as a result.

in diffejulia_ressatellite_model_matrix_3508

  %3 = alloca { {} addrspace(10)*, i32, i64 }, i64 1, align 8
  %4 = bitcast { {} addrspace(10)*, i32, i64 }* %3 to i8*
  %9 = bitcast i8* %4 to { {} addrspace(10)*, i32, i64 }*
  %getfield_addr = getelementptr inbounds { {} addrspace(10)*, i32, i64 }, { {} addrspace(10)*, i32, i64 }* %9, i64 0, i32 0, !dbg !3301
  %getfield = load atomic {} addrspace(10)*, {} addrspace(10)** %getfield_addr unordered, align 8, !dbg !3301

@wsmoses
Copy link
Member

wsmoses commented Jul 31, 2024

https://fwd.gymni.ch/mtUJBR

@wsmoses
Copy link
Member

wsmoses commented Jul 31, 2024

Nondeterminstic in enzyme repro: https://fwd.gymni.ch/kWvrgB

@wsmoses
Copy link
Member

wsmoses commented Aug 1, 2024

https://fwd.gymni.ch/1Xbe91

@wsmoses
Copy link
Member

wsmoses commented Aug 1, 2024

Resolved but due to the bug in the julia compiler itself on this code, requires use of Julia 1.10.5 or waiting til backported.

x/ref JuliaLang/julia#55306 and JuliaLang/julia#55305

@wsmoses wsmoses closed this as completed Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
gc garbage collection more information needed segfault
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wsmoses @mhauru