Exception: In serializer: Storage capacity [763] exceeded while writing value of size [16] from position [759]

[weefuzzy]

Summary:

Running a model produces the error above and also says:

This is an internal error, if you see it please report it as an issue on the Stan github repository.

Description:

Running the model code quoted below, which is generated code produced via https://github.com/rmcelreath/rethinking gives the error above.

I'm just in the process of learning, so (a) may well have done something silly (b) can't provide good answers for why the code is the way it is. I'm just reporting here because the error message asked me to.

Reproducible Steps:

Raw Data: https://github.com/rmcelreath/rethinking/blob/master/data/bangladesh.csv

Data to model:

library(rethinking)
data(bangladesh)

d <- bangladesh

dat <- list (
  C = d$use.contraception,
  D = as.integer(d$district),
  U = ifelse(d$urban==1,1,0),
  K = d$living.children, 
  A = d$living.children, 
  alpha = c(2, 2, 2)
)

Code

data{
    array[1934] int C;
     vector[1934] A;
    array[1934] int K;
    array[1934] int U;
    array[1934] int D;
     vector[3] alpha;
}
parameters{
     matrix[4,61] Z;
     vector[4] abar;
     cholesky_factor_corr[4] L_Rho;
     vector<lower=0>[4] sigma;
     simplex[3] delta;
}
transformed parameters{
     vector[61] a;
     vector[61] bU;
     vector[61] bK;
     vector[61] bA;
     matrix[61,4] v;
    v = (diag_pre_multiply(sigma, L_Rho) * Z)';
    bA = abar[4] + v[, 4];
    bK = abar[3] + v[, 3];
    bU = abar[2] + v[, 2];
    a = abar[1] + v[, 1];
}
model{
     vector[1934] p;
     vector[4] delta_j;
    delta ~ dirichlet( alpha );
    delta_j = append_row(0, delta);
    sigma ~ exponential( 1 );
    L_Rho ~ lkj_corr_cholesky( 4 );
    abar ~ normal( 0 , 1 );
    to_vector( Z ) ~ normal( 0 , 1 );
    for ( i in 1:1934 ) {
        p[i] = a[D[i]] + bU[D[i]] * U[i] + bK[D[i]] * (sum(delta_j[1:K[i]])) + bA[D[i]] * A[i];
        p[i] = inv_logit(p[i]);
    }
    C ~ bernoulli( p );
}
generated quantities{
     matrix[2,2] Rho;
    Rho = multiply_lower_tri_self_transpose(L_Rho);
}

Current Output:

Each chain does:

Chain 1 Rejecting initial value:
Chain 1   Log probability evaluates to log(0), i.e. negative infinity.
Chain 1   Stan can't start sampling from this initial value.
Chain 1 Iteration:   1 / 1000 [  0%]  (Warmup) 
Chain 1 Exception: In serializer: Storage capacity [763] exceeded while writing value of size [16] from position [759]. This is an internal error, if you see it please report it as an issue on the Stan github repository. (in '/var/folders/3d/czqqm9h550g3ktn68hmtkh080000gn/T/RtmpabL6Uf/model-a7f34d9ee239.stan', line 45, column 4 to column 51)

Expected Output:

Additional Information:

The /var/... folder with the generated code also contains some chunky csv files (7.2MB each; one per chain?) that may contain more detailed output. Can u/l somewhere if that's helpful.

Current Version:

v2.34.0

[weefuzzy]

Yes, I'd done something silly: if I correct to matrix[4,4] Rho; (not [2,2]) the error goes away. Hopefully that helps narrow it down.

[nhuurre]

Declaring matrix[2,2] Rho; instead of [4,4] should result in

Exception: matrix assign columns: assigning variable Rho (2) and right hand side columns (4) must match in size

The error about serializer storage capacity means the size check during assignment is skipped for some reason and the mistake is not caught until copying the data to the output (which had reserved space for a 2-by-2 matrix in accordance with the declaration)

This may be a problem in an older version of Stan. Are you using the latest RStan?

Do you have optimizations enabled? With optimizations the following model throws the "capacity exceeded" error:

generated quantities {
  matrix[2,2] Rho = lkj_corr_rng(4, 1.0);
}

Optimizations are experimental but even so I don't think they should throw internal errors.

[weefuzzy]

Thanks!

Are you using the latest RStan?

> rstan::stan_version()
[1] "2.32.2"

The current version is 2.34? I'll try upgrading, after checking the effect of optimization.

Do you have optimizations enabled?

I think so – the invocation of stan is coming via the rethinking package's ulam function, which claims that it passes O1 by default. I'll try with explicit settings and see if it changes.

[weefuzzy]

• With O0 and Oexperimental I get Exception: matrix assign columns:...
• With O1 I get capacity exceeded

Correction to version given above. Both rstan and StanHeaders are 2.32.5, which seems to be the latest on CRAN

[weefuzzy]

A more minimal and self-contained reproduction. tl;dr in O1 only, I get the internal error. Other flags correctly hint at what I did wrong:

stanmodelcode <- "
parameters{
  cholesky_factor_corr[4] L_Rho;
}

model{
  L_Rho ~ lkj_corr_cholesky( 4 );
}

generated quantities{
    matrix[2,2] Rho;
    Rho = multiply_lower_tri_self_transpose(L_Rho);
}
"
print("Running with O0")
stan(model_code=stanmodelcode,data=list(), iter=2, chains=1,stanc_options=list("O0"))
print("Running with Oexperimental")
stan(model_code=stanmodelcode,data=list(), iter=2, chains=1,stanc_options=list("Oexperimental"))
print("Running with O1")
stan(model_code=stanmodelcode,data=list(), iter=2, chains=1,stanc_options=list("O1"))

[WardBrian]

Sounds like this is another example of stan-dev/stanc3#1295.

One of the optimizations we currently have enabled prevents initializing a variable in Stan if it is immediately assigned to. However, this breaks the way we do bounds checking on containers. Just one of the reasons we recommend running the model at O0 (even if only for a handful of iterations) before setting O1.

Thank you for reporting!