Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

collect_metrics() doesn't return .weight_time and .pred_censored #764

Closed
EmilHvitfeldt opened this issue Nov 21, 2023 · 2 comments
Closed
Labels
bug an unexpected problem or unintended behavior

Comments

@EmilHvitfeldt
Copy link
Member

I expected to get those columns back in the .pred column. Especially since they are used to calculate the metrics

library(tidymodels)
library(censored)
#> Loading required package: survival

set.seed(1)
sim_dat <- prodlim::SimSurv(500) %>%
  mutate(event_time = Surv(time, event)) %>%
  select(event_time, X1, X2) %>%
  as_tibble()

set.seed(2)
split <- initial_split(sim_dat)
sim_tr <- training(split)
sim_te <- testing(split)
sim_rs <- vfold_cv(sim_tr)

time_points <- c(10, 1, 5, 15)

mod_spec <-
  proportional_hazards(penalty = tune(), mixture = 1) %>%
  set_engine("glmnet") %>%
  set_mode("censored regression")

grid <- tibble(penalty = 10^c(-4, -2, -1))

gctrl <- control_grid(save_pred = TRUE)

mix_mtrc <- metric_set(brier_survival, brier_survival_integrated, concordance_survival)

set.seed(2193)
tune_res <-
  mod_spec %>%
  tune_grid(
    event_time ~ X1 + X2,
    resamples = sim_rs,
    grid = grid,
    metrics = mix_mtrc,
    eval_time = time_points,
    control = gctrl
  )

collect_predictions(tune_res) %>%
  slice(1) %>%
  pull(.pred)
#> [[1]]
#> # A tibble: 4 × 3
#>   .eval_time .pred_survival .weight_censored
#>        <dbl>          <dbl>            <dbl>
#> 1         10        0.227               1.12
#> 2          1        0.983               1.02
#> 3          5        0.618               1.12
#> 4         15        0.00746             1.12
@EmilHvitfeldt EmilHvitfeldt added the bug an unexpected problem or unintended behavior label Nov 21, 2023
@hfrick
Copy link
Member

hfrick commented Nov 28, 2023

Chatting with Max, this is still how we want it to be: .weight_time and .pred_censored are used to calculate the metrics but not metrics themselves thus not showing up here!

@hfrick hfrick closed this as completed Nov 28, 2023
Copy link

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

@github-actions github-actions bot locked and limited conversation to collaborators Dec 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug an unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

2 participants