NA in levels vs NA in values #337

hadley · 2023-01-05T14:41:02Z

This is an attempt to take a comprehensive look at handling NAs in levels vs NAs in values:

A new terminology that leads to fct_na_levels_to_values() and fct_na_values_to_levels() (and deprecating fct_explicit_na())
A brief introduction to the issue in the forcats vignette (Fixes fct_infreq() doesn't reorder NA levels #326)
Tests and some small changes to ensure that you can use other_level = NA everywhere. (Fixes fct_collapse() should allow other_level = NA #291)

Fixes #291

#Conflicts: # NEWS.md

mine-cetinkaya-rundel

I like making it more explicit (ha!) what "explicit" means so moving from value->level / level->value language from "explicit NA" sounds good to me.

I left a few typo fixes as well as some suggestions for how to present this idea in the vignette.

R/na.R

vignettes/forcats.Rmd

mine-cetinkaya-rundel · 2023-01-05T16:18:12Z

vignettes/forcats.Rmd

+
+There are two ways to represent a missing value in a factor:
+
+-   You can include it in the values of the factor; it does not appear in the levels and `is.na()` reports it as missing. This is the usual way to encode a missing value in a factor:


It's not just the "usual" but also the "default", so maybe "This is how missing values are encoded in a factor by default"?

mine-cetinkaya-rundel · 2023-01-05T16:25:41Z

vignettes/forcats.Rmd

+-   You can include in the levels of the factor and `is.na()` does not report it as missing. This requires a little more work to create:
+
+```{r}
+f <- factor(c("x", "y", NA), exclude = NULL)


Is there a way to say "don't exclude NAs" without specifying exclude = NULL? If not, I think it would be good to add a sentence about why we brought NULL into play all of a sudden, like something about "this works because in R NULLs are ..."

vignettes/forcats.Rmd

mine-cetinkaya-rundel · 2023-01-05T16:27:14Z

vignettes/forcats.Rmd

+`NA`s in the values tend to be best for data analysis, since `is.na()` works as you'd expect.
+`NA`s in the levels are often better for display since you can control where they appear.


I would move these two sentences to the end of this section since they're like a summary.

Also, I wonder about saying something like

You should be careful about using fct_na_value_to_level() in a data transformation pipeline where you overwrite your data since that will affect all downstream analysis as well. This should only be done if you will be using that updated data frame or creating displays or summaries, chances are you don't want NAs as an explicit level in your data analysis. If you do, you might consider relabeling them as something like "Unknown" or "Not available".

Maybe this is an overreach, but I feel like this function should be used as a quick fix, but if you really care about those NAs it's more practical to not make them NAs but something else, so they don't accidentally get swooped up by another function that treats NAs in a special way, say in a modelling context or something...

If you do, you might consider relabeling them as something like "Unknown" or "Not available".

Which you'd do with fct_na_value_to_level(x, name = "Unknown") if we add the idea of fct_explicit_na(na_level =) back in

NAMESPACE

NEWS.md

R/count.R

DavisVaughan · 2023-01-05T18:51:07Z

R/collapse.R

  }

-  names(levs) <- names(new)[rep(seq_along(new), vapply(new, length, integer(1)))]
-  out <- fct_recode(.f, !!!levs)


Since fct_recode() disappeared does that mean this weirdness went away? #291 (comment)

I was expecting to see a test for that or something

I added #339 to think about this more. I can't figure out if it deserves a test or not. On the one hand, it was arguably a bug, so it should get a test. On the other hand, thinking about fct_collapse() from first principles, it seems strange to test specific values of other_level.

R/na.R

DavisVaughan · 2023-01-05T18:57:45Z

R/na.R

+  f <- check_factor(f)
+
+  f <- fct_expand(f, NA)
+  f[is.na(f)] <- NA


I can't figure out what this line is for. Do you have an example of why this is needed?

This is like f[is.na(f)] <- "x" where the RHS is matched into the levels (which now contain NA) because of the previous line.

I'm probably missing something but to me it looked like is.na(f) would always return FALSE because fct_expand() turns NA values into integer codes

library(forcats) x <- factor(c("a", NA)) unclass(x) #> [1] 1 NA #> attr(,"levels") #> [1] "a" # No NAs here anymore! x <- fct_expand(x, NA) unclass(x) #> [1] 1 2 #> attr(,"levels") #> [1] "a" NA

^{Created on 2023-01-05 with reprex v2.0.2.9000}

Oooooh, I didn't notice that. Yes, you're right that it's unnecessary.

DavisVaughan · 2023-01-05T19:04:52Z

vignettes/forcats.Rmd

+`NA`s in the values tend to be best for data analysis, since `is.na()` works as you'd expect.
+`NA`s in the levels are often better for display since you can control where they appear.


If you do, you might consider relabeling them as something like "Unknown" or "Not available".

Which you'd do with fct_na_value_to_level(x, name = "Unknown") if we add the idea of fct_explicit_na(na_level =) back in

DavisVaughan · 2023-01-05T19:07:00Z

vignettes/forcats.Rmd

+```{r}
+#| fig.alt: >
+#|   The bar chart of hair color, now ordered so that NAs, now labelled
+#|   "(missing)" are ordered where you'd expect: in between white (4) and


Narrator: "They were not, in fact, labelled "(missing)""

Co-authored-by: Davis Vaughan <davis@rstudio.com> Co-authored-by: Mine Cetinkaya-Rundel <cetinkaya.mine@gmail.com>

hadley · 2023-01-09T14:19:22Z

R/other.R

-    return(f)
+# Replace specified levels (if any), with other.
+# @param keep A logical vector the same length as `levels(f)`
+lvls_other <- function(f, keep, other_level = "Other") {


I also added this new helper function to ensure that the implementation of "othering" is identical for all code paths.

hadley · 2023-01-09T14:43:08Z

R/na.R

+#' @rdname fct_na_value_to_level
+#' @param extra_levels Optionally, a character vector giving additional levels
+#'   that should also be converted to `NA` values.
+fct_na_level_to_value <- function(f, extra_levels = NULL) {


I made this extra_levels because if you don't include NA, it's easy to create factors that have a mix of NAs in values and levels.

Makes sense. I was actually wondering if we even needed extra_levels at all, but there doesn't seem to be any other way to do this in forcats, so it seems useful to have

x <- factor(c("x", "missing")) forcats::fct_na_level_to_value(x, extra_levels = "missing") #> [1] x <NA> #> Levels: x

It is a nice way of declaring, "no, missing should really be a missing value"

I'm actually surprised this has never come up before. It seems pretty useful

DavisVaughan · 2023-01-09T14:59:58Z

R/na.R

+  if (!identical(level, NA) && !is_string(level)) {
+    cli::cli_abort(
+      "{.arg level} must be a string or {.code NA}, not {.obj_type_friendly level}."
+    )
+  }


check_string(level, allow_na = TRUE) allows for logical NA

Oh I guess you aren't using the checkers yet

DavisVaughan · 2023-01-09T15:01:37Z

R/na.R

+fct_na_value_to_level <- function(f, level = NA) {
+  if (!identical(level, NA) && !is_string(level)) {
+    cli::cli_abort(
+      "{.arg level} must be a string or {.code NA}, not {.obj_type_friendly level}."


I think you need {.obj_type_friendly {level}}

It isn't letting me do a "suggestion" for some reason

DavisVaughan · 2023-01-09T15:03:53Z

R/na.R

+  f <- fct_expand(f, NA)
+  new_levels <- levels(f)
+  new_levels[is.na(new_levels)] <- level
+
+  lvls_revalue(f, new_levels)


So if you already have an NA level, this will rename it to level? Seems reasonable

DavisVaughan · 2023-01-09T15:09:57Z

R/na.R

+#' @rdname fct_na_value_to_level
+#' @param extra_levels Optionally, a character vector giving additional levels
+#'   that should also be converted to `NA` values.
+fct_na_level_to_value <- function(f, extra_levels = NULL) {


Makes sense. I was actually wondering if we even needed extra_levels at all, but there doesn't seem to be any other way to do this in forcats, so it seems useful to have

x <- factor(c("x", "missing")) forcats::fct_na_level_to_value(x, extra_levels = "missing") #> [1] x <NA> #> Levels: x

It is a nice way of declaring, "no, missing should really be a missing value"

R/na.R

hadley added 6 commits January 4, 2023 18:00

Test other_level = NA in lump/collapse

5f2027b

Refactor fct_collapse() to allow other_level = NA

7db2914

Fixes #291

Add fct_na_value_to_level/fct_na_level_to_value

8456e72

Update fct_count() to work with NAs in levels

e62b172

Deprecate fct_explicit_na()

2d2abfe

withr only used in tests

3318455

hadley requested review from mine-cetinkaya-rundel and DavisVaughan January 5, 2023 14:41

hadley added 2 commits January 5, 2023 08:41

Merge commit '42fd258fe18d057ed814dc90a38a0bcb597d138f'

90705eb

#Conflicts: # NEWS.md

Add bullet for fct_count()

b52d2bb

mine-cetinkaya-rundel suggested changes Jan 5, 2023

View reviewed changes

DavisVaughan approved these changes Jan 5, 2023

View reviewed changes

hadley and others added 5 commits January 9, 2023 08:10

Apply suggestions from code review

907b6c9

Co-authored-by: Davis Vaughan <davis@rstudio.com> Co-authored-by: Mine Cetinkaya-Rundel <cetinkaya.mine@gmail.com>

Re-document

18d9611

Extract out lvls_other() for consistent handling

d316919

Remove unneeded code

bb8114d

Vignette tweaks based on code review feedback

b1e3f12

hadley commented Jan 9, 2023

View reviewed changes

Customise levels

e65decc

hadley requested a review from DavisVaughan January 9, 2023 14:42

hadley commented Jan 9, 2023

View reviewed changes

DavisVaughan approved these changes Jan 9, 2023

View reviewed changes

Fix typo

1b0774f

hadley merged commit 201f30e into main Jan 9, 2023

hadley deleted the other-na branch January 9, 2023 20:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NA in levels vs NA in values #337

NA in levels vs NA in values #337

hadley commented Jan 5, 2023 •

edited

Loading

mine-cetinkaya-rundel left a comment

mine-cetinkaya-rundel Jan 5, 2023

mine-cetinkaya-rundel Jan 5, 2023

mine-cetinkaya-rundel Jan 5, 2023

DavisVaughan Jan 5, 2023

DavisVaughan Jan 5, 2023

hadley Jan 9, 2023

DavisVaughan Jan 5, 2023

hadley Jan 5, 2023

DavisVaughan Jan 5, 2023

hadley Jan 9, 2023

DavisVaughan Jan 5, 2023

DavisVaughan Jan 5, 2023

hadley Jan 9, 2023

hadley Jan 9, 2023

DavisVaughan Jan 9, 2023

DavisVaughan Jan 9, 2023

DavisVaughan Jan 9, 2023

DavisVaughan Jan 9, 2023

DavisVaughan Jan 9, 2023

hadley Jan 9, 2023

DavisVaughan Jan 9, 2023

DavisVaughan Jan 9, 2023


		There are two ways to represent a missing value in a factor:

		- You can include it in the values of the factor; it does not appear in the levels and `is.na()` reports it as missing. This is the usual way to encode a missing value in a factor:

		`NA`s in the values tend to be best for data analysis, since `is.na()` works as you'd expect.
		`NA`s in the levels are often better for display since you can control where they appear.

NA in levels vs NA in values #337

NA in levels vs NA in values #337

Conversation

hadley commented Jan 5, 2023 • edited Loading

mine-cetinkaya-rundel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hadley commented Jan 5, 2023 •

edited

Loading