feat(deid): drop valueString instead of passing it through #364
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, we would let Observation.valueString (and its component form) through, optionally masking it with philter if you passed --philter in. But it held PHI too often, and if we ever will use the field, it will likely be via ETL-side NLP. We don't need to keep the string intact into Athena. So now, we unconditionally drop Observation.valueString, replacing it with a data-absent-reason extension marker (so consumers know it _was_ present).
dogversioning
approved these changes
Nov 19, 2024
| key == "valueString", | ||
| ) | ||
| ): | ||
| raise MaskValue |
There was a problem hiding this comment.
i'm sure this is correct, but just for my own edification, where is this caught?
There was a problem hiding this comment.
Fair question 😄
The loop of the scrubber is that it walks the JSON tree, and runs all these little scrubbing handlers that if they recognize something they should scrub, modify the value or raise an exception like this.
So the answer is: one layer up in the Scrubber code. See
cumulus-etl/cumulus_etl/deid/scrubber.py
Line 167 in 741fb34
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, we would let Observation.valueString (and its component form) through, optionally masking it with philter if you passed --philter in.
But it held PHI too often, and if we ever will use the field, it will likely be via ETL-side NLP. We don't need to keep the string intact into Athena.
So now, we unconditionally drop Observation.valueString, replacing it with a data-absent-reason extension marker (so consumers know it was present).
Checklist
docs/) needs to be updated