Download `Gene_full_name` and `Protein_RefSeq_ID` from
https://mygene.info/ using the mygene package.

Remove rows that have both Gene_full_name and Protein_RefSeq_ID values
missing.

Write NA for missing values rather than "NA" or "", in order to be
consistent with the data release.

Print messages after done running shell scripts.

mygene API may return results in different orders, so sorting the values
before output is necessary to reproduce previous results.

Add long-format-table-utils module.

Describe how to update
annotator/annotation-data/oncokb-cancer-gene-list.tsv.

Update OncoKB annotation table source to
analyses/long-format-table-utils/annotator/annotation-data/oncokb-cancer-gene-list.tsv

Update README.md.

For an annotation table, NA or duplicate should not exist in the key
column that is used to join_by for adding annotations to input table.

Gene symbol is not added in this function, and gene symbol is required.

It is also not necessary to check the gene_symbol column.

The annotate_long_format_table function now only supports tibble,
because handling data.frame or other types of table is complex, such as
preserving rownames, orders, and column types.

Support for other types of table could be added if required at a later
point.

add_Gene_type is more informative.

In annotate_long_format_table(), add a add_OncoKB_columns parameter.

add_OncoKB_columns: TRUE or FALSE on whether to add OncoKB_cancer_gene
and OncoKB_oncogene_TSG columns. Default value is FALSE.

The interface of annotate_long_format_table is changed, so that all
annotation columns are added by default, and the columns_to_add
parameter can be passed to add only certain columns.

The current interface annotate_long_format_table is designed to favor
flexibility and readability over efficiency. The funciton can be further
refactored to be more efficient, e.g. not check certain tables, process
certain tables, or add certain columns if they are not required.

The order of added columns may not be the same as the values of
columns_to_add, in annotate_long_format_table.

In annotate_long_format_table, reorder the columns of the returned table
to have the same column order as the input table and added columns in
the order of columns_to_add parameter.

Renamed to ensg_rmtl_df.

Only left_join an annotation table if it is required.

Use %>% to chain multiple processing steps

Add R API usage of long-format table annotator.

Add non-missing value descriptions for each annotation column.

`source` annotator-api.R.

Parse arguments.

Add documentation for R CLI usage of long-format table annotator.

Revise annotator R API example usage, so the code does not rely on any
loaded package.

Remove data reading line in the CLI example.

Co-authored-by: Jo Lynne Rokita <jharenza@gmail.com>

Co-authored-by: Jo Lynne Rokita <jharenza@gmail.com>

Co-authored-by: Jo Lynne Rokita <jharenza@gmail.com>

Added a note on TSV format in the help message:

*NOTE** on the --input-long-format-table-tsv file: 1) the TSV file
should use double quotes for field values thatneed escape, e.g. "NA" for
string literal "NA" and "\t" for tab; 2) only unquoted NA field values
are treated as missing values internally; 3) leading and trailing white
spaces in field values are **NOT** trimmed before parsing.

Changed parameters in read_tsv in order to preserve the TSV content. See
comments for more details.

Add notes to CLI input TSV specifications that are implemented in the
last commit.

…ble-utils.sh

README.md is also updated accordingly.

This is suggested by @jharenza at
d3b-center#55 (comment)
, in order to follow the shell script name convention of analysis
modules.

Add annotation data versions and data of the last update in the "Update
downloaded data that are used in this module" section, as suggested by
@jharenza at
d3b-center#55 (comment)

Combine gene and disease (/cancer_group) annotations into one table.

Add additional notes on annotation data versions to the "Implementation
of long-format table annotator" section.

Change the date of the last update of
annotator/annotation-data/oncokb-cancer-gene-list.tsv to 07/16/2021.

The 07/16/2021 annotator/annotation-data/oncokb-cancer-gene-list.tsv is
identical to the previous 06/16/2021 version, even though the website at
https://www.oncokb.org/cancerGenes has changed last update from
06/16/2021 to 07/16/2021.

Merge changes in the data downloading PR
d3b-center#55 .

Rename update-long-format-table-utils.sh to
run-update-long-format-table-utils.sh .

Specify annotation data versions in README.md.

Change the date of the last update of
annotator/annotation-data/oncokb-cancer-gene-list.tsv to 07/16/2021.

Merge changes in the data downloading PR
d3b-center#55 .

Rename update-long-format-table-utils.sh to
run-update-long-format-table-utils.sh .

Specify annotation data versions in README.md.

Change the date of the last update of
annotator/annotation-data/oncokb-cancer-gene-list.tsv to 07/16/2021.

As suggested by @jharenza at
<d3b-center#55 (comment)>,
test cases should be removed from the source code file.

Run `bash run-tests.sh` to run all tests.

In order to import a funciton for testing from an R file without running
the whole file, a helper function import_function is defined at
tests/helper_import_function.R, and the import_function is also tested
in the tests/test_helper_import_function.R file.

Suggested by @NHJohnson at
d3b-center#55 (review)

Add "Unit testing for long-format table annotator" section to descript
how to use the unit testing framework.

Edit.

Merge changes from the data downloading PR
<d3b-center#55>

Merge changes from the data downloading PR
<d3b-center#55>

Fix a typo.

Fix a typo.

Clarify the reasons for requiring both Gene_Ensembl_ID and Gene_symbol
in annotator/annotator-api.R, as suggested by @jharenza at
d3b-center#56 (review)

Group notes on annotation data versions into a list.

Note that the annotation columns to be added should not already exist in
the table that needs to be annotated, as suggested by @NHJohnson at
d3b-center#56 (review)

Note that that the names of the annotation columns will be standardized
at a later point, as suggested by @jharenza at
<d3b-center#56 (review)>,
so it is recommended to use the annotation column names in this module
for the results.

Add rprojroot::has_file(".git") as an alternative root criterion to
handle linked git working trees created by `git worktree add` in
rprojroot::find_root.

Also fixed typo identified by @NHJohnson at
d3b-center#56 (comment)

git commands do not work in a linked working tree in a Docker
image/container, because the linked working tree uses host absolute
paths to locate the main working tree.

Therefore, drop the git diff --stat command to improve compatibility
with Docker image/container. The users could still use git diff --stat
on host machine.

Suggested by @NHJohnson at
d3b-center#56 (review)

As suggested by @NHJohnson at
<https://github.com/PediatricOpenTargets/OpenPedCan-analysis/pull/56/files#r674342194>,
reduce code duplication by refactoring the annotate_long_format_table
funtion in annotator-api.R with a functional paradigm. See comments in
annotator-api.R for more details.

The interface is not changed, so previous examples and descriptions
should all still apply.

Explain where import_function is from.

Duplicated .x and .y annotation columns will be added in the following
scenario:

- annotation table has multiple columns

- one or more annotatoin columns in the annotatoin table are in the
columns_to_add vector

- one or more other annotation columns in the annotation table are
already in the table to be annotated, but these columns are not in the
columns_to_add vector

Fix this bug by joining only annotation columns that are not in the
table to be annotated.

As found by @NHJohnson at
<d3b-center#56 (comment)>,
some tests failed out side of the Docker image.

See comments in annotator/tests/test_annotate_long_format_table.R for
more details about the package version compatibility issues.

Add rprojroot::has_file(".git") as an alternative root criterion to
handle linked git working trees created by `git worktree add` in
rprojroot::find_root.

Previously, `-c ''` calls will fail because `""` is passed to API calls
as required columns to add, which is unavailable.

Now, `-c ''` calls will pass `character(0)` to API calls, so the output
table will not have any additional annotation column.

Add a note on the naming conventions of test files in the tsetthat
package 2.1.1.

Remove input files after running CLI tests, if the input files are
created by tests as intermediate files, in order to ensure a clean start
for the next test to run.

Should fail on annotator CLI calls with unavailable input files or
output dirs.

Note that nested functions cannot be imported, and unnamed functions
cannot be imported.

Should fail on importing functions that are nestedly defined in other
expressions.

Should fail, even if the function is defined in different ways.

Comments and line breaks should not change the behaviors of
import_function.

Corrected test helpfer function context.

Added test/ prefix to test annotator/tests/test_annotator_cli.R.

Put the imported function in the same environment as the import_function
being called, so the imported functions can call other imported
functions. Also add a test for such use cases.

In import_function, envir = parent.frame() makes the
environment(imported_function) to be the same as the import function
being called.

Even though the eval documentation says the default envir parameter is
parent.frame(), leaving envir = parent.frame() in the eval call will
surprisingly make the environment(imported_function) to be the same as
the environment of the import_function call. The tests also fail without
specifying envir = parent.frame(). Maybe this is caused by the place
where parent.frame() is evaluated? If specified, parent.frame() is
evaluated in the calling function; if not specified, parent.frame() is
evaluated in the eval call environment?

Examples executed in terminal R, in order to avoid RStudio customizations.
> print(environment())
<environment: R_GlobalEnv>
>
> foo <- function() {
+   print(parent.frame())
+   print(environment())
+   return(eval(quote(function() { return(2) })))
+ }
>
> bar <- foo()
<environment: R_GlobalEnv>
<environment: 0x5645ead67670>
> print(environment(bar))
<environment: 0x5645ead67670>
>
> baz <- function() {
+   print(parent.frame())
+   print(environment())
+   return(eval(quote(function() { return(2) }),
+               envir = parent.frame()))
+ }
>
> qux <- baz()
<environment: R_GlobalEnv>
<environment: 0x5645ead5fec0>
> print(environment(qux))
<environment: R_GlobalEnv>

Previously, if replace_na_with_empty_string=TRUE in
annotate_long_format_table, replace_na with empty string is applied to
all columns, including columns that have no NA. If a non-character
column that has no NA, its data type will be comverted to character,
which may break backward compatibility of the JSON/JSONL tables that
were generated without using the annotator API.

Now, if replace_na_with_empty_string=TRUE in annotate_long_format_table,
only replace NA with empty string in columns that have NA in them.  This
will not change the value types of the columns that have no NA, so the
output JSON/JSONL table will be backward compatible with the ones that
were generated without using the annotator API.

CLI help message is also changed accordingly.

Test that replace_na_with_empty_string = TRUE replaces NA with "". Test
that non-character columns without NA are not converted to character
columns.

Test that replace_na_with_empty_string = FALSE does not replace NA with
"".

Test that --replace-na-with-empty-string replaces NA with "".

Test that no --replace-na-with-empty-string does not replace NA with "".

The --replace-na-with-empty-string does not need to be tested for column
type conversions. The CLI write_tsv output table is the same even if a
non-character column is converted to a character column before
write_tsv, because "Values are only quoted if they contain a comma,
quote or newline" (-- help("write_tsv", "readr") 1.3.1).