Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add CellAssign process #479

Merged
merged 12 commits into from
Oct 3, 2023
Merged

Add CellAssign process #479

Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f255997
initial code for cellassign, not tested and probably buggy
sjspielman Oct 2, 2023
ead459e
Merge branch 'development' into sjspielman/add-process-for-cellassign
sjspielman Oct 2, 2023
3d7c1c9
Lots of debugging and updates, including passing through a singler_re…
sjspielman Oct 2, 2023
474d8de
spacing and delete NO_NAME which isn't needed
sjspielman Oct 2, 2023
61b1b6f
Apply suggestions from code review
sjspielman Oct 3, 2023
6ee6043
Apply suggestions from code review
sjspielman Oct 3, 2023
4ada0c8
cellassign reference file variable name
sjspielman Oct 3, 2023
88039d7
missing a quote
sjspielman Oct 3, 2023
9de11fc
remove singler and cellassign reference names from meta since those s…
sjspielman Oct 3, 2023
09565d1
Add reference name from model file to DataFrame metadata, and add ass…
sjspielman Oct 3, 2023
d686b60
switch to stringr::str_match
sjspielman Oct 3, 2023
3c2715c
Apply suggestions from code review
sjspielman Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 74 additions & 20 deletions modules/classify-celltypes.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,33 +38,54 @@ process classify_singler {
}


process predict_cellassign {
process classify_cellassign {
container params.SCPCATOOLS_CONTAINER
publishDir "${params.checkpoints_dir}/celltype/${meta.library_id}", mode: 'copy'
publishDir (
path: "${meta.celltype_publish_dir}",
mode: 'copy',
pattern: "${cellassign_dir}"
)
label 'mem_32'
label 'cpus_12'
input:
tuple val(meta), path(processed_hdf5), path(cellassign_reference_mtx), val(ref_name)
tuple val(meta), path(processed_rds), path(cellassign_reference_file)
sjspielman marked this conversation as resolved.
Show resolved Hide resolved
output:
tuple val(meta), path(cellassign_predictions), val(ref_name)
tuple val(meta.library_id), path(cellassign_dir)
sjspielman marked this conversation as resolved.
Show resolved Hide resolved
script:
cellassign_predictions = "${meta.library_id}_predictions.tsv"
cellassign_dir = file(meta.cellassign_dir).name

processed_hdf5 = "${meta.library_id}_processed.hdf5"
cellassign_predictions_tsv = "${meta.library_id}_predictions.tsv"
sjspielman marked this conversation as resolved.
Show resolved Hide resolved
"""
# create output directory
mkdir "${cellassign_dir}"

# Convert SCE to AnnData
sce_to_anndata.R \
--input_sce_file ${processed_rds} \
--output_rna_h5 ${processed_hdf5}
sjspielman marked this conversation as resolved.
Show resolved Hide resolved

# Run CellAssign
predict_cellassign.py \
--input_hdf5_file ${processed_hdf5} \
sjspielman marked this conversation as resolved.
Show resolved Hide resolved
--output_predictions ${cellassign_predictions} \
--reference ${cellassign_reference_mtx} \
--output_predictions ${cellassign_predictions_tsv} \
sjspielman marked this conversation as resolved.
Show resolved Hide resolved
--reference ${cellassign_ref} \
--seed ${params.seed} \
--threads ${task.cpus}

# write out meta file
echo "${Utils.makeJson(meta)}" > "${cellassign_dir}/scpca-meta.json"
"""
stub:
cellassign_predictions = "${meta.library_id}_predictions.tsv"
cellassign_dir = file(meta.cellassign_dir).name
"""
touch "${cellassign_predictions}"
mkdir "${cellassign_dir}"
echo "${Utils.makeJson(meta)}" > "${cellassign_dir}/scpca-meta.json"
"""
}

process classify_cellassign {
// TODO: overhaul this process next
process add_celltypes_to_sce {
container params.SCPCATOOLS_CONTAINER
publishDir "${params.results_dir}/${meta.project_id}/${meta.sample_id}", mode: 'copy'
label 'mem_4'
Expand Down Expand Up @@ -100,31 +121,34 @@ workflow annotate_celltypes {
it.scpca_project_id,
// singler model file
Utils.parseNA(it.singler_ref_file) ? "${params.singler_models_dir}/${it.singler_ref_file}" : null,
// singler reference name
Utils.parseNA(it.singler_ref_name),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we didn't have this before is because it is encoded in the rds file of the SingleR model. Since the cellassign input is just a tsv, we don't have that info.

When we pass along the SingleR results, that also should have the reference name in the output .rds file. Since we have to read that in anyway to get the full results tables we want to add to the output, we will have that information and don't need to pass it separately through the workflow.

Suggested change
// singler reference name
Utils.parseNA(it.singler_ref_name),

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noting i'll circle back to this suggestion after final.final discussion in the PR!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we didn't have this before is because it is encoded in the rds file of the SingleR model. Since the cellassign input is just a tsv, we don't have that info.

I was a bit confused: the reference name was being parsed out of the file name:

scpca-nf/bin/classify_SingleR.R

Line 78 in 67e175b

model_names <- stringr::str_remove(basename(model_files), "_model.rds")

Looking back at #402, it seems like we added that variable to the table mostly for convenience, but we could have parsed it out of the file name. It seems like we are using a shorthand at the moment, but the full file name (less .tsv) should probably be included as the name (i.e. "PanglaoDB-blood" rather than "blood"). In nextflow this can be done with file("path/my_model.tsv").baseName which strips both the directory and the extension.

And fun addition, if we want to do the same string parsing for the singler model files, which end with "_model.rds", we could do: file("path/my_model.tsv").baseName - '_model' which is so odd, but it works. If you want to be safer, use a little regex instead: file("path/my_model.tsv").baseName - ~'_model$'. Yes, you can make a regex by just adding a ~ before a quote. Groovy is so strange, but kind of cool sometimes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noting i'll circle back to this suggestion after final.final discussion in the PR!

My current suggestion is to punt this to a separate PR: it isn't really part of getting the cellassign process done.

// cellassign reference file
Utils.parseNA(it.cellassign_ref_file) ? "${params.cellassign_ref_dir}/${it.cellassign_ref_file}" : null,
// add ref name for cellassign since we cannot store it in the cellassign output
// singler ref name does not need to be added because it is stored in the singler model
// cellassign reference name
Utils.parseNA(it.cellassign_ref_name)
]}

// create input for typing: [augmented meta, processed_sce]
celltype_input_ch = processed_sce_channel
.map{[it[0]["project_id"]] + it}
.combine(celltype_ch, by: 0)
// current contents: [project_id, meta, processed_sce, singler_model_file, cellassign_reference_file, cellassign_reference_name]
// current contents: [project_id, meta, processed_sce, singler_model_file, singler_reference_name, cellassign_reference_file, cellassign_reference_name]
// add values to meta for later use
.map{ project_id, meta, processed_sce, singler_model_file, cellassign_reference_file, cellassign_reference_name ->
.map{ project_id, meta, processed_sce, singler_model_file, singler_reference_name, cellassign_reference_file, cellassign_reference_name ->
meta.celltype_publish_dir = "${params.checkpoints_dir}/celltype/${meta.library_id}";
meta.singler_dir = "${meta.celltype_publish_dir}/${meta.library_id}_singler";
meta.cellassign_dir = "${meta.celltype_publish_dir}/${meta.library_id}_cellassign";
meta.singler_model_file = singler_model_file;
meta.singler_model_file = singler_model_file;
sjspielman marked this conversation as resolved.
Show resolved Hide resolved
meta.singler_reference_name = singler_reference_name;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting updating these to remove the singler_reference_name as well.

meta.cellassign_reference_file = cellassign_reference_file;
meta.cellassign_reference_name = cellassign_reference_name;
// return simplified input:
[meta, processed_sce]
}



// creates [meta, processed sce, singler model file]
singler_input_ch = celltype_input_ch
// add in singler model or empty file
.map{it.toList() + [file(it[0].singler_model_file ?: empty_file)]}
Expand All @@ -133,23 +157,53 @@ workflow annotate_celltypes {
missing_ref: it[2].name == "NO_FILE"
do_singler: true
}


// perform singleR celltyping and export results
classify_singler(singler_input_ch.do_singler)

sjspielman marked this conversation as resolved.
Show resolved Hide resolved
// singleR output channel: [library_id, singler_results]
singler_output_ch = singler_input_ch.missing_ref
.map{[it[0]["library_id"], file(empty_file)]}
// add in channel outputs
.mix(classify_singler.out)


// input for rds assignment step

// create cellassign input channel: [meta, processed sce, cellassign reference file]
cellassign_input_ch = celltype_input_ch
// add in cellassign reference
.map{it.toList() + [file(it[0].cellassign_reference_file ?: empty_file)]}
// skip if no cellassign reference file or reference name is not defined
.branch{
missing_ref: it[2].name == "NO_FILE"
do_cellassign: true
}


// perform CellAssign celltyping and export results
classify_cellassign(cellassign_input_ch.do_cellassign)

// cellassign output channel: [library_id, cellassign_dir]
cellassign_output_ch = cellassign_input_ch.missing_ref
.map{[it[0]["library_id"], file(empty_file)]}
// add in channel outputs
.mix(classify_cellassign.out)

// prepare input for process to add celltypes to the processed SCE
assignment_input_ch = processed_sce_channel
.map{[it[0]["library_id"]] + it}
// add in singler results
.join(singler_output_ch, by: 0, failOnMismatch: true, failOnDuplicate: true)
// add in cell assign results
.join(cellassign_output_ch, by: 0, failOnMismatch: true, failOnDuplicate: true)
.map{it.drop(1)} // remove library_id
// bring out the reference names
.map{meta, processed_sce, singler_dir, cellassign_dir ->
[meta, processed_sce, singler_dir, meta.singler_reference_name, cellassign_dir, meta.cellassign_reference_name]}
sjspielman marked this conversation as resolved.
Show resolved Hide resolved


// Next PR:
//add_celltypes_to_sce(assignment_input_ch)

// add back in the unchanged sce files
// TODO update below with output channel results:
// export_channel = processed_sce_channel
Expand Down