notebooks/getting_started/exploring_clinical_data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/exploring_clinical_data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Working with IDC clinical data without BigQuery\n",
        "\n",
        "In this notebook we cover the basics of how you can access and search IDC clinical data without depending on Google BigQuery.\n",
        "\n",
        "In addition to maintaining clinical data in Google BigQuery tables, we also export those in Parquet format into a public cloud-based storage bucket. Those files are free to download, and are rather small (as of IDC v18, less than 65MB altogether).\n",
        "\n",
        "Once downloaded, you can search the content using Pandas sytax of SQL.\n",
        "\n",
        "This brief notebook will guide you through the steps of the above.\n",
        "\n",
        "If you have never worked with IDC before, we recommend you first complete the getting started tutorial [here](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part2_searching_basics.ipynb).\n",
        "\n",
        "---\n",
        "Initial version: Jul 2024\n",
        "\n",
        "Updated: Sep 2024"
      ],
      "metadata": {
        "id": "RVHEoPZJgVbl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prerequisites\n",
        "\n",
        "The only prerequisite is [`idc-index`](https://github.com/ImagingDataCommons/idc-index) - python package that contains various utilities to simplify access to IDC data. As part of this package installation, you will get several other packages that we will use later:\n",
        "* `s5cmd` for very efficient download of data from cloud buckets using S3 API\n",
        "* `pandas` for dataframe operations\n",
        "* `duckdb` for querying pandas dataframes using SQL syntax"
      ],
      "metadata": {
        "id": "t-tzpK4DakEP"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "k8B1uiZkYlHu"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install --upgrade idc-index"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fetch clinical data index\n",
        "\n",
        "`idc-index` packages various tables with the key metadata. We refer to those as _indices_. The main index that supports API calls related to download and search is installed by default. To support search of the clinical data accompanying IDC images you will need the `clinical_index` table, which contains the list of all columns and all tables across all of the IDC collections that are available."
      ],
      "metadata": {
        "id": "neM2YGeQamIu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from idc_index import index\n",
        "\n",
        "c = index.IDCClient()\n",
        "\n",
        "c.fetch_index('clinical_index')\n",
        "\n",
        "print('Columns avaialable in clinical_index:\\n'+'\\n'.join(c.clinical_index.keys()))"
      ],
      "metadata": {
        "id": "ehb4MeuPYy4c",
        "outputId": "c7267c6e-9a3f-4378-86aa-9868c78cf5dc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Columns avaialable in clinical_index:\n",
            "collection_id\n",
            "table_name\n",
            "short_table_name\n",
            "column\n",
            "column_label\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Find all clinical metadata available for the specific collection\n",
        "\n",
        "A common use case is to find all clinical data available for a specific IDC collection.\n",
        "\n",
        "The key columns of this dataframe are:\n",
        "* `collection_id`: which collection given metadata attribute corresponds to\n",
        "* `table_name`: the name of the table where this metadata attribute is located\n",
        "* `column`: name of the column (attribute)\n",
        "\n",
        "Depending on the specific attribute and how it was provided/documented by the submitter, you may find more information about it in the `column_label` column.\n",
        "\n",
        "Let's assume we are interested in the clinical data accompanying the `rms_mutation_prediction` collection. We can select all clinical data attributes that are available for this collection as shown next."
      ],
      "metadata": {
        "id": "DvIHHAg7ao8F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define the query that selects all rows where collection_id is 'rms_mutation_prediction'\n",
        "# note that we can refer to clinical_index table in the query\n",
        "query = \"\"\"\n",
        "SELECT *\n",
        "FROM clinical_index\n",
        "WHERE collection_id = 'rms_mutation_prediction'\n",
        "\"\"\"\n",
        "\n",
        "# execute the query\n",
        "matching_items = c.sql_query(query)\n"
      ],
      "metadata": {
        "id": "L6rMuMEHjiML"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "matching_items"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "X5jfORxlaQ-Q",
        "outputId": "76af1c40-a713-4639-a730-b7e82bf56414"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              collection_id  \\\n",
              "0   rms_mutation_prediction   \n",
              "1   rms_mutation_prediction   \n",
              "2   rms_mutation_prediction   \n",
              "3   rms_mutation_prediction   \n",
              "4   rms_mutation_prediction   \n",
              "5   rms_mutation_prediction   \n",
              "6   rms_mutation_prediction   \n",
              "7   rms_mutation_prediction   \n",
              "8   rms_mutation_prediction   \n",
              "9   rms_mutation_prediction   \n",
              "10  rms_mutation_prediction   \n",
              "11  rms_mutation_prediction   \n",
              "12  rms_mutation_prediction   \n",
              "13  rms_mutation_prediction   \n",
              "14  rms_mutation_prediction   \n",
              "15  rms_mutation_prediction   \n",
              "16  rms_mutation_prediction   \n",
              "17  rms_mutation_prediction   \n",
              "18  rms_mutation_prediction   \n",
              "19  rms_mutation_prediction   \n",
              "20  rms_mutation_prediction   \n",
              "21  rms_mutation_prediction   \n",
              "22  rms_mutation_prediction   \n",
              "23  rms_mutation_prediction   \n",
              "24  rms_mutation_prediction   \n",
              "25  rms_mutation_prediction   \n",
              "26  rms_mutation_prediction   \n",
              "27  rms_mutation_prediction   \n",
              "28  rms_mutation_prediction   \n",
              "29  rms_mutation_prediction   \n",
              "30  rms_mutation_prediction   \n",
              "31  rms_mutation_prediction   \n",
              "32  rms_mutation_prediction   \n",
              "33  rms_mutation_prediction   \n",
              "34  rms_mutation_prediction   \n",
              "35  rms_mutation_prediction   \n",
              "36  rms_mutation_prediction   \n",
              "37  rms_mutation_prediction   \n",
              "38  rms_mutation_prediction   \n",
              "\n",
              "                                           table_name  \\\n",
              "0   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "1   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "2   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "3   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "4   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "5   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "6   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "7   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "8   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "9   bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "10  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "11  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "12  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "13  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "14  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "15  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "16  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "17  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "18  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "19  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "20  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "21  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "22  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "23  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "24  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "25  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "26  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "27  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "28  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "29  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "30  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "31  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "32  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "33  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "34  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "35  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "36  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "37  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "38  bigquery-public-data.idc_v19_clinical.rms_muta...   \n",
              "\n",
              "                        short_table_name                              column  \\\n",
              "0   rms_mutation_prediction_demographics                        source_batch   \n",
              "1   rms_mutation_prediction_demographics                                race   \n",
              "2   rms_mutation_prediction_demographics                              gender   \n",
              "3   rms_mutation_prediction_demographics                    dicom_patient_id   \n",
              "4   rms_mutation_prediction_demographics                      participant_id   \n",
              "5      rms_mutation_prediction_diagnosis                        source_batch   \n",
              "6      rms_mutation_prediction_diagnosis                        diagnosis_id   \n",
              "7      rms_mutation_prediction_diagnosis                        disease_type   \n",
              "8      rms_mutation_prediction_diagnosis                   primary_diagnosis   \n",
              "9      rms_mutation_prediction_diagnosis  primary_diagnosis_reference_source   \n",
              "10     rms_mutation_prediction_diagnosis             metastasis_at_diagnosis   \n",
              "11     rms_mutation_prediction_diagnosis                               stage   \n",
              "12     rms_mutation_prediction_diagnosis           last_known_disease_status   \n",
              "13     rms_mutation_prediction_diagnosis   days_to_last_known_disease_status   \n",
              "14     rms_mutation_prediction_diagnosis           tissue_or_organ_of_origin   \n",
              "15     rms_mutation_prediction_diagnosis           progression_or_recurrence   \n",
              "16     rms_mutation_prediction_diagnosis         site_of_resection_or_biopsy   \n",
              "17     rms_mutation_prediction_diagnosis                    dicom_patient_id   \n",
              "18     rms_mutation_prediction_diagnosis           participantparticipant_id   \n",
              "19     rms_mutation_prediction_diagnosis                        primary_site   \n",
              "20     rms_mutation_prediction_diagnosis                  days_to_recurrence   \n",
              "21     rms_mutation_prediction_diagnosis               days_to_last_followup   \n",
              "22     rms_mutation_prediction_diagnosis                    age_at_diagnosis   \n",
              "23        rms_mutation_prediction_sample                        source_batch   \n",
              "24        rms_mutation_prediction_sample                         sample_type   \n",
              "25        rms_mutation_prediction_sample         histological_classification   \n",
              "26        rms_mutation_prediction_sample                         tumor_grade   \n",
              "27        rms_mutation_prediction_sample              tumor_stage_clinical_t   \n",
              "28        rms_mutation_prediction_sample              tumor_stage_clinical_n   \n",
              "29        rms_mutation_prediction_sample              tumor_stage_clinical_m   \n",
              "30        rms_mutation_prediction_sample                    tumor_morphology   \n",
              "31        rms_mutation_prediction_sample                tumor_incidence_type   \n",
              "32        rms_mutation_prediction_sample                  sample_description   \n",
              "33        rms_mutation_prediction_sample                 sample_tumor_status   \n",
              "34        rms_mutation_prediction_sample                    dicom_patient_id   \n",
              "35        rms_mutation_prediction_sample           participantparticipant_id   \n",
              "36        rms_mutation_prediction_sample                           sample_id   \n",
              "37        rms_mutation_prediction_sample                sample_anatomic_site   \n",
              "38        rms_mutation_prediction_sample       participant_age_at_collection   \n",
              "\n",
              "                          column_label  \n",
              "0          idc_provenance_source_batch  \n",
              "1                                 race  \n",
              "2                               gender  \n",
              "3      idc_provenance_dicom_patient_id  \n",
              "4                       participant_id  \n",
              "5          idc_provenance_source_batch  \n",
              "6                         diagnosis_id  \n",
              "7                         disease_type  \n",
              "8                    primary_diagnosis  \n",
              "9   primary_diagnosis_reference_source  \n",
              "10             Metastasis_at_diagnosis  \n",
              "11                               Stage  \n",
              "12           last_known_disease_status  \n",
              "13   days_to_last_known_disease_status  \n",
              "14           tissue_or_organ_of_origin  \n",
              "15           progression_or_recurrence  \n",
              "16         site_of_resection_or_biopsy  \n",
              "17     idc_provenance_dicom_patient_id  \n",
              "18          participant.participant_id  \n",
              "19                        primary_site  \n",
              "20                  days_to_recurrence  \n",
              "21               days_to_last_followup  \n",
              "22                    age_at_diagnosis  \n",
              "23         idc_provenance_source_batch  \n",
              "24                         sample_type  \n",
              "25         Histological_Classification  \n",
              "26                         tumor_grade  \n",
              "27              tumor_stage_clinical_t  \n",
              "28              tumor_stage_clinical_n  \n",
              "29              tumor_stage_clinical_m  \n",
              "30                    tumor_morphology  \n",
              "31                tumor_incidence_type  \n",
              "32                  sample_description  \n",
              "33                 sample_tumor_status  \n",
              "34     idc_provenance_dicom_patient_id  \n",
              "35          participant.participant_id  \n",
              "36                           sample_id  \n",
              "37                sample_anatomic_site  \n",
              "38       participant_age_at_collection  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c180b0c2-de5e-4be7-810d-163f93516b91\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>table_name</th>\n",
              "      <th>short_table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_demographics</td>\n",
              "      <td>source_batch</td>\n",
              "      <td>idc_provenance_source_batch</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_demographics</td>\n",
              "      <td>race</td>\n",
              "      <td>race</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_demographics</td>\n",
              "      <td>gender</td>\n",
              "      <td>gender</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_demographics</td>\n",
              "      <td>dicom_patient_id</td>\n",
              "      <td>idc_provenance_dicom_patient_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_demographics</td>\n",
              "      <td>participant_id</td>\n",
              "      <td>participant_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>source_batch</td>\n",
              "      <td>idc_provenance_source_batch</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>diagnosis_id</td>\n",
              "      <td>diagnosis_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>disease_type</td>\n",
              "      <td>disease_type</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>primary_diagnosis</td>\n",
              "      <td>primary_diagnosis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>primary_diagnosis_reference_source</td>\n",
              "      <td>primary_diagnosis_reference_source</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>metastasis_at_diagnosis</td>\n",
              "      <td>Metastasis_at_diagnosis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>stage</td>\n",
              "      <td>Stage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>last_known_disease_status</td>\n",
              "      <td>last_known_disease_status</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>days_to_last_known_disease_status</td>\n",
              "      <td>days_to_last_known_disease_status</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>tissue_or_organ_of_origin</td>\n",
              "      <td>tissue_or_organ_of_origin</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>progression_or_recurrence</td>\n",
              "      <td>progression_or_recurrence</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>site_of_resection_or_biopsy</td>\n",
              "      <td>site_of_resection_or_biopsy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>dicom_patient_id</td>\n",
              "      <td>idc_provenance_dicom_patient_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>participantparticipant_id</td>\n",
              "      <td>participant.participant_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>primary_site</td>\n",
              "      <td>primary_site</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>days_to_recurrence</td>\n",
              "      <td>days_to_recurrence</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>days_to_last_followup</td>\n",
              "      <td>days_to_last_followup</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_diagnosis</td>\n",
              "      <td>age_at_diagnosis</td>\n",
              "      <td>age_at_diagnosis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>source_batch</td>\n",
              "      <td>idc_provenance_source_batch</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>sample_type</td>\n",
              "      <td>sample_type</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>histological_classification</td>\n",
              "      <td>Histological_Classification</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>tumor_grade</td>\n",
              "      <td>tumor_grade</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>tumor_stage_clinical_t</td>\n",
              "      <td>tumor_stage_clinical_t</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>tumor_stage_clinical_n</td>\n",
              "      <td>tumor_stage_clinical_n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>tumor_stage_clinical_m</td>\n",
              "      <td>tumor_stage_clinical_m</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>tumor_morphology</td>\n",
              "      <td>tumor_morphology</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>tumor_incidence_type</td>\n",
              "      <td>tumor_incidence_type</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>sample_description</td>\n",
              "      <td>sample_description</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>sample_tumor_status</td>\n",
              "      <td>sample_tumor_status</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>dicom_patient_id</td>\n",
              "      <td>idc_provenance_dicom_patient_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>participantparticipant_id</td>\n",
              "      <td>participant.participant_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>sample_id</td>\n",
              "      <td>sample_id</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>sample_anatomic_site</td>\n",
              "      <td>sample_anatomic_site</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>rms_mutation_prediction</td>\n",
              "      <td>bigquery-public-data.idc_v19_clinical.rms_muta...</td>\n",
              "      <td>rms_mutation_prediction_sample</td>\n",
              "      <td>participant_age_at_collection</td>\n",
              "      <td>participant_age_at_collection</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c180b0c2-de5e-4be7-810d-163f93516b91')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c180b0c2-de5e-4be7-810d-163f93516b91 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c180b0c2-de5e-4be7-810d-163f93516b91');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-586b5f88-5b81-4a6d-92db-a46c1f959bff\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-586b5f88-5b81-4a6d-92db-a46c1f959bff')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-586b5f88-5b81-4a6d-92db-a46c1f959bff button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_839fe134-abe4-4550-996b-3c709908af12\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('matching_items')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_839fe134-abe4-4550-996b-3c709908af12 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('matching_items');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "matching_items",
              "summary": "{\n  \"name\": \"matching_items\",\n  \"rows\": 39,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"rms_mutation_prediction\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"bigquery-public-data.idc_v19_clinical.rms_mutation_prediction_demographics\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"short_table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"rms_mutation_prediction_demographics\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 34,\n        \"samples\": [\n          \"site_of_resection_or_biopsy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 34,\n        \"samples\": [\n          \"site_of_resection_or_biopsy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Downloading the tables with clinical data\n",
        "\n",
        "The `table` column in `clinical_index` refers to the actual tables that contain the listed columns. Those tables are relatively small, and are maintained by IDC both in Google BigQuery and in Parquet files available for download from a public AWS bucket.\n",
        "\n",
        "Since the total size of those tables is small, it is easiest to download all of them (but of course you can also download just the individual ones)."
      ],
      "metadata": {
        "id": "dFqOIXInR0N0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "idc_version = c.get_idc_version()\n",
        "\n",
        "clinical_data_aws_path = f\"s3://idc-open-metadata/bigquery_export/idc_{idc_version}_clinical/*\"\n",
        "\n",
        "!mkdir -p idc_clinical_data\n",
        "!s5cmd --no-sign-request cp {clinical_data_aws_path} ./idc_clinical_data"
      ],
      "metadata": {
        "id": "BksJlKqwSPPw"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Access table that contains specific metadata attribute\n",
        "\n",
        "Let's assume we are interested in the `tumor_grade` attribute (row 13 in the table above). From the above, column `table_name` is telling us that it is contained in the table `bigquery-public-data.idc_v18_clinical.rms_mutation_prediction_sample`. The table name is the fully resolved name in BigQuery. For locating this table as downloaded, we need just the last component of the name: `rms_mutation_prediction_sample`.\n",
        "\n",
        "We can access and search this table next."
      ],
      "metadata": {
        "id": "3dbLIUCea0hd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "a_table_df = pd.read_parquet('./idc_clinical_data/rms_mutation_prediction_sample')\n",
        "\n",
        "a_table_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 583
        },
        "id": "geX2KhQqcWHc",
        "outputId": "390f6bb1-91f5-46c0-886c-400175275650"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    dicom_patient_id  source_batch participantparticipant_id sample_id  \\\n",
              "0            RMS2325             0                   RMS2325    PAWDLM   \n",
              "1            RMS2124             0                   RMS2124    PATMDI   \n",
              "2            RMS2137             0                   RMS2137    PATVPL   \n",
              "3            RMS2140             0                   RMS2140    PATYYW   \n",
              "4            RMS2145             0                   RMS2145    PAUKHP   \n",
              "..               ...           ...                       ...       ...   \n",
              "398          RMS2374             0                   RMS2374    PAUPVA   \n",
              "399          RMS2352             0                   RMS2352    PASGZC   \n",
              "400          RMS2205             0                   RMS2205    PAMSJL   \n",
              "401          RMS2267             0                   RMS2267    PALWAA   \n",
              "402          RMS2459             0                   RMS2459    PAPRFM   \n",
              "\n",
              "    sample_type   sample_anatomic_site  participant_age_at_collection  \\\n",
              "0         Tumor                    Leg                          44.56   \n",
              "1         Tumor                                                  0.90   \n",
              "2         Tumor                                                  0.83   \n",
              "3         Tumor                                                  1.07   \n",
              "4         Tumor                                                  2.72   \n",
              "..          ...                    ...                            ...   \n",
              "398       Tumor   Paratesticular, left                           2.46   \n",
              "399       Tumor  Paratesticular, right                           0.68   \n",
              "400       Tumor                 Pelvis                           2.76   \n",
              "401       Tumor   Soft tissue, abdomen                          17.96   \n",
              "402       Tumor               Prostate                           8.39   \n",
              "\n",
              "                           histological_classification tumor_grade  \\\n",
              "0                                                                    \n",
              "1                                             BOTRYOID               \n",
              "2                                             BOTRYOID               \n",
              "3                                             BOTRYOID               \n",
              "4                                             BOTRYOID               \n",
              "..                                                 ...         ...   \n",
              "398                      SPINDLE CELL RHABDOMYOSARCOMA               \n",
              "399                      SPINDLE CELL RHABDOMYOSARCOMA               \n",
              "400      MIXED ALVEOLAR AND EMBRYONAL RHABDOMYOSARCOMA               \n",
              "401      MIXED ALVEOLAR AND EMBRYONAL RHABDOMYOSARCOMA               \n",
              "402  EMBRYONAL RHABDOMYOSARCOMA WITH DIFFUSE ANAPLASIA               \n",
              "\n",
              "    tumor_stage_clinical_t tumor_stage_clinical_n tumor_stage_clinical_m  \\\n",
              "0                                                                          \n",
              "1                                                                          \n",
              "2                                                                          \n",
              "3                                                                          \n",
              "4                                                                          \n",
              "..                     ...                    ...                    ...   \n",
              "398                                                                        \n",
              "399                                                                        \n",
              "400                                                                        \n",
              "401                                                                        \n",
              "402                                                                        \n",
              "\n",
              "    tumor_morphology tumor_incidence_type sample_description  \\\n",
              "0                                                              \n",
              "1                                                              \n",
              "2                                                              \n",
              "3                                                              \n",
              "4                                                              \n",
              "..               ...                  ...                ...   \n",
              "398                                                            \n",
              "399                                                            \n",
              "400                                                            \n",
              "401                                                            \n",
              "402                                                            \n",
              "\n",
              "    sample_tumor_status  \n",
              "0                 Tumor  \n",
              "1                 Tumor  \n",
              "2                 Tumor  \n",
              "3                 Tumor  \n",
              "4                 Tumor  \n",
              "..                  ...  \n",
              "398               Tumor  \n",
              "399               Tumor  \n",
              "400               Tumor  \n",
              "401               Tumor  \n",
              "402               Tumor  \n",
              "\n",
              "[403 rows x 16 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5bfbdb6c-7531-48c0-b05f-e7db786c7aaa\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>dicom_patient_id</th>\n",
              "      <th>source_batch</th>\n",
              "      <th>participantparticipant_id</th>\n",
              "      <th>sample_id</th>\n",
              "      <th>sample_type</th>\n",
              "      <th>sample_anatomic_site</th>\n",
              "      <th>participant_age_at_collection</th>\n",
              "      <th>histological_classification</th>\n",
              "      <th>tumor_grade</th>\n",
              "      <th>tumor_stage_clinical_t</th>\n",
              "      <th>tumor_stage_clinical_n</th>\n",
              "      <th>tumor_stage_clinical_m</th>\n",
              "      <th>tumor_morphology</th>\n",
              "      <th>tumor_incidence_type</th>\n",
              "      <th>sample_description</th>\n",
              "      <th>sample_tumor_status</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>RMS2325</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2325</td>\n",
              "      <td>PAWDLM</td>\n",
              "      <td>Tumor</td>\n",
              "      <td>Leg</td>\n",
              "      <td>44.56</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>RMS2124</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2124</td>\n",
              "      <td>PATMDI</td>\n",
              "      <td>Tumor</td>\n",
              "      <td></td>\n",
              "      <td>0.90</td>\n",
              "      <td>BOTRYOID</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>RMS2137</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2137</td>\n",
              "      <td>PATVPL</td>\n",
              "      <td>Tumor</td>\n",
              "      <td></td>\n",
              "      <td>0.83</td>\n",
              "      <td>BOTRYOID</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>RMS2140</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2140</td>\n",
              "      <td>PATYYW</td>\n",
              "      <td>Tumor</td>\n",
              "      <td></td>\n",
              "      <td>1.07</td>\n",
              "      <td>BOTRYOID</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>RMS2145</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2145</td>\n",
              "      <td>PAUKHP</td>\n",
              "      <td>Tumor</td>\n",
              "      <td></td>\n",
              "      <td>2.72</td>\n",
              "      <td>BOTRYOID</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>398</th>\n",
              "      <td>RMS2374</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2374</td>\n",
              "      <td>PAUPVA</td>\n",
              "      <td>Tumor</td>\n",
              "      <td>Paratesticular, left</td>\n",
              "      <td>2.46</td>\n",
              "      <td>SPINDLE CELL RHABDOMYOSARCOMA</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>399</th>\n",
              "      <td>RMS2352</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2352</td>\n",
              "      <td>PASGZC</td>\n",
              "      <td>Tumor</td>\n",
              "      <td>Paratesticular, right</td>\n",
              "      <td>0.68</td>\n",
              "      <td>SPINDLE CELL RHABDOMYOSARCOMA</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>400</th>\n",
              "      <td>RMS2205</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2205</td>\n",
              "      <td>PAMSJL</td>\n",
              "      <td>Tumor</td>\n",
              "      <td>Pelvis</td>\n",
              "      <td>2.76</td>\n",
              "      <td>MIXED ALVEOLAR AND EMBRYONAL RHABDOMYOSARCOMA</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>401</th>\n",
              "      <td>RMS2267</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2267</td>\n",
              "      <td>PALWAA</td>\n",
              "      <td>Tumor</td>\n",
              "      <td>Soft tissue, abdomen</td>\n",
              "      <td>17.96</td>\n",
              "      <td>MIXED ALVEOLAR AND EMBRYONAL RHABDOMYOSARCOMA</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>402</th>\n",
              "      <td>RMS2459</td>\n",
              "      <td>0</td>\n",
              "      <td>RMS2459</td>\n",
              "      <td>PAPRFM</td>\n",
              "      <td>Tumor</td>\n",
              "      <td>Prostate</td>\n",
              "      <td>8.39</td>\n",
              "      <td>EMBRYONAL RHABDOMYOSARCOMA WITH DIFFUSE ANAPLASIA</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>Tumor</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>403 rows × 16 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5bfbdb6c-7531-48c0-b05f-e7db786c7aaa')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5bfbdb6c-7531-48c0-b05f-e7db786c7aaa button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5bfbdb6c-7531-48c0-b05f-e7db786c7aaa');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-499e2ee0-0a76-4083-9985-65f1c0dc688c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-499e2ee0-0a76-4083-9985-65f1c0dc688c')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-499e2ee0-0a76-4083-9985-65f1c0dc688c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_facf97a2-363d-4ac3-8996-288f95850c72\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('a_table_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_facf97a2-363d-4ac3-8996-288f95850c72 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('a_table_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "a_table_df",
              "summary": "{\n  \"name\": \"a_table_df\",\n  \"rows\": 403,\n  \"fields\": [\n    {\n      \"column\": \"dicom_patient_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 403,\n        \"samples\": [\n          \"RMS2394\",\n          \"RMS2429\",\n          \"RMS2435\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source_batch\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"participantparticipant_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 403,\n        \"samples\": [\n          \"RMS2394\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sample_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 403,\n        \"samples\": [\n          \"PAKXHZ\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sample_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Tumor\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sample_anatomic_site\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 179,\n        \"samples\": [\n          \"Skull Base, Right\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"participant_age_at_collection\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 6.371507327410722,\n        \"min\": 0.02,\n        \"max\": 45.25,\n        \"num_unique_values\": 360,\n        \"samples\": [\n          12.75\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"histological_classification\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"BOTRYOID\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_grade\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_stage_clinical_t\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_stage_clinical_n\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_stage_clinical_m\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_morphology\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_incidence_type\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sample_description\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sample_tumor_status\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Tumor\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that this table is loaded, you can search it as you would any pandas dataframe (or you can use SQL with duckdb, as shown earlier!).\n",
        "\n",
        "Note that the `dicom_patient_id` column, which you will find in **every** clinical data table, can be used to link clinical metadata attributes to the DICOM image metadata!"
      ],
      "metadata": {
        "id": "NXVWCXJTk5Dk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Summary\n",
        "\n",
        "We hope you enjoyed this tutorial! If something didn't work as expected, if you have any feedback or suggestions of what should be added to this tutorial, please contact IDC support by sending email to support@canceridc.dev or posting your question on [IDC User forum](https://discourse.canceridc.dev)."
      ],
      "metadata": {
        "id": "C1T3OjYBleUW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Acknowledgments\n",
        "\n",
        "Imaging Data Commons has been funded in whole or in part with Federal funds from the National Cancer Institute, National Institutes of Health, under Task Order No. HHSN26110071 under Contract No. HHSN261201500003l.\n",
        "\n",
        "If you use IDC in your research, please cite the following publication:\n",
        "\n",
        "> Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. _National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence_. RadioGraphics (2023). https://doi.org/10.1148/rg.230180"
      ],
      "metadata": {
        "id": "EldMrJ_llh5B"
      }
    }
  ]
}