dutch-umls_to_concept-table.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ab651d93",
   "metadata": {},
   "source": [
    "# Dutch UMLS to concept table\n",
    "This notebook describes how to create a UMLS concept table containing Dutch names, to be used in a named entity recognition and linking tool such as MedCAT. In the second part of this notebook, names from Dutch SNOMED are added. In the third part, English drug names are added, because these concepts are not well represented in the Dutch UMLS source vocabularies. \n",
    "\n",
    "Mapping from SNOMED Dutch to UMLS can be difficult because of many-to-many mapping, explained in this notebook.\n",
    "\n",
    "Requirements:\n",
    "- UMLS MySQL database containing Dutch ontologies.\n",
    "\n",
    "For adding Dutch SNOMED names:\n",
    "- Dutch SNOMED concept table, created in `dutch-snomed_to_concept-table.ipynb`\n",
    "- UMLS MySQL database containing SNOMED-US, which is used for mapping SNOMED Dutch -> UMLS\n",
    "\n",
    "For adding English drug names:\n",
    "- UMLS MySQL database containing English drug ontologies such as RXNORM, ATC and Drugbank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce8b674a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from dotenv import load_dotenv\n",
    "from IPython.display import display\n",
    "from pathlib import Path\n",
    "from sqlalchemy import create_engine\n",
    "from utils import clean_name_status_column, convert_title_to_lowercase\n",
    "\n",
    "pd.set_option('max_colwidth', 400)\n",
    "\n",
    "# Set output version of the generated UMLS dutch concept table\n",
    "UMLS_DUTCH_VERSION = 'v1.11'\n",
    "\n",
    "# Set version of SNOMED to append to UMLS terms\n",
    "snomed_dutch_file = Path('04_ConceptDB') / 'snomedct-dutch_v1.3.csv'\n",
    "\n",
    "# Set custom concepts file\n",
    "custom_concepts_file = Path(\"05_CustomConcepts\") / \"dutch-umls_custom_concepts.csv\"\n",
    "\n",
    "# Output files\n",
    "output_file = Path(\"04_ConceptDB\") / f'umls-dutch_{UMLS_DUTCH_VERSION}.csv'\n",
    "output_file_with_drug_names = Path('04_ConceptDB') / f'umls-dutch_{UMLS_DUTCH_VERSION}_with_drugs.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f3587d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Credentials to connect to UMLS MySQL database\n",
    "load_dotenv()\n",
    "user = os.getenv('MYSQL_USER')\n",
    "password = os.getenv('MYSQL_PASSWORD')\n",
    "host = os.getenv('MYSQL_HOST')\n",
    "port = os.getenv('MYSQL_PORT')\n",
    "database = os.getenv('MYSQL_DATABASE')\n",
    "\n",
    "# Create the connection\n",
    "connection_string = f'mysql://{user}:{password}@{host}:{port}/{database}'\n",
    "connection = create_engine(connection_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e627875",
   "metadata": {},
   "source": [
    "## Retrieve medical concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6b5c52f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve Dutch UMLS concepts\n",
    "query = \"\"\"\n",
    "SELECT cui, str, tty, sab FROM MRCONSO WHERE LAT = 'DUT'\n",
    "\"\"\"\n",
    "dutch_umls_original = pd.read_sql_query(query, con=connection)\n",
    "dutch_umls_original.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166d44df",
   "metadata": {},
   "source": [
    "## Manual corrections\n",
    "Some manual corrections. Easiest to do this as close to the source as possible, so they are processed downstream correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "498f8bde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Correct Respiratory Failure\n",
    "# Respiratory Insufficiency / C0035229 / 409623005 / https://uts.nlm.nih.gov/uts/umls/concept/C0035229\n",
    "# Respiratory Failure / C1145670 / 409622000 / https://uts.nlm.nih.gov/uts/umls/concept/C1145670\n",
    "display(dutch_umls_original.loc[dutch_umls_original.cui == 'C0035229'])\n",
    "display(dutch_umls_original.loc[dutch_umls_original.cui == 'C1145670'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27e9cab5",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_original.loc[dutch_umls_original.str == 'Respiratoire insufficiëntie, niet gespecificeerd', 'cui'] = 'C0035229'\n",
    "dutch_umls_original.loc[dutch_umls_original.str == 'Ademhalingsinsufficiëntie', 'cui'] = 'C0035229'\n",
    "display(dutch_umls_original.loc[dutch_umls_original.cui == 'C0035229'])\n",
    "display(dutch_umls_original.loc[dutch_umls_original.cui == 'C1145670'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec63d914",
   "metadata": {},
   "source": [
    "## Filter on terminology and type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fca77c25",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_original.sab.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f74cfc08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assess terms per terminology\n",
    "dutch_umls_original.loc[dutch_umls_original['sab'].isin(['LNC-NL-NL', 'ICPC2ICD10DUT'])].sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acb9a096",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'LNC-NL-NL' and 'ICPC2ICD10DUT' names are not usefull for named entities linking, so we exclude these\n",
    "dutch_umls_sab_filtered = dutch_umls_original.loc[~dutch_umls_original['sab'].isin(['LNC-NL-NL', 'ICPC2ICD10DUT'])].copy()\n",
    "dutch_umls_sab_filtered.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b1d2819",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assess terms per type\n",
    "dutch_umls_sab_filtered.loc[dutch_umls_sab_filtered['tty']=='HT'].sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f766e50",
   "metadata": {},
   "source": [
    "## Term type in source\n",
    "Some source-defined term types are not relevant for our use case. In the next part we will drop those. See https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/abbreviations.html "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "270be30d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_sab_filtered.tty.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "075b53fa",
   "metadata": {},
   "source": [
    "| TTY  | Description | Example |\n",
    "| - | - | - |\n",
    "| LLT | Lower Level Term | heupkombreuk, buikkramp|\n",
    "| PT | Designated preferred name | harthypertrofie, Pancoast-syndroom |\n",
    "| MH | Main heading | Dehydratie, Astma |\n",
    "| SY | Designated synonym | Spanningshoofdpijn, Ziekte van Hodgkin |\n",
    "| HT | Hierarchical term | calciummetabolismestoornissen, oculaire hemorragische aandoeningen |\n",
    "| HG | High Level Group Term  | complicaties geassocieerd met medisch hulpmiddel, zuur-basestoornissen |\n",
    "| SMQ| Standardised MedDRA Query | Leveraandoeningen (SMQ) , Tumormarkers (SMQ) |\n",
    "| CP | ICPC component process (in original form) | Ander bloedonderzoek, Medicatie/recept/injectie |\n",
    "| AB | Abbreviation in any source vocabulary | Infec, Neopl, Ear, Endo |\n",
    "| OS | System-organ class | Bloed- en lymfestelselaandoeningen, Infecties en parasitaire aandoeningen |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1088acde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a set of TTYs that seem most relevant for named entity recognition\n",
    "tty_selection = ['PT', 'LLT', 'MH', 'SY']\n",
    "dutch_umls_tty_filtered = dutch_umls_sab_filtered.loc[dutch_umls_sab_filtered.tty.isin(tty_selection)].copy()\n",
    "\n",
    "# Keep only relevant columns\n",
    "dutch_umls_tty_filtered = dutch_umls_tty_filtered[['cui', 'str', 'tty', 'sab']]\n",
    "dutch_umls_tty_filtered.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1faf74bb",
   "metadata": {},
   "source": [
    "## Preferred/pretty/primary names\n",
    "For MedCAT, and other named entity linking methods, it's useful to designate a single name as the preferred name, also sometimes called primary name or pretty name. This name can be presented to the end-user in webapplications, and should therefore be the most descriptive and commonly used name. All other names (synonyms, abreviations and common mispellings) will then be considered synonyms. MedCAT used 'P' as the value for preferred name, and all other names should have the value 'A' (see https://github.com/CogStack/MedCAT/blob/master/examples/README.md).\n",
    "\n",
    "Most, if not all, of UMLS concepts have a preferred in English. For other languages,\n",
    "it can be difficult to select a preferred name, because each source vocabulary has one or\n",
    "multiple preferred names for a concepts.\n",
    "\n",
    "It's not possible to keep the English UMLS preferred names, because MedCAT would add those names to the concept table for entity linking. Perhaps future functionality can be added for MedCAT to prevent taking these preferred names into account during entity linking.\n",
    "\n",
    "### Solution 1: Use UMLS source vocabularies preferred names\n",
    "For a rough but effective solution to get decent preferred names for the Dutch terms, change the terms that have the value \"Designated preferred name\" (PT) for the \"Term Type in Source\" (TTY) to MedCAT's preferred name value (P), and all others can be saved as (A). This leads to many concepts having multiple preferred names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32542b2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dutch_umls_tty_filtered.tty.replace({'PT': 'P',\n",
    "#                                      'LLT': 'A',\n",
    "#                                      'MH': 'A',\n",
    "#                                      'SY': 'A'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6597fb19",
   "metadata": {},
   "source": [
    "### Solution 2: Use preferred names from Dutch SNOMED\n",
    "In previous experiments we have shown that the Dutch vocabularies from UMLS and Dutch SNOMED complement each other. SNOMED provides most of the names, and contains excellent primary names. So we could use the preferred names from Dutch SNOMED, and for the terms not in that vocabulary, let MedCAT pick a random one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2e5db1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop tty column, put it back in just before merging with SNOMED\n",
    "dutch_umls_tty_filtered.drop(['tty'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97bc7f1e",
   "metadata": {},
   "source": [
    "## Clean values\n",
    "ICD10DUT and MDRDUT contain names that are more definitions than then how they would be found in text. For example, \"colonkanker NAO\" (see https://alt.meddra.org/files_acrobat/intguide_25_0_Dutch.pdf) and \"Aandoening van ooglid, niet gespecificeerd\". Names are more useful for named entity recongition when the descriptive part is removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "242a3b8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_clean = dutch_umls_tty_filtered.copy()\n",
    "\n",
    "# Remove ' nao' and ' NAO'\n",
    "print(f\"Number of terms containing ' NAO': {len(dutch_umls_clean.loc[dutch_umls_clean['str'].str.contains(' NAO')])}\")\n",
    "print(f\"Number of terms containing ' nao': {len(dutch_umls_clean.loc[dutch_umls_clean['str'].str.contains(' nao')])}\")\n",
    "dutch_umls_clean.str = dutch_umls_clean.str.replace({' NAO': '', ' \\(NAO\\)': '', ' nao': ''}, regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba345c03",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of terms containing 'gespecificeerd': {len(dutch_umls_clean.loc[dutch_umls_clean['str'].str.contains('gespecificeerd')])}\")\n",
    "\n",
    "# Remove suffix\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd onderzoek')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerde graad')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerde oorzaak')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerde plaats')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd type')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd deel')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd gebruik')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd naar behandelperiode')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd naar betrokkenheid')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerde toestand')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet-gespecificeerd naar oorzaak')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix(', niet gespecificeerd')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removesuffix('; Niet gespecificeerd')\n",
    "\n",
    "# Remove prefix\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removeprefix('niet-gespecificeerde ')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removeprefix('niet-gespecificeerd ')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removeprefix('Niet gespecificeerd ')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].str.removeprefix('Niet gespecificeerde ')\n",
    "\n",
    "print(f\"Number of terms containing 'gespecificeerd': {len(dutch_umls_clean.loc[dutch_umls_clean['str'].str.contains('gespecificeerd')])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a79e8dab",
   "metadata": {},
   "source": [
    "There are many occurences of 'gespecificeerd' in names left, but these are more difficult to clean. For now, these are likely not causing any issues (but probably also not adding much) so we'll keep them in our table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b47d0331",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert title-formatted names to lowercase\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].apply(convert_title_to_lowercase, split_char=' ')\n",
    "dutch_umls_clean['str'] = dutch_umls_clean['str'].apply(convert_title_to_lowercase, split_char='-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "925fcf68",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop duplicates\n",
    "print(f'Records before dropping duplicates: {dutch_umls_clean.shape[0]}')\n",
    "dutch_umls_clean = dutch_umls_clean.drop_duplicates(subset=['cui', 'str', 'sab'], keep='first').reset_index(drop=True)\n",
    "print(f'Records before dropping duplicates: {dutch_umls_clean.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70c71d04",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_clean[dutch_umls_clean.cui == 'C0002736']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cb632f1",
   "metadata": {},
   "source": [
    "## Merge rows from different vocabularies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0243660",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls = dutch_umls_clean.copy()\n",
    "\n",
    "# Merge SAB into single row\n",
    "print(f'Records before merging rows: {dutch_umls.shape[0]}')\n",
    "dutch_umls = dutch_umls.groupby(['cui','str'])['sab'].apply('|'.join).reset_index()\n",
    "print(f'Records after merging rows: {dutch_umls.shape[0]}')\n",
    "dutch_umls[dutch_umls.cui == 'C0002736']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993b1531",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add tty column with value 'A' to set these names as synonyms \n",
    "dutch_umls['tty'] = 'A'\n",
    "dutch_umls.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986283d0",
   "metadata": {},
   "source": [
    "# Add Dutch names from SNOMED\n",
    "UMLS does not contain the Dutch SNOMEDCT, but it does contain the English SNOMEDCT. So through the English SNOMED concepts, we can map the Dutch SNOMED names to UMLS.\n",
    "\n",
    "Dutch SNOMED names with SNOMED ID **->** Get English SNOMED ID to UMLS ID mapping **->** Map Dutch SNOMED names with SNOMED ID to UMLS ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5169a5c2",
   "metadata": {},
   "source": [
    "### Load SNOMED US"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bb1d0b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT distinct cui, scui FROM MRCONSO where sab = 'SNOMEDCT_US'\"\n",
    "snomed_us = pd.read_sql_query(query, con=connection)\n",
    "snomed_us.scui = snomed_us.scui.astype(str)\n",
    "print(f'SNOMED US terms with UMLS CUI: {snomed_us.shape[0]}')\n",
    "snomed_us.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c7b0218",
   "metadata": {},
   "source": [
    "### Load SNOMED NL\n",
    "We're using a cleaned and filtered list of Dutch SNOMED names, see other notebook in this repository how this is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4814eb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_dutch = pd.read_csv(snomed_dutch_file, dtype='str')\n",
    "snomed_dutch.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c4e4573",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_dutch.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14785a8a",
   "metadata": {},
   "source": [
    "## Find ambiguous mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3450fbb5",
   "metadata": {},
   "source": [
    "First find which SNOMED concepts can map to UMLS concepts. SNOMED concepts can map to multiple UMLS concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc4bb757",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create SNOMED - UMLS mapping\n",
    "snomed_to_umls_mapping = snomed_us.groupby('scui')['cui'].apply(list).to_dict()\n",
    "print(f'Number of SNOMED concepts that map to at least 1 UMLS concept: {len(snomed_to_umls_mapping)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec25f572",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check ambiguity of UMLS-SNOMED mapping\n",
    "unambiguous_mapping_ids = set()\n",
    "ambiguous_mapping_ids = set()\n",
    "for snomed_id in snomed_to_umls_mapping:\n",
    "    if len(snomed_to_umls_mapping[snomed_id]) == 1:\n",
    "        unambiguous_mapping_ids.add(snomed_id)\n",
    "    else:\n",
    "        ambiguous_mapping_ids.add(snomed_id)\n",
    "print(f'Number of SNOMED concepts that map to only 1 concept: {len(unambiguous_mapping_ids)}')\n",
    "print(f'Number of SNOMED concepts that map to multiple concepts: {len(ambiguous_mapping_ids)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e36cf048",
   "metadata": {},
   "source": [
    "So 2073 SNOMED concepts map to multiple UMLS concepts. If the Dutch names from these concepts are added to the UMLS concept table, it will introduce ambiguity, which could lead to problems in our downstream named entity linking methods. Therefore, for now, this method does not add names for these ambiguously mapping SNOMED concepts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cc6d14a",
   "metadata": {},
   "source": [
    "## Example of ambiguous mapping\n",
    "This section illustrates the ambiguous mapping problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5f6be88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find example\n",
    "ambiguous_mapping_ids = [int(code) for code in ambiguous_mapping_ids]\n",
    "ambiguous_mapping_ids.sort()\n",
    "ambiguous_mapping_ids = [str(code) for code in ambiguous_mapping_ids]\n",
    "ambiguous_mapping_ids[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d55263e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT distinct cui, code, str FROM MRCONSO where sab = 'SNOMEDCT_US' and CODE = '216004'\"\n",
    "snomed_us_example = pd.read_sql_query(query, con=connection)\n",
    "snomed_us_example.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a0039a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_dutch[snomed_dutch.cui == '216004']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e326ff18",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls[dutch_umls.cui.isin(['C0151836', 'C1704268'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74cd0d86",
   "metadata": {},
   "source": [
    "So SNOMED US has four names for 216004. Three of these names map to C1704268 and one maps to C0151836. In SNOMED NL, there is only one name for this concept. In our current UMLS table, we have 2 names for each UMLS concept. We could map the SNOMED name to both concepts (1), to a specific one (2), or skip it (3):\n",
    "\n",
    "1. Mapping to both will cause ambiguity. It could have an effect on entity linking, or it could be solved during MedCAT's disambiguation functionality based on unsupervised training, but that depends on the synonyms and their presence in the training corpus. In this example there is only 1 Dutch SNOMED term, but when there are multiple Dutch SNOMED terms, adding all to both terms will lead to many duplicates.\n",
    "2. Mapping to a single one is the best option for a single example, but this requires manual curreation, is time consuming, not within the scope of this project and can be quite difficult. There are about 2000 of these terms. SNOMED US names are in UMLS, so those ambiguously mapping names are added by either the UMLS or SNOMED team. Perhaps in future versions, this is corrected at either UMLS or SNOMED level.\n",
    "3. Skipping the name is the easiest option and will not lead to potential difficult downstream interpretation. The drawback is that the name, which in this example is not in any other Dutch ontology, will not be in the Dutch UMLS concept table.\n",
    "\n",
    "Currently, approach #3 is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944cb580",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another example of ambiguous SNOMED-CT -> UMLS mapping:\n",
    "snomed_dutch[snomed_dutch.cui == '2776000']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90d86982",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show that a single SNOMED ID maps to multiple UMLS concepts\n",
    "query = \"SELECT distinct cui, code, str FROM MRCONSO where sab = 'SNOMEDCT_US' and CODE = '2776000'\"\n",
    "snomed_us_example = pd.read_sql_query(query, con=connection)\n",
    "snomed_us_example.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d27ae809",
   "metadata": {},
   "source": [
    "## Merge SNOMED Dutch with UMLS Dutch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78158388",
   "metadata": {},
   "outputs": [],
   "source": [
    "def map_dutch_snomed_to_umls(row):\n",
    "    snomed_id = row['cui']\n",
    "    if snomed_id in unambiguous_mapping_ids:\n",
    "        cui = snomed_to_umls_mapping[snomed_id][0]\n",
    "        snomed_names_to_add.append([cui, row['str'], row['tty']])\n",
    "    else:\n",
    "        snomed_names_to_skip.append([snomed_id, row['str'], row['tty']])\n",
    "        \n",
    "# Create lists to fill with SNOMED names and their UMLS CUIs\n",
    "snomed_names_to_add = list()\n",
    "snomed_names_to_skip = list()\n",
    "\n",
    "# Map Dutch SNOMED to UMLS\n",
    "snomed_dutch.apply(map_dutch_snomed_to_umls, axis = 1)\n",
    "\n",
    "print(f'Number of Dutch names in existing UMLS table: {dutch_umls.shape[0]}')\n",
    "print(f'Number of Dutch SNOMED names to add: {len(snomed_names_to_add)}')\n",
    "print(f'Number of Dutch SNOMED names to skip: {len(snomed_names_to_skip)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e369e83",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Format SNOMED names in pandas dataframe\n",
    "snomed_names_with_cui = pd.DataFrame(snomed_names_to_add, columns = ['cui', 'str', 'tty'])\n",
    "snomed_names_with_cui['sab'] = 'SNOMEDCT_NL'\n",
    "snomed_names_with_cui.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212f098e",
   "metadata": {},
   "source": [
    "### Remove duplicate SNOMED concepts\n",
    "Earlier the problem was discussed of a SNOMED term that maps to multiple UMLS concepts. There's also the problem of snomed names that are ambiguous in SNOMED itself. Some of these, like \"abces\" are not ambiguous in UMLS. So when mapping these concepts to UMLS, the ambiguity is solved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2a4193f",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_dutch[snomed_dutch.str == 'abces']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e047b944",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_to_umls_mapping['44132006']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f2ae1f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_to_umls_mapping['128477000']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddb00ea5",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_names_with_cui[snomed_names_with_cui.cui == 'C0000833']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57c9c83a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Number of SNOMED concepts that include names that are ambiguous in SNOMED: {snomed_names_with_cui.shape[0]}')\n",
    "snomed_names_with_cui = snomed_names_with_cui.drop_duplicates(subset=['cui', 'str', 'sab', 'tty'], keep='first').reset_index(drop=True)\n",
    "print(f'Number of SNOMED names: {snomed_names_with_cui.shape[0]}')\n",
    "snomed_names_with_cui[snomed_names_with_cui.cui == 'C0000833']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28672019",
   "metadata": {},
   "source": [
    "### Clean SNOMED names\n",
    "Some names from SNOMED are also in Title-format, such as ziekte van Parkinson. To prevent duplication, lowercase these terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df8d4bf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(snomed_names_with_cui[snomed_names_with_cui.cui.isin(['C0030567', 'C0002736'])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8707247a",
   "metadata": {},
   "outputs": [],
   "source": [
    "snomed_names_with_cui['str'] = snomed_names_with_cui['str'].apply(convert_title_to_lowercase, split_char=' ')\n",
    "snomed_names_with_cui['str'] = snomed_names_with_cui['str'].apply(convert_title_to_lowercase, split_char='-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9066e31a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examples\n",
    "display(snomed_names_with_cui[snomed_names_with_cui.cui.isin(['C0030567', 'C0002736'])])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "200038dd",
   "metadata": {},
   "source": [
    "### Concatenate SNOMED names to UMLS table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56fcf1eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add SNOMED names to UMLS\n",
    "dutch_umls_snomed = pd.concat([dutch_umls, snomed_names_with_cui])\n",
    "print(f'Number of Dutch names in UMLS + SNOMED table: {dutch_umls_snomed.shape[0]}')\n",
    "\n",
    "dutch_umls_snomed.sort_values(by=['cui', 'tty', 'sab', 'str'], ascending=[True, False, True, True], inplace=True)\n",
    "dutch_umls_snomed.loc[dutch_umls_snomed.cui == 'C0030567']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "479d6591",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grouping rows on SAB\n",
    "dutch_umls_snomed = dutch_umls_snomed.groupby(['cui', 'str'], as_index=False).agg({'sab' : '|'.join, 'tty' : '|'.join}).copy()\n",
    "\n",
    "# Clean tty column\n",
    "dutch_umls_snomed.tty = dutch_umls_snomed.tty.apply(clean_name_status_column)\n",
    "dutch_umls_snomed.sort_values(by=['cui', 'tty'], ascending=[True, False], inplace=True)\n",
    "dutch_umls_snomed.reset_index(drop=True,inplace=True)\n",
    "print(f'Number of names after merging rows on SAB: {dutch_umls_snomed.shape[0]}')\n",
    "\n",
    "# Check example\n",
    "dutch_umls_snomed.loc[dutch_umls_snomed.cui == 'C0030567']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7739ab69",
   "metadata": {},
   "source": [
    "## Remove problematic names\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "066966fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "names_to_remove = ['bij', # C0004923\n",
    "                   'bijen', # C0004923\n",
    "                   'haar', # C0018494\n",
    "                   'bleek', # C0678215\n",
    "                   'weer', # C0043085\n",
    "                   'na+'] # C0337443\n",
    "dutch_umls_snomed[dutch_umls_snomed.str.isin(names_to_remove)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8b0951f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove rows\n",
    "rows_to_remove = dutch_umls_snomed[dutch_umls_snomed.str.isin(names_to_remove)].index\n",
    "print(f'Number of rows before removing rows: {dutch_umls_snomed.shape[0]}')\n",
    "dutch_umls_snomed = dutch_umls_snomed.drop(dutch_umls_snomed.index[rows_to_remove])\n",
    "print(f'Number of rows after removing rows: {dutch_umls_snomed.shape[0]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fce1b91",
   "metadata": {},
   "source": [
    "## Add custom CUIs\n",
    "Sometimes names or concept are not captured in any of the Dutch terminologies. By looking up the English name for these concepts, we can add custom Dutch names using the real UMLS identifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baae037b",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_snomed.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93933124",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_concepts = pd.read_csv(custom_concepts_file)\n",
    "custom_concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1427293",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Number of rows before adding rows: {dutch_umls_snomed.shape[0]}')\n",
    "dutch_umls_snomed = pd.concat([dutch_umls_snomed, custom_concepts])\n",
    "print(f'Number of rows after adding rows: {dutch_umls_snomed.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79a37f02",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_snomed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c826b181",
   "metadata": {},
   "source": [
    "## Add TUI (types)\n",
    "UMLS concepts have one or multiple types. These types are kept in a separate table, `MRSTY`. See https://semanticnetwork.nlm.nih.gov/download/SemGroups.txt for all types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f30e7d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load TUI table from MySQL\n",
    "query = \"\"\"\n",
    "SELECT cui, tui FROM MRSTY\n",
    "\"\"\"\n",
    "tui_original = pd.read_sql_query(query, con=connection)\n",
    "tui_original.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb08932",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add TUI column to UMLS + SNOMED CUI table\n",
    "dutch_umls_snomed = dutch_umls_snomed.merge(tui_original, how='left', on='cui')\n",
    "dutch_umls_snomed.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd6997c3",
   "metadata": {},
   "source": [
    "## TUI Filtering\n",
    "What types of concepts (TUIs) should be removed depends on the domain and question of subsequent analysis. After assessing performance of named entity linking of these names on a few clinical documents, our team dediced to remove the following TUIs.\n",
    "\n",
    "SNOMED names are already filtered in a seperate filtering step based on type, which is done in the notebook that creates the SNOMED concept table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04778bfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "tuis_to_remove = [\n",
    "    \n",
    "    # Concepts & Ideas\n",
    "    'T078', # Idea or Concept\n",
    "    'T089', # Regulation or Law\n",
    "\n",
    "    # Living beings\n",
    "    'T011', # Amphibian\n",
    "    'T008', # Animal\n",
    "    'T012', # Bird\n",
    "    'T013', # Fish\n",
    "    'T015', # Mammal\n",
    "    'T001', # Organism\n",
    "    'T001', # Plant\n",
    "    'T014', # Reptile\n",
    "    'T010', # Vertebrate\n",
    "    \n",
    "    # Objects\n",
    "    'T168', # Food\n",
    "    \n",
    "    # Organizations\n",
    "    'T093', # Healthcare Related Organization\n",
    "    \n",
    "    # Geographic areas\n",
    "    'T083', #Geographic Aera\n",
    "]\n",
    "                  \n",
    "dutch_umls_snomed[dutch_umls_snomed.tui.isin(tuis_to_remove)].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60d1cfaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove rows based on TUI\n",
    "rows_to_remove = dutch_umls_snomed[dutch_umls_snomed.tui.isin(tuis_to_remove)].index\n",
    "print(f'Number of rows before removing rows: {dutch_umls_snomed.shape[0]}')\n",
    "dutch_umls_snomed = dutch_umls_snomed.drop(dutch_umls_snomed.index[rows_to_remove])\n",
    "print(f'Number of rows after removing rows: {dutch_umls_snomed.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "167caec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_snomed = dutch_umls_snomed.groupby(['cui', 'str', 'tty', 'sab'])['tui'].apply('|'.join).reset_index()\n",
    "print(f'Number of rows after merging TUIs in single value: {dutch_umls_snomed.shape[0]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad56eaf3",
   "metadata": {},
   "source": [
    "### Update column names\n",
    "In MedCAT v1.0 the column name specification has changed and is defined as in the [README.md in examples](https://github.com/CogStack/MedCAT/tree/master/examples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9436af30",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_snomed.rename(columns={'str': 'name', 'tty': 'name_status', 'sab': 'ontologies', 'tui': 'type_ids'}, inplace=True)\n",
    "dutch_umls_snomed.sort_values(by=['cui', 'name_status'], ascending=[True, False], inplace=True)\n",
    "dutch_umls_snomed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db78d6e1",
   "metadata": {},
   "source": [
    "### Save output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "897a1c0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print statistics\n",
    "print(f'Unique concepts: {len(dutch_umls_snomed.cui.unique())}')\n",
    "print(f'Unique names: {len(dutch_umls_snomed.name.unique())}')\n",
    "print(f'Ambiguous names: {len(dutch_umls_snomed) - len(dutch_umls_snomed.name.unique())}')\n",
    "print(f'Total names in concepts: {len(dutch_umls_snomed)}')\n",
    "\n",
    "# Save final concept table\n",
    "dutch_umls_snomed.to_csv(output_file, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00f6580e",
   "metadata": {},
   "source": [
    "## Add drug names\n",
    "Only run this part below if you want to further expand the concept database with drug names, which adds around 270k lines. Many drugs only have an international name, or use the international name more often than the Dutch name, so adding these from ATC, Drugbank and RXNorm can be a good addition to the concept table. \n",
    "\n",
    "After assessing the resulting list it will be clear that many names will not be useful in named entity recognition, because they will probably never be used in natural language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "385e18aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "#In case you want to begin from here, load existing concept table:\n",
    "#dutch_umls_snomed = pd.read_csv(\"04_ConceptDB/umls-dutch_{UMLS_DUTCH_VERSION}.csv\", dtype=str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4edc4c91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve Dutch UMLS concepts\n",
    "query = \"\"\"\n",
    "SELECT distinct MRCONSO.cui, str as name, sab as ontologies \n",
    "FROM MRCONSO\n",
    "WHERE SAB in ('ATC','DRUGBANK','RXNORM')\n",
    "\"\"\"\n",
    "drugs_original = pd.read_sql_query(query, con=connection)\n",
    "drugs_original.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d58959a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert title format to lowercase\n",
    "drugs_original['name'] = drugs_original['name'].apply(convert_title_to_lowercase, split_char=' ')\n",
    "drugs_original['name'] = drugs_original['name'].apply(convert_title_to_lowercase, split_char='-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8b038a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "drugs_original[drugs_original.cui =='C0020740']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "449e5dfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use previously defined preferred names\n",
    "drugs_original['name_status'] = 'A'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df6362d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge drugs dataframe with umls_snomed dataframe\n",
    "dutch_umls_snomed_drugs = pd.concat([dutch_umls_snomed, drugs_original], axis=0)\n",
    "\n",
    "print(\"UMLS & SNOMED records: \", len(dutch_umls_snomed))\n",
    "print(\"Drug name records: \", len(drugs_original))\n",
    "print(\"Combined: \", len(dutch_umls_snomed_drugs))\n",
    "\n",
    "dutch_umls_snomed_drugs = dutch_umls_snomed_drugs.groupby(['cui', 'name', 'name_status'])['ontologies'].apply('|'.join).reset_index()\n",
    "print(\"Records after merging ontologies in single value: \", len(dutch_umls_snomed_drugs))\n",
    "dutch_umls_snomed_drugs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86b2478b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add TUI column\n",
    "dutch_umls_snomed_drugs = dutch_umls_snomed_drugs.merge(tui_original, how='left', on='cui')\n",
    "print(f'Number of rows containing TUIs: {dutch_umls_snomed_drugs.shape[0]}')\n",
    "\n",
    "# Remove TUIs that we decided to filter\n",
    "rows_to_remove = dutch_umls_snomed_drugs[dutch_umls_snomed_drugs.tui.isin(tuis_to_remove)].index\n",
    "dutch_umls_snomed_drugs = dutch_umls_snomed_drugs.drop(dutch_umls_snomed_drugs.index[rows_to_remove])\n",
    "print(f'Number of rows filtering TUIs: {dutch_umls_snomed_drugs.shape[0]}')\n",
    "\n",
    "# Merge TUIs in single value\n",
    "dutch_umls_snomed_drugs = dutch_umls_snomed_drugs.groupby(['cui', 'name', 'name_status', 'ontologies'])['tui'].apply('|'.join).reset_index()\n",
    "print(f'Number of rows after merging TUIs in single value: {dutch_umls_snomed_drugs.shape[0]}')\n",
    "\n",
    "# Rename TUI column to type_ids\n",
    "dutch_umls_snomed_drugs.rename(columns={'tui': 'type_ids'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4d55e08",
   "metadata": {},
   "outputs": [],
   "source": [
    "dutch_umls_snomed_drugs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ad53b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grouping rows on SAB\n",
    "print(f'Number of names before merging rows: {dutch_umls_snomed_drugs.shape[0]}')\n",
    "dutch_umls_snomed_drugs = dutch_umls_snomed_drugs.groupby(['cui', 'name', 'type_ids'], as_index=False).agg({'ontologies' : '|'.join, 'name_status' : '|'.join}).copy()\n",
    "\n",
    "# Clean name status column\n",
    "dutch_umls_snomed_drugs.name_status = dutch_umls_snomed_drugs.name_status.apply(clean_name_status_column)\n",
    "dutch_umls_snomed_drugs.sort_values(by=['cui', 'name_status'], ascending=[True, False], inplace=True)\n",
    "dutch_umls_snomed_drugs.reset_index(drop=True,inplace=True)\n",
    "print(f'Number of names after merging rows: {dutch_umls_snomed_drugs.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6fa5bf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check example\n",
    "dutch_umls_snomed_drugs.loc[dutch_umls_snomed_drugs.cui == 'C0020740']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c96d8881",
   "metadata": {},
   "source": [
    "### Save output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "089e68a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print statistics\n",
    "print(f'Unique concepts: {len(dutch_umls_snomed_drugs.cui.unique())}')\n",
    "print(f'Unique names: {len(dutch_umls_snomed_drugs.name.unique())}')\n",
    "print(f'Ambiguous names: {len(dutch_umls_snomed_drugs) - len(dutch_umls_snomed_drugs.name.unique())}')\n",
    "print(f'Total names in concepts: {len(dutch_umls_snomed_drugs)}')\n",
    "\n",
    "# Save final concept table with drugs\n",
    "dutch_umls_snomed_drugs.to_csv(output_file_with_drug_names, index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}