dutch-snomed_to_concept-table.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "80e1c145",
   "metadata": {},
   "source": [
    "# Dutch SNOMED to concept table\n",
    "This notebook describes how to create a SNOMED concept table containing Dutch names, to be used in a named entity recognition and linking tool such as MedCAT. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cca26e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "\n",
    "# Input\n",
    "nl_terms = Path('01_Download/SnomedCT_ManagedServiceNL_PRODUCTION_NL1000146_20220930T120000Z/Snapshot/Terminology/sct2_Description_Snapshot-nl_NL1000146_20220930.txt')\n",
    "\n",
    "# Output\n",
    "output_file_name = '04_ConceptDB/snomedct-dutch_v1.3.csv'\n",
    "output_file_name_unfiltered = '04_ConceptDB/snomedct-dutch_v1.3-unfiltered.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2c0de3a",
   "metadata": {},
   "source": [
    "## Sources files\n",
    "\n",
    "### SNOMED September 2020 release\n",
    "| Edition | Total names | NL names | Description |\n",
    "| - | - | - | - |\n",
    "| Edition | 2263140 | 736619 | Include international SNOMED |\n",
    "| Extension | 770016 | 736619 | Some terms are only in English |\n",
    "| Patient Friendly | 1437 | 1437 | Small but potentially useful list of synonyms |\n",
    "\n",
    "### SNOMED March 2021 release\n",
    "| Edition | Total names | NL names | Description |\n",
    "| - | - | - | - |\n",
    "| Edition | 2422738 | 880806 | Include international SNOMED |\n",
    "| Extension | 916553 | 880806 | Some terms are only in English |\n",
    "| Patient Friendly | 2004 | 2004 | Small but potentially useful list of synonyms |\n",
    "\n",
    "### SNOMED September 2021 release\n",
    "\n",
    "| Edition | Total names | NL names | Description |\n",
    "| - | - | - | - |\n",
    "| Edition | 2469845 | 910228 | Include international SNOMED |\n",
    "| Extension | 948571 | 910228 | Some terms are only in English |\n",
    "| Patient Friendly | 2385 | 2385 | Small but potentially useful list of synonyms |\n",
    "\n",
    "\n",
    "### SNOMED September 2022 release\n",
    "To count total number of records in the SNOMED files, use `wc -l` and substract 1 for header.\n",
    "```bash\n",
    "wc -l SnomedCT_*/Snapshot/Terminology/sct2_Description_*.txt\n",
    "```\n",
    "\n",
    "| File | Language | NL names |\n",
    "| - | - | - |\n",
    "| sct2_Description_Snapshot-en_NL1000146_20220930.txt | English | 1611131 |\n",
    "| sct2_Description_Snapshot-nl_NL1000146_20220930.txt | Dutch | 945292 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea192d7",
   "metadata": {},
   "source": [
    "### Description type\n",
    "The SNOMED description table contains 3 types of records:\n",
    "\n",
    "|Type id|Term|\n",
    "|-|-|\n",
    "|900000000000003001|Fully specified name (FSN)|\n",
    "|900000000000013009|Synonym|\n",
    "|900000000000550004|Definition|\n",
    "\n",
    "The the purpose of creating a list of names for entity recognation, terms must be:\n",
    "- FSN or Synonym\n",
    "- Active"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d555b3a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_file(filename):\n",
    "    with open(filename, encoding='utf-8') as f:\n",
    "        entities = [[n.strip() for n in line.split('\\t')] for line in f]\n",
    "        return pd.DataFrame(entities[1:], columns=entities[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ae1847d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select active terms\n",
    "df_terms = parse_file(nl_terms)\n",
    "df_terms_active = df_terms.loc[df_terms.active == '1'].copy()\n",
    "df_terms_inactive = df_terms.loc[df_terms.active != '1'].copy()\n",
    "print(f'Active terms: {df_terms_active.shape[0]}')\n",
    "print(f'Inactive terms: {df_terms_inactive.shape[0]}\\n')\n",
    "\n",
    "# Extract fully specified names\n",
    "df_fsn = df_terms_active[(df_terms_active.typeId == '900000000000003001')].copy()\n",
    "print(f'Active FSN: {df_fsn.shape[0]}')\n",
    "\n",
    "# Extract synonyms\n",
    "df_terms_synonyms = df_terms_active[(df_terms_active.typeId == '900000000000013009')].copy()\n",
    "print(f'Active Synonyms: {df_terms_synonyms.shape[0]}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f5557fb",
   "metadata": {},
   "source": [
    "### Primary concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e85c79d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_primary_concepts = df_fsn.copy()\n",
    "df_primary_concepts['tui'] = df_primary_concepts['term'].str.extract(r\"\\(([^)]*)\\)[^(]*$\")\n",
    "df_primary_concepts['str'] = df_primary_concepts['term'].str.extract(r\"(^[^\\(]+)\")\n",
    "df_primary_concepts['str'] = df_primary_concepts['str'].str.strip()\n",
    "df_primary_concepts = df_primary_concepts[['conceptId', 'typeId', 'tui', 'str']]\n",
    "\n",
    "# Create CUI-TUI mapping\n",
    "cui_tui_mapping = dict(zip(df_primary_concepts.conceptId, df_primary_concepts.tui))\n",
    "df_primary_concepts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49198a19",
   "metadata": {},
   "source": [
    "### Synonyms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50b6d6a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean synonym table\n",
    "df_synonyms = df_terms_synonyms.copy()\n",
    "df_synonyms = df_synonyms[['conceptId', 'term', 'typeId']]\n",
    "df_synonyms.rename({'term': 'str'}, inplace=True, axis=1)\n",
    "\n",
    "# Add TUI to synonyms\n",
    "df_synonyms['tui'] = df_synonyms.conceptId.map(cui_tui_mapping)\n",
    "# df_synonyms[df_synonyms['tui'].isna()].shape\n",
    "# 67 synonyms without type\n",
    "\n",
    "df_synonyms.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2dd5a4c",
   "metadata": {},
   "source": [
    "### Combined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9882328",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create combined \n",
    "df_all = pd.concat([df_primary_concepts, df_synonyms]).reset_index(drop=True)\n",
    "df_all.rename({'typeId': 'tty', 'conceptId': 'cui'}, inplace=True, axis=1)\n",
    "\n",
    "# Map to MedCAT's P (Preferred term) & A values\n",
    "# See https://github.com/CogStack/MedCAT/blob/master/examples/README.md\n",
    "df_all.tty.replace({'900000000000003001': 'P',\n",
    "                    '900000000000013009': 'A'}, inplace=True)\n",
    "\n",
    "# Use convention in UMLS where default English SNOMED is called SNOMEDCT_US\n",
    "df_all['sab'] = 'SNOMEDCT_NL'\n",
    "df_all = df_all[['cui', 'str', 'tty', 'tui', 'sab']]\n",
    "\n",
    "# Drop synonyms that are the same as primary name\n",
    "print(f'Records before dropping duplicates: {df_all.shape[0]}')\n",
    "df_all_unique = df_all.drop(df_all[(df_all.duplicated(subset=['cui', 'str', 'tui'], keep=False)) & (df_all.tty=='A')].index)\n",
    "print(f'Records after dropping duplicates: {df_all_unique.shape[0]}')\n",
    "\n",
    "# Sort column on cui and tty\n",
    "df_all_unique['cui'] = df_all_unique['cui'].astype(int)\n",
    "df_all_unique.sort_values(['cui', 'tty'], ascending=[True, False], inplace=True)\n",
    "\n",
    "df_all_unique.head(25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7ee548a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A few rows contain NaN\n",
    "# Easiest way to deal with it is drop them. \n",
    "display(df_all_unique[df_all_unique.isnull().any(axis=1)].head())\n",
    "print(len(df_all_unique))\n",
    "df_all_unique.dropna(inplace=True)\n",
    "print(len(df_all_unique))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0d328a2",
   "metadata": {},
   "source": [
    "### Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb3e8d6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.str == 'ALS']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5664211",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.cui == 86044005]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8071318",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.str == 'longkanker']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e373b3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.cui == 93880001]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bad4c0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.cui == 22298006]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d527a483",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.str == 'methotrexaat']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4634b3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.cui == 387381009]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72c009f5",
   "metadata": {},
   "source": [
    "## Evaluation of SNOMED types"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faa7cba8",
   "metadata": {},
   "source": [
    "To select the types relevant for named entity linking, we assessed the performance of a MedCAT model on a set of example documents using the unfiltered SNOMED terms. We noticed some types are not useful for our general purpose (named entity recognition), and introduce false positives and ambiguity. We exclude the less useful types from our concept table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19af7c02",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique.tui.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f435a509",
   "metadata": {},
   "source": [
    "| tui | usefulness for NER | useful examples for NER | useless examples for NER |\n",
    "| :- | :- | :- | :-|\n",
    "|aandoening |good|hypertensie, boezemfibrilleren, av-blok| |\n",
    "|monster|good|trombocyten, leukocyten,basofiele granulocyten||\n",
    "|regime/therapie|good|fysiotherapie, hartrevalidatie, therapie||\n",
    "|waarneembare entities|good|leeftijd, bloeddruk, hartas, LVEF||\n",
    "|bevinding|good|koorts, zwelling, tachycardie||\n",
    "|attribuut|good|bij, na||\n",
    "|kwalificatiewaarde|good|ontslag, beloop, gestaakt, geen||\n",
    "|afwijkende morfologie|good|thermisch letsel, blaar, luxatie||\n",
    "|cel|good, but rare|erythrocytes||\n",
    "|gegevensobject|good, but rare| Echocardiogram, operatieverslag||\n",
    "|sociaal concept|good, but rare|familie||\n",
    "|situatie|good, but includes negation|geen pijn, geen dispneu, geen hoesten||\n",
    "|verrichting |decent|lokale anesthesie, lichamelijk onderzoek, palpatie|erg (elektroretinografie), weken, post (peritoneale transfer van eicel en sperma), beleid (management)|\n",
    "|substantie|decent|nebivolol, amlodipine, ceftriaxon|wortel, PM (fijnstof)|\n",
    "|omgeving|decent|ziekenhuis, polikliniek, huis, afdeling cardiologie, afdeling fysiologie|meer, stroop, plaats|\n",
    "|lichaamsstructuur|decent|pols, aortaklep, AV-knoop|mid (mesioincisodistale vlakken van gebitselement)|\n",
    "|persoon|decent|patient, dochter, vader|bekende (kennis)|\n",
    "|fysische kracht|decent, but rare|druk||\n",
    "|fysiek object|discutable, but rare|pacemaker|verband|\n",
    "|beroep|bad||rechter, belang (behanger), herkende (werkende)|\n",
    "|metadata|bad||beeld|\n",
    "|gebeurtenis|bad, only 1 term matched||het (hoog energetisch trauma)|\n",
    "\n",
    "Terms that were never found: omgeving/locatie, organisme, physical object: fout, religie/filosofie, product, disorder, navigatieconcept, lifestyle, procedure, gradering, tumorgradering, beoordelingsschaal, inactief concept, speciaal concept, ras, foundation metadata concept, physical object, eigenschap, celstructuur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1b4f3ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Total number of concepts\n",
    "df_all_unique.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3524d57",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of primary concepts\n",
    "df_all_unique[df_all_unique.tty == 'P'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4954e9e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of synonyms\n",
    "df_all_unique[df_all_unique.tty == 'A'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9156f0a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define list of relevant terms for general use\n",
    "relevant_tuis = ['aandoening',\n",
    "                'monster',\n",
    "                'regime/therapie',\n",
    "                'bevinding',\n",
    "                'afwijkende morfologie',\n",
    "                'cel',\n",
    "                'gegevensobject',\n",
    "                'verrichting',\n",
    "                'substantie',\n",
    "                'lichaamsstructuur']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cb5bdfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.tui.isin(relevant_tuis)].tui.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f106994",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_unique[df_all_unique.tui.isin(relevant_tuis)].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "788dfad3",
   "metadata": {},
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f1aca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unfiltered concepts\n",
    "print(f'Number of concepts: {len(df_all_unique.cui.unique())}')\n",
    "print(f'Number of names: {len(df_all_unique)}')\n",
    "df_all_unique.to_csv(output_file_name_unfiltered, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "776cbd5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filtered concepts\n",
    "filtered_concepts = df_all_unique[df_all_unique.tui.isin(relevant_tuis)]\n",
    "print(f'Number of concepts: {len(filtered_concepts.cui.unique())}')\n",
    "print(f'Number of names: {len(filtered_concepts)}')\n",
    "filtered_concepts.to_csv(output_file_name, index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}