Create Train_Dev_Test Splits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Train/Dev/Test Split for BTC Dataset\n",
    "\n",
    "Hello, I made this to create a train/dev/test split from Derczynski et al.'s Broad Twitter Corpus Dataset (https://github.com/GateNLP/broad_twitter_corpus).\n",
    "\n",
    "I made this real quick so it isn't really top-notch quality coding, but it works.\n",
    "\n",
    "\n",
    "\n",
    "Made by Christopher Goh (https://github.com/chrisgzf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List of conll files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a.conll', 'b.conll', 'e.conll', 'f.conll', 'g.conll', 'h.conll']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "\n",
    "filelist = glob.glob(\"*.conll\")\n",
    "filelist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To count the number of tweets in each file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a.conll 1000\n",
      "b.conll 2000\n",
      "e.conll 200\n",
      "f.conll 2001\n",
      "g.conll 2138\n",
      "h.conll 2001\n"
     ]
    }
   ],
   "source": [
    "for file in filelist:\n",
    "    tweets = list()\n",
    "    tweet = list()\n",
    "    with open(file, encoding=\"utf-8\") as f:\n",
    "        contents = f.readlines()\n",
    "    for line in contents:\n",
    "        if line == \"\\n\" or line == \" \\n\" or line == \"  \\n\" or line == \"   \\n\":\n",
    "            if len(tweet) > 0:\n",
    "                tweets.append(tweet)\n",
    "                tweet = list()\n",
    "        else:\n",
    "            tweet.append(line)\n",
    "    if len(tweet) > 0:\n",
    "        tweets.append(tweet)\n",
    "    print(file, len(tweets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The BTC GitHub Repo states that `f`, `g`, `h` should have 2000, 2351, 2000 tweets respectively. Weird."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the BTC GitHub Repo:\n",
    "\n",
    "- Test: Section F\n",
    "- Development: second half of Section H\n",
    "- Training: everything else"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Processing `H` first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2001\n"
     ]
    }
   ],
   "source": [
    "tweets = list()\n",
    "tweet = list()\n",
    "\n",
    "with open(\"h.conll\", encoding=\"utf-8\") as f:\n",
    "    contents = f.readlines()\n",
    "for line in contents:\n",
    "    if line == \"\\n\" or line == \" \\n\" or line == \"  \\n\" or line == \"   \\n\":\n",
    "        if len(tweet) > 0:\n",
    "            tweets.append(tweet)\n",
    "            tweet = list()\n",
    "    else:\n",
    "        tweet.append(line)\n",
    "if len(tweet) > 0:\n",
    "    tweets.append(tweet)\n",
    "print(len(tweets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting H into half, and writing the dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = tweets[1000:]\n",
    "train_from_h = tweets[:1000]\n",
    "\n",
    "with open(\"btc.dev\", \"w+\", encoding=\"utf-8\") as f:\n",
    "    for tweet in dev:\n",
    "        for line in tweet:\n",
    "            f.write(line)\n",
    "        f.write(\"\\n\")    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the train set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a.conll', 'b.conll', 'e.conll', 'g.conll']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filelist.remove(\"f.conll\")\n",
    "filelist.remove(\"h.conll\")\n",
    "filelist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets = list()\n",
    "for file in filelist:\n",
    "    tweet = list()\n",
    "    with open(file, encoding=\"utf-8\") as f:\n",
    "        contents = f.readlines()\n",
    "    for line in contents:\n",
    "        if line == \"\\n\" or line == \" \\n\" or line == \"  \\n\" or line == \"   \\n\":\n",
    "            if len(tweet) > 0:\n",
    "                tweets.append(tweet)\n",
    "                tweet = list()\n",
    "        else:\n",
    "            tweet.append(line)\n",
    "    if len(tweet) > 0:\n",
    "        tweets.append(tweet)\n",
    "tweets += train_from_h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6338"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The math adds up, time to write the train set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"btc.train\", \"w+\", encoding=\"utf-8\") as f:\n",
    "    for tweet in tweets:\n",
    "        for line in tweet:\n",
    "            f.write(line)\n",
    "        f.write(\"\\n\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the test set just requires you to copy the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'btc.test'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from shutil import copyfile\n",
    "\n",
    "copyfile(\"f.conll\", \"btc.test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Counting the number of tweets in each file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "btc.dev 1001\n",
      "btc.test 2001\n",
      "btc.train 6338\n"
     ]
    }
   ],
   "source": [
    "for file in [\"btc.dev\", \"btc.test\", \"btc.train\"]:\n",
    "    tweets = list()\n",
    "    tweet = list()\n",
    "    with open(file, encoding=\"utf-8\") as f:\n",
    "        contents = f.readlines()\n",
    "    for line in contents:\n",
    "        if line == \"\\n\" or line == \" \\n\" or line == \"  \\n\" or line == \"   \\n\":\n",
    "            if len(tweet) > 0:\n",
    "                tweets.append(tweet)\n",
    "                tweet = list()\n",
    "        else:\n",
    "            tweet.append(line)\n",
    "    if len(tweet) > 0:\n",
    "        tweets.append(tweet)\n",
    "    print(file, len(tweets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}